dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
hermes-skills/devops/proxmox-ve-administration/references/zfs-to-ceph-migration-2026-07.md
T

8.2 KiB
Raw Permalink Blame History

ZFS-to-Ceph Migration Analysis (2026-07-03)

Context

User wants to decommission osd.1 (USB-connected Hitachi HDD on proxmox1, slow BlueStore ops) and replace it with disks from the ZFS pool on 10.0.30.100. Initial assumption was "9×2.7TB" but actual inventory revealed 4×3TB in RAIDZ2 + 2 free disks + 2 already in Ceph.

Disk Inventory (10.0.30.100, 2026-07-03)

Disk Model Serial Size POH Age SMART Role Verdict
sda Crucial MX300 SSD 164914EEB886 275GB 81,871 9.3y ✅ 0/0/0 Boot SSD (swap→DB for osd.8) Keep — sda3 now DB device
sdb WD30EFRX Red+ WCC4N4VRJ5UU 3TB 81,786 9.3y ✅ 0/0/0, CRC=3 ZFS pool Oldest in pool, first offline candidate
sdc WD30EZRX Green WMC1T2256499 3TB 89,262 10.2y ⛔ 371 pending, 243 uncorr Free REJECT — defective
sdd WD30EFRX Red+ WCC4N4SVR8JL 3TB 57,925 6.6y ⚠️ 16 pending, 15 uncorr, Self-test FAIL Free REJECT — confirmed read failure
sde WD30EFRX Red+ WMC4N2399595 3TB 104,327 11.9y ✅ 0/0/0 Ceph osd.8 Active, very old. NOW HAS DB ON SSD
sdf WD30EFRX Red+ WMC4N2400243 3TB 98,475 11.2y ✅ 0/0/0 Ceph osd.9 Active, very old. NO DB device
sdg WD30EFZX Purple WX12DA01FUA7 3TB 30,901 3.5y ✅ 0/0/0 ZFS pool Best candidate
sdh WD30EFZX Purple WX12DA0R1NFH 3TB 30,901 3.5y ✅ 0/0/0 ZFS pool Best candidate
sdi WD30EFZX Purple WX22DA0D7EP3 3TB 30,900 3.5y ✅ 0/0/0 ZFS pool Best candidate

SMART Self-Test History (sdd)

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       90%     57925         62939576
# 2  Short offline       Completed without error       00%     50673         -
# 3  Extended offline    Completed: read failure       10%     50659         63009384
# 4  Short offline       Completed without error       00%     50651         -

Two read failures at LBA ~63M (~32GB region), 7000h apart. Progressive degradation.

badblocks Test (sdd)

  • Command: badblocks -wsv -b 4096 /dev/sdd
  • Started: 2026-07-03 12:25:31 CEST
  • Killed by user request at ~3.4% (Pattern 1/4: 0xaa)
  • Errors found: 0 (but test was aborted early)
  • Log: /tmp/badblocks_sdd.log on 10.0.30.100
  • Verdict: Inconclusive (early abort), but SMART self-test failure already disqualifies disk

ZFS Pool Details

Pool: pool01_n2_redundant
Type: RAIDZ2 (2 parity)
Disks: sdg, sdh, sdi, sdb (4× 2.73TB)
Raw: 10.9TB
Usable: ~5.46TB (2 data + 2 parity)
Used: 3.63TB (66%)
Free: 1.87TB
Fragmentation: 57%
Health: ONLINE, 0 errors
Dedup: 1.12x
Scrub: Monthly, last 2026-06-28, 0 errors

Datasets: Docker volumes, TimeMachine (~951GB), proxmox_nfs (~963GB), HomeAssistant.

Ceph State (2026-07-03)

  • Health: HEALTH_WARN (BLUESTORE_SLOW_OP_ALERT on osd.0 & osd.1)
  • 10 OSDs up/in, 12TB raw, 5.1TB used, 7.0TB free
  • osd.1: USB Hitachi HDD 3.6TiB on proxmox1 (10.0.20.10), slow ops
  • 7 pools active

Migration Strategies Evaluated

Option A: One-shot migration (recommended)

  1. Migrate all 3.63TB ZFS data → Ceph (has 7.0TB free)
  2. zpool destroy → all 4 disks freed
  3. Create 4 new Ceph OSDs on n5pro
  4. Purge osd.1

Option B: Phased with ZFS degradation (CHOSEN)

  1. zpool offline pool sdb (oldest disk) → wipe → Ceph OSD (+3TB)
  2. ZFS now degraded (1 parity left), migrate data to Ceph
  3. zpool destroy → wipe sdg/sdh/sdi → 3 more Ceph OSDs (+9TB)

Chosen: Option B — allows incremental Ceph growth while ZFS data is migrated gradually.

Rejected: 2-disk offline

Offlining 2 disks from RAIDZ2 leaves 0 redundancy. Too risky during active data migration.

OSD DB/WAL Acceleration Analysis (2026-07-03)

Which OSDs Have/Lack DB Devices

OSD Host HDD DB on Flash? Flash Available Free on Flash
osd.1 proxmox1 sdc 3.6TB ❌ ✅ 1TB Toshiba NVMe 848 GB — BUT osd.1 being decommissioned
osd.7 proxmox7 sda 982GB ✅ 50GB ceph-db ✅ 256GB Toshiba NVMe 110 GB
osd.8 ubuntu(.30.100) sde 2.7TB ✅ NOW 30GB on sda3 256GB Crucial SSD 0 (swap consumed)
osd.9 ubuntu(.30.100) sdf 2.7TB ❌ Same SSD, no space ❌ Needs root shrink or new SSD
osd.10 proxmox6 sda 982GB ✅ 50GB ceph-db ✅ 256GB Toshiba NVMe 110 GB

DB Device Addition: osd.8 (2026-07-03, SUCCESSFUL)

Method: Repurposed 32GB swap partition (sda3) on boot SSD as LVM-backed DB device.

Steps taken:

  1. swapoff /dev/sda3 (1.5GB used, 6.4GB RAM free — safe)
  2. wipefs -a /dev/sda3
  3. pvcreate /dev/sda3 → vgcreate ceph-db-vg /dev/sda3 → lvcreate -L 30G -n osd-8-db ceph-db-vg
  4. systemctl stop ceph-osd@8
  5. ceph-volume lvm new-db --osd-id 8 --osd-fsid <fsid> --target ceph-db-vg/osd-8-db
  6. systemctl start ceph-osd@8

Result: osd.8 up with block.db → /dev/dm-2, devices = sda,sde. BlueFS spillover warning (1.1GB on HDD, 231MB on SSD) — expected to clear as compaction migrates data.

Key Lesson: ceph-bluestore-tool vs ceph-volume

ceph-bluestore-tool --command bluefs-bdev-new-db FAILS on Ceph Squid 19.2.x with "No valid bdev label found" — tried raw partition, LVM LV, with/without zap, as root and ceph user. The working tool is ceph-volume lvm new-db (requires LVM-formatted target, not raw partition).

Key Lessons

  1. RAIDZ cannot shrink — zpool remove doesn't work on RAIDZ vdevs. Only zpool offline (temporary) or zpool destroy (permanent) can free disks.
  2. SMART PASSED ≠ healthy — sdd passed overall SMART assessment but had confirmed self-test read failures. Always check self-test log, not just overall health.
  3. Assumptions about disk counts are dangerous — "9×2.7TB" was wrong; actual was 4×3TB in ZFS + 2 free + 2 in Ceph. Always verify with lsblk + zpool status before planning.
  4. badblocks -wsv is destructive and slow — 4 patterns × ~5h per 3TB = ~20h total. Only run on disks committed to repurposing. Can be killed early if SMART already disqualifies.
  5. SSH key selection — 10.0.30.100 accepts ~/.ssh/id_ed25519_proxmox (root), NOT the Galera key from 1Password.
  6. ceph-bluestore-tool bluefs-bdev-new-db is broken on Squid — Use ceph-volume lvm new-db instead. Requires LVM LV as target, not raw partition.
  7. BlueFS spillover after adding DB is normal — Existing RocksDB SSTs stay on HDD, new writes go to flash. Warning clears as compaction progresses.
  8. Swap can be safely repurposed for DB — If RAM is adequate (>4GB free with swap nearly empty), disabling swap to reclaim flash for Ceph DB is viable. Consider zram as swap replacement.
  9. ZFS has no defrag command — Fragmentation (e.g. 57%) only affects freespace map efficiency, not file integrity. The ONLY way to "defrag" ZFS is: copy data elsewhere -> destroy pool -> recreate -> copy back. Irrelevant if migrating off ZFS entirely.
  10. RAIDZ disk removal is permanent degradation — Pulling 1 disk from RAIDZ2 drops to RAIDZ1-equivalent (1 parity). ZFS marks the disk UNAVAIL but it stays in the vdev config. zpool detach does NOT work on RAIDZ members. Pool continues degraded until destroyed.
  11. zfs destroy -r enters D-state on degraded pools — On RAIDZ2 with defective disks, zfs destroy -r can hang in uninterruptible I/O (D-state) for 10+ minutes. Cannot be killed. Plan accordingly -- run in background.
  12. ZFS async free lag — After rm -rf on ZFS, zfs list shows OLD usage for minutes/hours. du -sh shows actual live data. The gap is TXG-async free, resolves automatically. Don't panic if zfs list says 290GB but du says 3GB.
  13. Background monitor processes block zfs destroy — Processes with cwd inside the dataset (including your own polling loops) prevent zfs destroy. Use lsof +D <path> and fuser -mv <path> to find ALL blockers, kill them first.
  14. n5pro WAL/DB prep pattern — Remove unused pve/data thin pool (0% allocated, not in storage.cfg) via lvremove pve/data, then lvcreate -L 30G -n wal-db-osd-X pve for each new OSD. Verified on n5pro (10.0.20.91, 128GB NVMe, gained 70GB for 2x 30GB DB LVs).
Reference in New Issue View Git Blame Copy Permalink