133 lines
8.2 KiB
Markdown
133 lines
8.2 KiB
Markdown
# ZFS-to-Ceph Migration Analysis (2026-07-03)
|
||
|
||
## Context
|
||
|
||
User wants to decommission osd.1 (USB-connected Hitachi HDD on proxmox1, slow BlueStore ops)
|
||
and replace it with disks from the ZFS pool on 10.0.30.100. Initial assumption was "9×2.7TB"
|
||
but actual inventory revealed 4×3TB in RAIDZ2 + 2 free disks + 2 already in Ceph.
|
||
|
||
## Disk Inventory (10.0.30.100, 2026-07-03)
|
||
|
||
| Disk | Model | Serial | Size | POH | Age | SMART | Role | Verdict |
|
||
|------|-------|--------|------|-----|-----|-------|------|---------|
|
||
| sda | Crucial MX300 SSD | 164914EEB886 | 275GB | 81,871 | 9.3y | ✅ 0/0/0 | Boot SSD (swap→DB for osd.8) | Keep — sda3 now DB device |
|
||
| sdb | WD30EFRX Red+ | WCC4N4VRJ5UU | 3TB | 81,786 | 9.3y | ✅ 0/0/0, CRC=3 | ZFS pool | Oldest in pool, first offline candidate |
|
||
| sdc | WD30EZRX Green | WMC1T2256499 | 3TB | 89,262 | 10.2y | ⛔ 371 pending, 243 uncorr | Free | REJECT — defective |
|
||
| sdd | WD30EFRX Red+ | WCC4N4SVR8JL | 3TB | 57,925 | 6.6y | ⚠️ 16 pending, 15 uncorr, Self-test FAIL | Free | REJECT — confirmed read failure |
|
||
| sde | WD30EFRX Red+ | WMC4N2399595 | 3TB | 104,327 | 11.9y | ✅ 0/0/0 | Ceph osd.8 | Active, very old. NOW HAS DB ON SSD |
|
||
| sdf | WD30EFRX Red+ | WMC4N2400243 | 3TB | 98,475 | 11.2y | ✅ 0/0/0 | Ceph osd.9 | Active, very old. NO DB device |
|
||
| sdg | WD30EFZX Purple | WX12DA01FUA7 | 3TB | 30,901 | 3.5y | ✅ 0/0/0 | ZFS pool | Best candidate |
|
||
| sdh | WD30EFZX Purple | WX12DA0R1NFH | 3TB | 30,901 | 3.5y | ✅ 0/0/0 | ZFS pool | Best candidate |
|
||
| sdi | WD30EFZX Purple | WX22DA0D7EP3 | 3TB | 30,900 | 3.5y | ✅ 0/0/0 | ZFS pool | Best candidate |
|
||
|
||
## SMART Self-Test History (sdd)
|
||
|
||
```
|
||
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
|
||
# 1 Short offline Completed: read failure 90% 57925 62939576
|
||
# 2 Short offline Completed without error 00% 50673 -
|
||
# 3 Extended offline Completed: read failure 10% 50659 63009384
|
||
# 4 Short offline Completed without error 00% 50651 -
|
||
```
|
||
|
||
Two read failures at LBA ~63M (~32GB region), 7000h apart. Progressive degradation.
|
||
|
||
## badblocks Test (sdd)
|
||
|
||
- Command: `badblocks -wsv -b 4096 /dev/sdd`
|
||
- Started: 2026-07-03 12:25:31 CEST
|
||
- Killed by user request at ~3.4% (Pattern 1/4: 0xaa)
|
||
- Errors found: 0 (but test was aborted early)
|
||
- Log: `/tmp/badblocks_sdd.log` on 10.0.30.100
|
||
- Verdict: Inconclusive (early abort), but SMART self-test failure already disqualifies disk
|
||
|
||
## ZFS Pool Details
|
||
|
||
```
|
||
Pool: pool01_n2_redundant
|
||
Type: RAIDZ2 (2 parity)
|
||
Disks: sdg, sdh, sdi, sdb (4× 2.73TB)
|
||
Raw: 10.9TB
|
||
Usable: ~5.46TB (2 data + 2 parity)
|
||
Used: 3.63TB (66%)
|
||
Free: 1.87TB
|
||
Fragmentation: 57%
|
||
Health: ONLINE, 0 errors
|
||
Dedup: 1.12x
|
||
Scrub: Monthly, last 2026-06-28, 0 errors
|
||
```
|
||
|
||
Datasets: Docker volumes, TimeMachine (~951GB), proxmox_nfs (~963GB), HomeAssistant.
|
||
|
||
## Ceph State (2026-07-03)
|
||
|
||
- Health: HEALTH_WARN (BLUESTORE_SLOW_OP_ALERT on osd.0 & osd.1)
|
||
- 10 OSDs up/in, 12TB raw, 5.1TB used, 7.0TB free
|
||
- osd.1: USB Hitachi HDD 3.6TiB on proxmox1 (10.0.20.10), slow ops
|
||
- 7 pools active
|
||
|
||
## Migration Strategies Evaluated
|
||
|
||
### Option A: One-shot migration (recommended)
|
||
1. Migrate all 3.63TB ZFS data → Ceph (has 7.0TB free)
|
||
2. `zpool destroy` → all 4 disks freed
|
||
3. Create 4 new Ceph OSDs on n5pro
|
||
4. Purge osd.1
|
||
|
||
### Option B: Phased with ZFS degradation (CHOSEN)
|
||
1. `zpool offline pool sdb` (oldest disk) → wipe → Ceph OSD (+3TB)
|
||
2. ZFS now degraded (1 parity left), migrate data to Ceph
|
||
3. `zpool destroy` → wipe sdg/sdh/sdi → 3 more Ceph OSDs (+9TB)
|
||
|
||
**Chosen**: Option B — allows incremental Ceph growth while ZFS data is migrated gradually.
|
||
|
||
### Rejected: 2-disk offline
|
||
Offlining 2 disks from RAIDZ2 leaves 0 redundancy. Too risky during active data migration.
|
||
|
||
## OSD DB/WAL Acceleration Analysis (2026-07-03)
|
||
|
||
### Which OSDs Have/Lack DB Devices
|
||
|
||
| OSD | Host | HDD | DB on Flash? | Flash Available | Free on Flash |
|
||
|-----|------|-----|-------------|-----------------|---------------|
|
||
| osd.1 | proxmox1 | sdc 3.6TB | ❌ | ✅ 1TB Toshiba NVMe | 848 GB — BUT osd.1 being decommissioned |
|
||
| osd.7 | proxmox7 | sda 982GB | ✅ 50GB ceph-db | ✅ 256GB Toshiba NVMe | 110 GB |
|
||
| osd.8 | ubuntu(.30.100) | sde 2.7TB | ✅ NOW 30GB on sda3 | 256GB Crucial SSD | 0 (swap consumed) |
|
||
| osd.9 | ubuntu(.30.100) | sdf 2.7TB | ❌ | Same SSD, no space | ❌ Needs root shrink or new SSD |
|
||
| osd.10 | proxmox6 | sda 982GB | ✅ 50GB ceph-db | ✅ 256GB Toshiba NVMe | 110 GB |
|
||
|
||
### DB Device Addition: osd.8 (2026-07-03, SUCCESSFUL)
|
||
|
||
**Method**: Repurposed 32GB swap partition (sda3) on boot SSD as LVM-backed DB device.
|
||
|
||
Steps taken:
|
||
1. `swapoff /dev/sda3` (1.5GB used, 6.4GB RAM free — safe)
|
||
2. `wipefs -a /dev/sda3`
|
||
3. `pvcreate /dev/sda3` → `vgcreate ceph-db-vg /dev/sda3` → `lvcreate -L 30G -n osd-8-db ceph-db-vg`
|
||
4. `systemctl stop ceph-osd@8`
|
||
5. `ceph-volume lvm new-db --osd-id 8 --osd-fsid <fsid> --target ceph-db-vg/osd-8-db`
|
||
6. `systemctl start ceph-osd@8`
|
||
|
||
Result: osd.8 up with `block.db → /dev/dm-2`, `devices = sda,sde`. BlueFS spillover warning (1.1GB on HDD, 231MB on SSD) — expected to clear as compaction migrates data.
|
||
|
||
### Key Lesson: ceph-bluestore-tool vs ceph-volume
|
||
|
||
`ceph-bluestore-tool --command bluefs-bdev-new-db` FAILS on Ceph Squid 19.2.x with `"No valid bdev label found"` — tried raw partition, LVM LV, with/without zap, as root and ceph user. The working tool is `ceph-volume lvm new-db` (requires LVM-formatted target, not raw partition).
|
||
|
||
## Key Lessons
|
||
|
||
1. **RAIDZ cannot shrink** — `zpool remove` doesn't work on RAIDZ vdevs. Only `zpool offline` (temporary) or `zpool destroy` (permanent) can free disks.
|
||
2. **SMART PASSED ≠ healthy** — sdd passed overall SMART assessment but had confirmed self-test read failures. Always check self-test log, not just overall health.
|
||
3. **Assumptions about disk counts are dangerous** — "9×2.7TB" was wrong; actual was 4×3TB in ZFS + 2 free + 2 in Ceph. Always verify with `lsblk` + `zpool status` before planning.
|
||
4. **badblocks -wsv is destructive and slow** — 4 patterns × ~5h per 3TB = ~20h total. Only run on disks committed to repurposing. Can be killed early if SMART already disqualifies.
|
||
5. **SSH key selection** — 10.0.30.100 accepts `~/.ssh/id_ed25519_proxmox` (root), NOT the Galera key from 1Password.
|
||
6. **ceph-bluestore-tool bluefs-bdev-new-db is broken on Squid** — Use `ceph-volume lvm new-db` instead. Requires LVM LV as target, not raw partition.
|
||
7. **BlueFS spillover after adding DB is normal** — Existing RocksDB SSTs stay on HDD, new writes go to flash. Warning clears as compaction progresses.
|
||
8. **Swap can be safely repurposed for DB** — If RAM is adequate (>4GB free with swap nearly empty), disabling swap to reclaim flash for Ceph DB is viable. Consider zram as swap replacement.
|
||
9. **ZFS has no defrag command** — Fragmentation (e.g. 57%) only affects freespace map efficiency, not file integrity. The ONLY way to "defrag" ZFS is: copy data elsewhere -> destroy pool -> recreate -> copy back. Irrelevant if migrating off ZFS entirely.
|
||
10. **RAIDZ disk removal is permanent degradation** — Pulling 1 disk from RAIDZ2 drops to RAIDZ1-equivalent (1 parity). ZFS marks the disk UNAVAIL but it stays in the vdev config. `zpool detach` does NOT work on RAIDZ members. Pool continues degraded until destroyed.
|
||
11. **zfs destroy -r enters D-state on degraded pools** — On RAIDZ2 with defective disks, `zfs destroy -r` can hang in uninterruptible I/O (D-state) for 10+ minutes. Cannot be killed. Plan accordingly -- run in background.
|
||
12. **ZFS async free lag** — After `rm -rf` on ZFS, `zfs list` shows OLD usage for minutes/hours. `du -sh` shows actual live data. The gap is TXG-async free, resolves automatically. Don't panic if `zfs list` says 290GB but `du` says 3GB.
|
||
13. **Background monitor processes block zfs destroy** — Processes with `cwd` inside the dataset (including your own polling loops) prevent `zfs destroy`. Use `lsof +D <path>` and `fuser -mv <path>` to find ALL blockers, kill them first.
|
||
14. **n5pro WAL/DB prep pattern** — Remove unused `pve/data` thin pool (0% allocated, not in storage.cfg) via `lvremove pve/data`, then `lvcreate -L 30G -n wal-db-osd-X pve` for each new OSD. Verified on n5pro (10.0.20.91, 128GB NVMe, gained 70GB for 2x 30GB DB LVs).
|