Selecting a Spare HDD for New Ceph OSD

When to Use

When the Ceph cluster needs an additional HDD OSD to relieve pressure on overloaded OSDs (especially in EC4+1 configurations with exactly k+m OSDs, where reweighting is impossible).

Selection Criteria (Priority Order)

Lowest power-on hours — least worn, longest remaining life
Same device class — must be hdd (rotational) for HDD crush rule
NAS-grade preferred — WD Red Plus / IronWolf over WD Green / desktop
Size ≥ existing OSDs — larger is better, helps CRUSH balance
Not currently an OSD — check ceph-volume lvm list and lsblk

Discovery Workflow

# 1. List all disks with serial numbers and rotation type
ssh root@<host> "
for d in /dev/sd[a-z]; do
  [ -b \$d ] && echo \"--- \$d ---\" && \
  smartctl -i \$d 2>/dev/null | grep -E 'Model Family|Device Model|Serial|Capacity|Rotation'
done"

# 2. Check which disks are already Ceph OSDs
ssh root@<host> "ceph-volume lvm list 2>/dev/null | grep -E 'osd.id|devices'"

# 3. Check power-on hours for all candidate disks
ssh root@<host> "
for d in /dev/sd[b-z]; do
  echo -n \"\$d \"
  smartctl -A \$d 2>/dev/null | grep Power_On_Hours | awk '{print \$10}'
done"

# 4. Check which disks have leftover filesystems (ZFS, etc.)
ssh root@<host> "
for d in /dev/sd[b-z]; do
  echo \"--- \$d ---\"
  lsblk \$d -o NAME,FSTYPE,MOUNTPOINT,SIZE 2>/dev/null
done"

Candidate Evaluation Table (Example: 10.0.30.100, 2026-07-07)

Disk	Model	Serial	Power-On Hours	Status	Score
sdb	WD30EFRX (Red)	WD-WCC4N4VRJ5UU	81,891	ZFS remnants	❌ High wear
sdc	WD30EZRX (Green)	WD-WMC1T2256499	89,367	ZFS remnants	❌ High wear + Green
sdd	WD30EFRX (Red)	WD-WCC4N4SVR8JL	58,031	ZFS remnants	⚠️ Medium wear
sde	WD30EFRX (Red)	WD-WMC4N2399595	104,433	osd.8	❌ In use
sdf	WD30EFRX (Red)	WD-WMC4N2400243	98,581	osd.6	❌ In use
sdg	WD30EFZX (Red+)	WD-WX12DA01FUA7	31,006	ZFS remnants	✅ Good
sdh	WD30EFZX (Red+)	WD-WX12DA0R1NFH	31,007	ZFS remnants	✅ Good
sdi	WD30EFZX (Red+)	WD-WX22DA0D7EP3	31,006	ZFS remnants	✅ Best (lowest hours)

Selected: /dev/sdi — WD Red Plus, 31k hours (~3.5 years), NAS-grade.

Preparation Steps (After Physical Installation)

# 1. Wipe any leftover filesystem signatures
sgdisk -Z /dev/sdX
wipefs -a /dev/sdX

# 2. Create BlueStore OSD with DB/WAL on SSD (if available)
# On the target node (e.g. n5pro with NVMe):
ceph-volume lvm prepare --bluestore --data /dev/sdX \
  --block.db /dev/nvme0n1pX --block.wal /dev/nvme0n1pX

# 3. Or use PVE's ceph-volume integration:
pveceph osd create /dev/sdX --db_dev /dev/nvme0n1pX

# 4. Verify OSD joins and rebalancing begins
ceph osd tree | grep hdd
ceph status | grep recovery

Context: Why This OSD Was Needed

EC4+1 (k=4, m=1) with exactly 5 HDD OSDs creates a CRUSH trap where reweighting any OSD causes 2147483647 ("no OSD available") in PG up-sets. Adding a 6th HDD OSD gives CRUSH room to redistribute, enabling future reweighting and rebalancing without trapping PGs.

See references/ceph-recovery-ec-reweight-2026-07.md for the full EC reweight trap diagnosis and fix.

3.3 KiB Raw Permalink Blame History