# Selecting a Spare HDD for New Ceph OSD

## When to Use

When the Ceph cluster needs an additional HDD OSD to relieve pressure on
overloaded OSDs (especially in EC4+1 configurations with exactly k+m OSDs,
where reweighting is impossible).

## Selection Criteria (Priority Order)

1. **Lowest power-on hours** — least worn, longest remaining life
2. **Same device class** — must be `hdd` (rotational) for HDD crush rule
3. **NAS-grade preferred** — WD Red Plus / IronWolf over WD Green / desktop
4. **Size ≥ existing OSDs** — larger is better, helps CRUSH balance
5. **Not currently an OSD** — check `ceph-volume lvm list` and `lsblk`

## Discovery Workflow

```bash
# 1. List all disks with serial numbers and rotation type
ssh root@<host> "
for d in /dev/sd[a-z]; do
  [ -b \$d ] && echo \"--- \$d ---\" && \
  smartctl -i \$d 2>/dev/null | grep -E 'Model Family|Device Model|Serial|Capacity|Rotation'
done"

# 2. Check which disks are already Ceph OSDs
ssh root@<host> "ceph-volume lvm list 2>/dev/null | grep -E 'osd.id|devices'"

# 3. Check power-on hours for all candidate disks
ssh root@<host> "
for d in /dev/sd[b-z]; do
  echo -n \"\$d \"
  smartctl -A \$d 2>/dev/null | grep Power_On_Hours | awk '{print \$10}'
done"

# 4. Check which disks have leftover filesystems (ZFS, etc.)
ssh root@<host> "
for d in /dev/sd[b-z]; do
  echo \"--- \$d ---\"
  lsblk \$d -o NAME,FSTYPE,MOUNTPOINT,SIZE 2>/dev/null
done"
```

## Candidate Evaluation Table (Example: 10.0.30.100, 2026-07-07)

| Disk | Model | Serial | Power-On Hours | Status | Score |
|------|-------|--------|---------------|--------|-------|
| sdb | WD30EFRX (Red) | WD-WCC4N4VRJ5UU | 81,891 | ZFS remnants | ❌ High wear |
| sdc | WD30EZRX (Green) | WD-WMC1T2256499 | 89,367 | ZFS remnants | ❌ High wear + Green |
| sdd | WD30EFRX (Red) | WD-WCC4N4SVR8JL | 58,031 | ZFS remnants | ⚠️ Medium wear |
| sde | WD30EFRX (Red) | WD-WMC4N2399595 | 104,433 | **osd.8** | ❌ In use |
| sdf | WD30EFRX (Red) | WD-WMC4N2400243 | 98,581 | **osd.6** | ❌ In use |
| sdg | WD30EFZX (Red+) | WD-WX12DA01FUA7 | 31,006 | ZFS remnants | ✅ Good |
| sdh | WD30EFZX (Red+) | WD-WX12DA0R1NFH | 31,007 | ZFS remnants | ✅ Good |
| sdi | WD30EFZX (Red+) | WD-WX22DA0D7EP3 | 31,006 | ZFS remnants | ✅ Best (lowest hours) |

**Selected: /dev/sdi** — WD Red Plus, 31k hours (~3.5 years), NAS-grade.

## Preparation Steps (After Physical Installation)

```bash
# 1. Wipe any leftover filesystem signatures
sgdisk -Z /dev/sdX
wipefs -a /dev/sdX

# 2. Create BlueStore OSD with DB/WAL on SSD (if available)
# On the target node (e.g. n5pro with NVMe):
ceph-volume lvm prepare --bluestore --data /dev/sdX \
  --block.db /dev/nvme0n1pX --block.wal /dev/nvme0n1pX

# 3. Or use PVE's ceph-volume integration:
pveceph osd create /dev/sdX --db_dev /dev/nvme0n1pX

# 4. Verify OSD joins and rebalancing begins
ceph osd tree | grep hdd
ceph status | grep recovery
```

## Context: Why This OSD Was Needed

EC4+1 (k=4, m=1) with exactly 5 HDD OSDs creates a CRUSH trap where
reweighting any OSD causes `2147483647` ("no OSD available") in PG
up-sets. Adding a 6th HDD OSD gives CRUSH room to redistribute,
enabling future reweighting and rebalancing without trapping PGs.

See `references/ceph-recovery-ec-reweight-2026-07.md` for the full
EC reweight trap diagnosis and fix.
