89 lines
3.3 KiB
Markdown
89 lines
3.3 KiB
Markdown
# Selecting a Spare HDD for New Ceph OSD
|
|
|
|
## When to Use
|
|
|
|
When the Ceph cluster needs an additional HDD OSD to relieve pressure on
|
|
overloaded OSDs (especially in EC4+1 configurations with exactly k+m OSDs,
|
|
where reweighting is impossible).
|
|
|
|
## Selection Criteria (Priority Order)
|
|
|
|
1. **Lowest power-on hours** — least worn, longest remaining life
|
|
2. **Same device class** — must be `hdd` (rotational) for HDD crush rule
|
|
3. **NAS-grade preferred** — WD Red Plus / IronWolf over WD Green / desktop
|
|
4. **Size ≥ existing OSDs** — larger is better, helps CRUSH balance
|
|
5. **Not currently an OSD** — check `ceph-volume lvm list` and `lsblk`
|
|
|
|
## Discovery Workflow
|
|
|
|
```bash
|
|
# 1. List all disks with serial numbers and rotation type
|
|
ssh root@<host> "
|
|
for d in /dev/sd[a-z]; do
|
|
[ -b \$d ] && echo \"--- \$d ---\" && \
|
|
smartctl -i \$d 2>/dev/null | grep -E 'Model Family|Device Model|Serial|Capacity|Rotation'
|
|
done"
|
|
|
|
# 2. Check which disks are already Ceph OSDs
|
|
ssh root@<host> "ceph-volume lvm list 2>/dev/null | grep -E 'osd.id|devices'"
|
|
|
|
# 3. Check power-on hours for all candidate disks
|
|
ssh root@<host> "
|
|
for d in /dev/sd[b-z]; do
|
|
echo -n \"\$d \"
|
|
smartctl -A \$d 2>/dev/null | grep Power_On_Hours | awk '{print \$10}'
|
|
done"
|
|
|
|
# 4. Check which disks have leftover filesystems (ZFS, etc.)
|
|
ssh root@<host> "
|
|
for d in /dev/sd[b-z]; do
|
|
echo \"--- \$d ---\"
|
|
lsblk \$d -o NAME,FSTYPE,MOUNTPOINT,SIZE 2>/dev/null
|
|
done"
|
|
```
|
|
|
|
## Candidate Evaluation Table (Example: 10.0.30.100, 2026-07-07)
|
|
|
|
| Disk | Model | Serial | Power-On Hours | Status | Score |
|
|
|------|-------|--------|---------------|--------|-------|
|
|
| sdb | WD30EFRX (Red) | WD-WCC4N4VRJ5UU | 81,891 | ZFS remnants | ❌ High wear |
|
|
| sdc | WD30EZRX (Green) | WD-WMC1T2256499 | 89,367 | ZFS remnants | ❌ High wear + Green |
|
|
| sdd | WD30EFRX (Red) | WD-WCC4N4SVR8JL | 58,031 | ZFS remnants | ⚠️ Medium wear |
|
|
| sde | WD30EFRX (Red) | WD-WMC4N2399595 | 104,433 | **osd.8** | ❌ In use |
|
|
| sdf | WD30EFRX (Red) | WD-WMC4N2400243 | 98,581 | **osd.6** | ❌ In use |
|
|
| sdg | WD30EFZX (Red+) | WD-WX12DA01FUA7 | 31,006 | ZFS remnants | ✅ Good |
|
|
| sdh | WD30EFZX (Red+) | WD-WX12DA0R1NFH | 31,007 | ZFS remnants | ✅ Good |
|
|
| sdi | WD30EFZX (Red+) | WD-WX22DA0D7EP3 | 31,006 | ZFS remnants | ✅ Best (lowest hours) |
|
|
|
|
**Selected: /dev/sdi** — WD Red Plus, 31k hours (~3.5 years), NAS-grade.
|
|
|
|
## Preparation Steps (After Physical Installation)
|
|
|
|
```bash
|
|
# 1. Wipe any leftover filesystem signatures
|
|
sgdisk -Z /dev/sdX
|
|
wipefs -a /dev/sdX
|
|
|
|
# 2. Create BlueStore OSD with DB/WAL on SSD (if available)
|
|
# On the target node (e.g. n5pro with NVMe):
|
|
ceph-volume lvm prepare --bluestore --data /dev/sdX \
|
|
--block.db /dev/nvme0n1pX --block.wal /dev/nvme0n1pX
|
|
|
|
# 3. Or use PVE's ceph-volume integration:
|
|
pveceph osd create /dev/sdX --db_dev /dev/nvme0n1pX
|
|
|
|
# 4. Verify OSD joins and rebalancing begins
|
|
ceph osd tree | grep hdd
|
|
ceph status | grep recovery
|
|
```
|
|
|
|
## Context: Why This OSD Was Needed
|
|
|
|
EC4+1 (k=4, m=1) with exactly 5 HDD OSDs creates a CRUSH trap where
|
|
reweighting any OSD causes `2147483647` ("no OSD available") in PG
|
|
up-sets. Adding a 6th HDD OSD gives CRUSH room to redistribute,
|
|
enabling future reweighting and rebalancing without trapping PGs.
|
|
|
|
See `references/ceph-recovery-ec-reweight-2026-07.md` for the full
|
|
EC reweight trap diagnosis and fix.
|