Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,88 @@
+# Selecting a Spare HDD for New Ceph OSD
+
+## When to Use
+
+When the Ceph cluster needs an additional HDD OSD to relieve pressure on
+overloaded OSDs (especially in EC4+1 configurations with exactly k+m OSDs,
+where reweighting is impossible).
+
+## Selection Criteria (Priority Order)
+
+1. **Lowest power-on hours** — least worn, longest remaining life
+2. **Same device class** — must be `hdd` (rotational) for HDD crush rule
+3. **NAS-grade preferred** — WD Red Plus / IronWolf over WD Green / desktop
+4. **Size ≥ existing OSDs** — larger is better, helps CRUSH balance
+5. **Not currently an OSD** — check `ceph-volume lvm list` and `lsblk`
+
+## Discovery Workflow
+
+```bash
+# 1. List all disks with serial numbers and rotation type
+ssh root@<host> "
+for d in /dev/sd[a-z]; do
+  [ -b \$d ] && echo \"--- \$d ---\" && \
+  smartctl -i \$d 2>/dev/null | grep -E 'Model Family|Device Model|Serial|Capacity|Rotation'
+done"
+
+# 2. Check which disks are already Ceph OSDs
+ssh root@<host> "ceph-volume lvm list 2>/dev/null | grep -E 'osd.id|devices'"
+
+# 3. Check power-on hours for all candidate disks
+ssh root@<host> "
+for d in /dev/sd[b-z]; do
+  echo -n \"\$d \"
+  smartctl -A \$d 2>/dev/null | grep Power_On_Hours | awk '{print \$10}'
+done"
+
+# 4. Check which disks have leftover filesystems (ZFS, etc.)
+ssh root@<host> "
+for d in /dev/sd[b-z]; do
+  echo \"--- \$d ---\"
+  lsblk \$d -o NAME,FSTYPE,MOUNTPOINT,SIZE 2>/dev/null
+done"
+```
+
+## Candidate Evaluation Table (Example: 10.0.30.100, 2026-07-07)
+
+| Disk | Model | Serial | Power-On Hours | Status | Score |
+|------|-------|--------|---------------|--------|-------|
+| sdb | WD30EFRX (Red) | WD-WCC4N4VRJ5UU | 81,891 | ZFS remnants | ❌ High wear |
+| sdc | WD30EZRX (Green) | WD-WMC1T2256499 | 89,367 | ZFS remnants | ❌ High wear + Green |
+| sdd | WD30EFRX (Red) | WD-WCC4N4SVR8JL | 58,031 | ZFS remnants | ⚠️ Medium wear |
+| sde | WD30EFRX (Red) | WD-WMC4N2399595 | 104,433 | **osd.8** | ❌ In use |
+| sdf | WD30EFRX (Red) | WD-WMC4N2400243 | 98,581 | **osd.6** | ❌ In use |
+| sdg | WD30EFZX (Red+) | WD-WX12DA01FUA7 | 31,006 | ZFS remnants | ✅ Good |
+| sdh | WD30EFZX (Red+) | WD-WX12DA0R1NFH | 31,007 | ZFS remnants | ✅ Good |
+| sdi | WD30EFZX (Red+) | WD-WX22DA0D7EP3 | 31,006 | ZFS remnants | ✅ Best (lowest hours) |
+
+**Selected: /dev/sdi** — WD Red Plus, 31k hours (~3.5 years), NAS-grade.
+
+## Preparation Steps (After Physical Installation)
+
+```bash
+# 1. Wipe any leftover filesystem signatures
+sgdisk -Z /dev/sdX
+wipefs -a /dev/sdX
+
+# 2. Create BlueStore OSD with DB/WAL on SSD (if available)
+# On the target node (e.g. n5pro with NVMe):
+ceph-volume lvm prepare --bluestore --data /dev/sdX \
+  --block.db /dev/nvme0n1pX --block.wal /dev/nvme0n1pX
+
+# 3. Or use PVE's ceph-volume integration:
+pveceph osd create /dev/sdX --db_dev /dev/nvme0n1pX
+
+# 4. Verify OSD joins and rebalancing begins
+ceph osd tree | grep hdd
+ceph status | grep recovery
+```
+
+## Context: Why This OSD Was Needed
+
+EC4+1 (k=4, m=1) with exactly 5 HDD OSDs creates a CRUSH trap where
+reweighting any OSD causes `2147483647` ("no OSD available") in PG
+up-sets. Adding a 6th HDD OSD gives CRUSH room to redistribute,
+enabling future reweighting and rebalancing without trapping PGs.
+
+See `references/ceph-recovery-ec-reweight-2026-07.md` for the full
+EC reweight trap diagnosis and fix.