Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,167 @@
+# Ceph Recovery Acceleration — Advanced Tuning — 2026-07-05
+
+## Context
+
+Follow-up to `ceph-pool-full-recovery-2026-07.md`. After unblocking backfill by raising `backfillfull_ratio` and reweighting OSD 7, recovery was progressing but slowly (22 MiB/s, 5 obj/s, only 2 PGs backfilling). This session applied additional tuning to accelerate recovery.
+
+## Additional Recovery Parameters
+
+### `osd_recovery_op_priority` (effective)
+
+```bash
+# Set globally via MON config
+ceph config set osd osd_recovery_op_priority 10
+# Default: 3. Range: 1-63. Higher = recovery ops preempt client ops.
+
+# Verify on running OSD:
+ceph daemon osd.7 config get osd_recovery_op_priority
+# Returns: {"osd_recovery_op_priority": "10"}
+```
+
+Unlike `osd_max_backfills`, this one DOES apply at runtime via `ceph config set` — no restart needed. OSDs pick it up within seconds.
+
+### `osd_recovery_sleep` (set to 0)
+
+```bash
+ceph config set osd osd_recovery_sleep 0
+# Default: 0.001 (1ms pause between recovery ops). Set to 0 for maximum throughput.
+```
+
+### `osd_recovery_max_chunk`
+
+```bash
+ceph config set osd osd_recovery_max_chunk 104857600
+# Default: 8MiB. Set to 100MiB for larger transfer windows.
+```
+
+### `ceph tell osd.N config set` — Alternative to `injectargs`
+
+Newer Ceph (Reef/Squid) supports `ceph tell osd.N config set` as a cleaner alternative to `injectargs`:
+
+```bash
+# Instead of: ceph tell osd.7 injectargs "--osd_max_backfills 3"
+# Use:
+ceph tell osd.7 config set osd_max_backfills 3
+```
+
+However, in testing, `ceph config set osd osd_max_backfills 3` (MON-level) did NOT propagate to running OSDs — the OSDs kept using value 1. The `ceph tell osd.N config set` also didn't reliably stick. The most reliable method remains:
+
+```bash
+# Most reliable runtime method:
+ceph tell osd.7 injectargs "--osd_max_backfills 3"
+# Verify:
+ceph daemon osd.7 config get osd_max_backfills
+# Must return "3" — if still "1", the config hasn't been applied.
+```
+
+### `osd_recovery_max_active` — NOT effectively tunable
+
+`ceph config set osd osd_recovery_max_active 3` accepts the value but OSDs report 0 (meaning auto/default). `injectargs` also shows empty. This appears to be a Ceph limitation — the parameter is managed internally. Do not waste time trying to force it.
+
+## Full Recovery Tuning Recipe
+
+```bash
+# 1. Unblock backfill (if backfillfull)
+ceph osd set-backfillfull-ratio 0.97
+ceph osd set-nearfull-ratio 0.97
+
+# 2. Reweight overfull OSD
+ceph osd crush reweight osd.7 0.50
+
+# 3. Set recovery parameters via MON config (applies to new/restarted OSDs)
+ceph config set osd osd_max_backfills 3
+ceph config set osd osd_recovery_op_priority 10
+ceph config set osd osd_recovery_sleep 0
+ceph config set osd osd_recovery_max_chunk 104857600
+
+# 4. Force runtime application on each HDD OSD (MON config doesn't propagate to running OSDs)
+for osd in 1 6 7 8 10; do
+  ceph tell osd.$osd injectargs "--osd_max_backfills 3"
+  ceph tell osd.$osd injectargs "--osd_recovery_op_priority 10"
+done
+
+# 5. Verify each OSD picked up the config
+for osd in 1 6 7 8 10; do
+  echo "=== osd.$osd ==="
+  ceph daemon osd.$osd config get osd_max_backfills
+  ceph daemon osd.$osd config get osd_recovery_op_priority
+done
+```
+
+## Recovery Speed Progression
+
+| Stage | Recovery Speed | Concurrent Backfills | Notes |
+|-------|---------------|---------------------|-------|
+| Before intervention | 0 MiB/s (stuck) | 0 | backfillfull blocks all |
+| After ratio + reweight | 22-23 MiB/s, 5 obj/s | 1-2 | Backfill unblocked |
+| After injectargs (max_backfills=3) | 34 MiB/s, 8 obj/s | 2-3 | Even with max=3, Ceph schedules conservatively |
+| After op_priority=10 + sleep=0 | ~34 MiB/s, 8 obj/s | 2-3 | Priority helps under client I/O load |
+
+Even with `osd_max_backfills=3`, expect only 2-3 PGs backfilling simultaneously — Ceph schedules conservatively. The throughput improvement comes from larger transfer windows and reduced pauses, not dramatically more parallel PGs.
+
+## OSD Restart Considerations
+
+If `injectargs` doesn't work, restarting the OSD daemon forces it to pick up the MON config:
+
+```bash
+# ⚠️ MUST restart on the correct node! Use ceph osd find to locate:
+ceph osd find 7
+# Returns JSON with host name and IP
+
+# SSH to THAT node and restart:
+ssh root@<correct_node_ip> "systemctl restart ceph-osd@7"
+# ❌ WRONG: restarting on a node that doesn't host the OSD →
+#   "OSD data directory /var/lib/ceph/osd/ceph-7 does not exist; bailing out"
+
+# After restart, OSD goes through peering — may take 30-60s
+# During peering, PGs may show as "peering" or "activating"
+```
+
+## Monitoring Recovery
+
+```bash
+# Quick status
+ceph -s | grep -E "osd:|pgs:|recovery:|misplaced"
+
+# Detailed PG states
+ceph -s | grep -A15 "pgs:"
+
+# Per-OSD utilization (track if overfull OSD is draining)
+ceph osd df | grep "^ 7 "
+
+# Estimated duration: misplaced_objects / objects_per_second
+```
+
+## Recovery Monitoring Cron Pattern
+
+```
+Schedule: every 30 minutes
+Command: ssh to monitor node, run `ceph status`, report:
+  - PGs remapped/backfilling count
+  - % misplaced objects
+  - OSD 7 utilization
+  - Recovery speed
+  - Alert if any OSD down or health ERR
+  - Announce "COMPLETE" when 0 remapped PGs
+```
+
+## Post-Recovery Cleanup
+
+After all PGs return to active+clean:
+
+```bash
+# Restore default ratios
+ceph osd set-backfillfull-ratio 0.95
+ceph osd set-nearfull-ratio 0.93
+
+# Restore default backfill speed
+ceph config set osd osd_max_backfills 1
+ceph tell osd.* injectargs "--osd_max_backfills 1"
+
+# Restore OSD weight (if it was reduced)
+ceph osd crush reweight osd.7 1.0  # or original weight
+
+# Verify cluster health
+ceph health
+ceph -s
+```