Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,178 @@
+# Ceph PG Management & OSD Balancing
+
+## Reducing PG Count (pg_num) — Autoscale Mode Pitfall
+
+### Problem
+
+`ceph health` warns `too many PGs per OSD (259 > max 250)`. Reducing
+`pg_num` on pools with little or no data (e.g. `cephfs_data` with 0 B,
+`cephfs_metadata` with 85 KiB) should be straightforward, but the PG
+autoscaler silently reverts manual changes.
+
+### What Happens
+
+```bash
+# Attempt to reduce PGs:
+ceph osd pool set cephfs_data pg_num 32
+# → appears to succeed (exit 0)
+
+# Check:
+ceph osd pool get cephfs_data pg_num
+# → pg_num: 128  (UNCHANGED!)
+```
+
+The autoscaler (`pg_autoscale_mode on`) overrides manual `pg_num` changes.
+Even after setting `pg_autoscale_mode off`, the `pg_num` reduction may
+not take effect immediately — it only merges when all PGs are in
+`active+clean` state with no recovery/backfill in progress.
+
+### Correct Procedure
+
+```bash
+# 1. Disable autoscale for the pool
+ceph osd pool set cephfs_data pg_autoscale_mode off
+ceph osd pool set cephfs_metadata pg_autoscale_mode off
+
+# 2. Set pgp_num FIRST (controls placement), then pg_num
+ceph osd pool set cephfs_data pgp_num 32
+ceph osd pool set cephfs_data pg_num 32
+# PVE may accept pgp_num=32 but show pg_num_target=32 (deferred merge)
+
+# 3. Verify the target is set
+ceph osd pool ls detail | grep cephfs_data
+# Look for: pg_num_target 32
+
+# 4. The actual PG merge happens AFTER all recovery/backfill completes
+# Monitor: ceph pg dump pgs_brief | tail -n +2 | awk '{print $NF}' | sort | uniq -c
+# All PGs must be "active+clean" for the merge to proceed
+```
+
+### Increasing mon_max_pg_per_osd (Temporary Relief)
+
+If the PG merge is deferred (waiting for recovery), increase the
+threshold to suppress the warning:
+
+```bash
+# Set via config (may not propagate immediately to MONs)
+ceph config set mon mon_max_pg_per_osd 400
+
+# Force MONs to reload config (CRITICAL — config set alone doesn't always work)
+ceph tell mon.* injectargs --mon_max_pg_per_osd=400
+
+# Verify
+ceph config get mon mon_max_pg_per_osd
+ceph health
+```
+
+**Pitfall**: `ceph config set mon mon_max_pg_per_osd 400` alone may NOT
+propagate to running MONs. The health check still shows the old threshold.
+Use `ceph tell mon.* injectargs` to force a live config reload.
+
+## OSD Reweight vs Concurrent Writes
+
+### Scenario
+
+OSD.7 at 88% utilization. Applied `ceph osd reweight osd.7 0.65` to
+gradually redistribute data. Simultaneously, an RBD copy to an EC pool
+(`media_ec`) was running, writing new data to the same HDD OSDs.
+
+### Observation
+
+OSD.7 initially dropped to ~80% (reweight working), then climbed back
+to ~85% (new writes from RBD copy landing on OSD.7 via CRUSH).
+
+### Explanation
+
+Two opposing forces:
+1. **Reweight** moves existing PGs away from OSD.7 → utilization drops
+2. **New writes** to EC pools distribute chunks to all HDD OSDs including
+   OSD.7 → utilization rises
+
+The reweight only affects PG placement for EXISTING data. New writes
+follow CRUSH rules with the reduced weight, but the EC pool has its own
+crush rule and may still place chunks on OSD.7.
+
+### Takeaway
+
+When rebalancing an OSD, avoid concurrent heavy writes to pools that
+use the same OSD device class. The reweight will win AFTER the writes
+stop. Don't expect monotonic decrease during concurrent I/O.
+
+## Calculating PG Impact
+
+```bash
+# Total PG instances across all pools
+for pool in $(ceph osd pool ls); do
+  pgnum=$(ceph osd pool get $pool pg_num 2>/dev/null | awk '{print $2}')
+  size=$(ceph osd pool get $pool size 2>/dev/null | awk '{print $2}')
+  stored=$(ceph df | grep "^$pool " | awk '{print $3}')
+  echo "$pool: $pgnum PGs × size $size, stored: $stored"
+done
+
+# Approximate PGs per OSD: sum(pg_num × size) / num_osds
+# For 9 pools on 10 OSDs with mixed sizes:
+# 2627 PG instances / 10 OSDs ≈ 263 per OSD (>250 threshold)
+```
+
+Reducing empty/near-empty pools from 128→32 PGs saves:
+`(128-32) × size × 2 pools = 576 PG instances` → ~58 per OSD reduction.
+
+## EC4+1 Reweight Pitfall — CRITICAL
+
+### Never Reweight OSDs in EC Pools with Minimal OSD Count
+
+An EC4+1 pool requires exactly 5 OSDs (k=4, m=1). If the cluster has
+exactly 5 HDD OSDs, reweighting any of them causes CRUSH to place
+`2147483647` (sentinel = "no OSD available") in the PG up-set. This
+creates remapped PGs that can **never recover** — CRUSH has no
+alternative OSD to move data to.
+
+### Symptoms
+
+- `ceph status` shows ~11% objects misplaced, recovery at 0 MiB/s
+- Recovery settings tuned (`osd_max_backfills=5`, `osd_recovery_sleep=0`)
+  but recovery doesn't progress
+- `ceph pg dump pgs_brief` shows 100+ PGs in `active+clean+remapped`
+- Remapped PGs show `2147483647` in their up-set:
+  ```
+  8.30 active+clean+remapped [8,10,1,2147483647,6] 8
+  8.31 active+clean+remapped [6,1,8,2147483647,2147483647] 6
+  ```
+
+### Fix
+
+Reset all HDD OSD weights to 1.0:
+```bash
+ceph osd reweight osd.7 1.0
+ceph osd reweight osd.10 1.0
+```
+Within 60 seconds, remapped PGs begin recovering.
+
+### General Rule
+
+**Never reweight OSDs in EC pools unless OSD count > k+m.** EC(k=4, m=1)
+needs 5 OSDs — with exactly 5, all must participate at full weight. To
+balance an overly-full OSD:
+1. Add a 6th HDD OSD (gives CRUSH redistribution flexibility)
+2. Move data to a different pool
+3. Accept the imbalance — EC with minimal OSDs has no rebalancing slack
+
+### Diagnostic Flow for Stagnant Recovery
+
+1. `ceph pg dump pgs_brief | awk '{print $2}' | sort | uniq -c | sort -rn`
+   — look for `active+clean+remapped` count
+2. Inspect remapped PG up-sets — `2147483647` = CRUSH can't place
+3. `ceph osd erasure-code-profile get <profile>` — check k+m
+4. `ceph osd tree | grep hdd | wc -l` — compare OSD count vs k+m
+5. If OSD count == k+m: reset all weights to 1.0
+
+## Session Log — 2026-07-06 (cont.)
+
+- Reweighted osd.7 0.80→0.65 to relieve 88% utilization → caused 129
+  remapped PGs in `media_ec` (EC4+1, only 5 HDD OSDs)
+- Recovery stagnated at 11.5% misplaced for 7+ hours (0 MiB/s)
+- Fixed by resetting osd.7 and osd.10 to weight 1.0
+- Recovery resumed at 67 MiB/s after reset
+- `mon_max_pg_per_osd` raised to 400 (didn't propagate to MONs without
+  restart — left as-is, PG reduction is the proper fix)
+- PG merge (cephfs_data/metadata 128→32) still queued, awaits recovery