feat: add home-assistant-dashboard-conventions skill + update multiple skills

- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
2026-07-14 18:35:16 +00:00
parent e9cc106625
commit 01bd921ced
37 changed files with 6133 additions and 126 deletions
@@ -0,0 +1,144 @@
+# Ceph CRUSH Weight Optimization for Mixed OSD Sizes — 2026-07-13
+
+## Problem
+
+When a cluster has OSDs of different sizes (1 TB and 3 TB), CRUSH weights
+should be proportional to device size. But in practice, operators sometimes
+manually lower CRUSH weights on small/full OSDs to reduce incoming data —
+which creates a vicious cycle:
+
+1. Small OSD fills up → operator lowers CRUSH weight
+2. Lower weight → fewer new PGs assigned → old data stays
+3. OSD stays full → operator lowers weight further
+4. Large OSDs (especially new ones) remain underutilized
+
+## Diagnosis
+
+```bash
+# Show all HDD OSDs with weight, reweight, utilization, PG count
+ceph osd df tree | grep hdd
+
+# Key columns to compare:
+# WEIGHT  — CRUSH weight (should ≈ size_in_TB)
+# REWEIGHT — runtime reweight (should be 1.0 normally)
+# %USE    — utilization (should be roughly equal across OSDs)
+# PGS     — PG count (should be proportional to weight)
+```
+
+### Symptoms of misaligned weights
+
+| Signal | Meaning |
+|--------|---------|
+| WEIGHT << SIZE (e.g. 0.3 for 1 TB) | Artificially suppressed weight |
+| %USE varies wildly (27% vs 94%) | Data not distributed proportionally |
+| PGS count disproportional (150 vs 450) | CRUSH assigns PGs by weight, not size |
+| New large OSD barely fills (2% after hours) | Weight correct but recovery slow |
+
+## Solution
+
+### Step 1: Identify correct CRUSH weights
+
+Set CRUSH weight = device size in TiB (Ceph convention):
+
+| Device Size | Correct CRUSH Weight |
+|-------------|---------------------|
+| 1 TB (982 GiB) | ~0.96 |
+| 2 TB | ~1.82 |
+| 3 TB (2.8 TiB) | ~2.73 |
+| 3.6 TiB | ~3.64 |
+
+```bash
+# Check current weights
+ceph osd tree | grep hdd
+# Compare WEIGHT column to SIZE in: ceph osd df
+```
+
+### Step 2: Reset artificial weights (after recovery!)
+
+⚠️ **Timing matters**: If a rebalance is already running, changing CRUSH
+weights triggers ADDITIONAL remapping. Either:
+- Wait for current rebalance to finish, THEN adjust weights, OR
+- Adjust weights now and accept a longer combined rebalance
+
+```bash
+# Reset CRUSH weight to match device size
+ceph osd crush reweight osd.7 0.96   # was 0.50, should be ~0.96 for 1 TB
+ceph osd crush reweight osd.10 0.96  # was 0.30, should be ~0.96 for 1 TB
+
+# Also reset runtime reweight to 1.0 (if it was lowered for emergency drain)
+ceph osd reweight 7 1.0
+ceph osd reweight 10 1.0
+```
+
+### Step 3: Let the upmap balancer handle fine-tuning
+
+The `upmap` balancer module optimizes PG placement without full remapping:
+
+```bash
+# Check balancer status
+ceph balancer status
+
+# If "no_optimization_needed": false, the balancer is actively working
+# If "optimize_result" mentions "too many objects misplaced": wait for
+# current recovery to drop below 5% misplaced, then balancer kicks in
+```
+
+The balancer cannot run while >5% objects are misplaced (it skips to avoid
+compounding recovery load). Once the initial rebalance from weight changes
+completes, the balancer will fine-tune PG distribution automatically.
+
+### Step 4: Optional — `reweight-by-utilization` for dynamic balancing
+
+```bash
+# Automatically reweight OSDs based on utilization (temporary reweights)
+ceph osd reweight-by-utilization
+# Or with custom threshold:
+ceph osd test-reweight-by-utilization 120  # dry run, shows what would change
+ceph osd reweight-by-utilization 120       # apply (120 = 1.2x average = overfull)
+```
+
+⚠️ Use cautiously — this changes runtime reweights, not CRUSH weights.
+The effect is temporary and can interact with ongoing recovery.
+
+## Mixed-Size Cluster Example
+
+Real-world cluster with 1 TB and 3 TB HDDs:
+
+| OSD | Size | Old Weight | New Weight | Old %Used | Expected %Used |
+|-----|------|-----------|------------|-----------|----------------|
+| osd.1 | 3.6 TiB | 3.64 | 3.64 (OK) | 33% | ~33% |
+| osd.6 | 2.8 TiB | 2.76 | 2.76 (OK) | 27% | ~27% |
+| osd.8 | 2.8 TiB | 2.73 | 2.73 (OK) | 27% | ~27% |
+| osd.11 | 2.8 TiB | 2.76 | 2.76 (OK) | 2% | ~27% (filling) |
+| osd.7 | 982 GiB | **0.50** | **0.96** | 94% | ~33% |
+| osd.10 | 982 GiB | **0.30** | **0.96** | 57% | ~33% |
+
+After resetting osd.7 and osd.10 to their true weights, CRUSH will assign
+them proportionally more PGs, and data will distribute evenly across all
+HDDs regardless of size. The 3 TB OSDs will naturally hold ~3× the data
+of 1 TB OSDs.
+
+## Pitfalls
+
+### Don't lower weight to "protect" a full OSD
+
+Lowering CRUSH weight on a full OSD prevents new PGs from landing there,
+but does NOT move existing data away. The OSD stays full. Meanwhile, the
+reduced weight means the OSD contributes less to the cluster's apparent
+capacity, causing `nearfull`/`backfillfull` warnings on remaining OSDs.
+
+Instead: use `ceph osd reweight` (runtime, temporary) for emergency drain,
+then fix the root cause (add capacity, redistribute, or accept the OSD
+size and set proper CRUSH weight).
+
+### Balancer won't help during active recovery
+
+The `upmap` balancer skips when >5% objects are misplaced. Don't expect
+it to optimize placement while a major rebalance is in progress. Wait for
+recovery to complete, then let the balancer run.
+
+### CRUSH weight changes trigger remapping
+
+Every `ceph osd crush reweight` causes CRUSH to recompute PG placements.
+On a cluster with existing data, this triggers backfill. Schedule weight
+changes during maintenance windows or combine with planned capacity additions.
@@ -0,0 +1,248 @@
+# Ceph EC Pool Optimization for Mixed HDD Sizes — 2026-07-13
+
+## Problem
+
+When a Ceph cluster has HDDs of different sizes (1 TB and 3 TB) and uses
+erasure coding, the 3 TB drives are severely underutilized. This is because
+EC stores equally-sized chunks per PG on each OSD — regardless of OSD
+capacity. A 1 TB OSD and a 3 TB OSD in the same PG each store 1/(k+m) of
+the object.
+
+### Why `ceph osd reweight` Doesn't Help for EC
+
+With EC k=4, m=1 (5 OSDs per PG) and 6 HDD OSDs total, CRUSH selects 5 of 6.
+`ceph osd reweight` only changes the probability of being selected — but
+with 5/6 selection, even a heavily downweighted OSD is chosen in most PGs.
+The effect is marginal compared to replicated pools where each OSD
+independently participates.
+
+### Symptoms
+
+| Signal | Cause |
+|--------|-------|
+| 3 TB OSD at 2% while 1 TB OSD at 94% | Equal chunk sizes fill small OSDs first |
+| Same PG count on 3 TB and 1 TB OSDs | CRUSH weight not proportional, or EC masking the effect |
+| `backfill_toofull` on small OSDs | Small OSDs hit full ratio before large ones fill |
+| `ceph osd reweight` has no visible effect | EC selection ratio too tight (5/6) |
+
+## Architectural Solution
+
+Separate large and small HDDs into different device classes, place them
+under a dedicated CRUSH root, and create an EC pool that only uses the
+large HDDs with a smaller k+m profile.
+
+### New CRUSH Topology
+
+```
+root default                    root hdd-root (NEW)
+├── host proxmox2-5 (SSDs)       ├── host proxmox1    → osd.1  (hdd-large, 3.6 TB)
+│                               ├── host ubuntu       → osd.6  (hdd-large, 2.8 TB)
+│                               │                   └ → osd.8  (hdd-large, 2.8 TB)
+│                               ├── host n5pro        → osd.11 (hdd-large, 2.8 TB)
+│                               ├── host proxmox6     → osd.10 (hdd-small, 1 TB)
+│                               └── host proxmox7     → osd.7  (hdd-small, 1 TB)
+```
+
+### Key Insight: `take` with vs without Class Filter
+
+- `take hdd-root` (no class filter) → selects ALL HDDs (large + small) → for replicated pools
+- `take hdd-root~hdd-large` → selects only large HDDs → for EC pool
+
+This allows large HDDs to serve both EC and replicated pools, while small
+HDDs only participate in replicated pools.
+
+### New EC Profile
+
+| Parameter | Old (ec41-hdd) | New (ec31-hdd-large) |
+|-----------|----------------|---------------------|
+| k | 4 | **3** |
+| m | 1 | **1** |
+| OSDs per PG | 5 | **4** |
+| Overhead | 20% | **33%** |
+| Available OSDs | 6 (2 too small) | **4 (all fully utilized)** |
+| Failure tolerance | 1 OSD | 1 OSD |
+
+Trade-off: EC overhead increases from 20% → 33%, but all 4 large OSDs are
+fully utilized. Net usable capacity is HIGHER because the 3 TB drives
+actually fill up instead of sitting at 2%.
+
+## Migration Plan (6 Phases)
+
+### Phase 0: Wait for Current Rebalance
+
+```bash
+ceph -s  # until "misplaced 0%" and "recovery: 0 B/s"
+```
+
+### Phase 1: CRUSH Structure (Triggers Remapping)
+
+```bash
+# 1a. Device classes
+ceph osd crush set-device-class hdd-large osd.1 osd.6 osd.8 osd.11
+ceph osd crush set-device-class hdd-small osd.7 osd.10
+
+# 1b. New root
+ceph osd crush add-bucket hdd-root root
+
+# 1c. Move HDD hosts to hdd-root (triggers remapping!)
+ceph osd crush move proxmox1 root=hdd-root
+ceph osd crush move ubuntu root=hdd-root
+ceph osd crush move n5pro root=hdd-root
+ceph osd crush move proxmox6 root=hdd-root
+ceph osd crush move proxmox7 root=hdd-root
+```
+
+⚠️ Step 1c triggers IMMEDIATE remapping of all HDD PGs. For gentler
+migration: move hosts one at a time with 30-min intervals between each,
+waiting for rebalance to stabilize.
+
+### Phase 2: New Rules + EC Profile
+
+```bash
+# 2a. New EC profile (k=3, m=1, large HDDs only)
+ceph osd erasure-code-profile set ec31-hdd-large \
+  k=3 m=1 \
+  crush-root=hdd-root \
+  crush-device-class=hdd-large \
+  crush-failure-domain=osd \
+  plugin=jerasure technique=reed_sol_van
+
+# 2b. New EC rule
+ceph osd crush rule create-erasure media_ec_large ec31-hdd-large
+
+# 2c. New replicated rule (ALL HDDs, no class filter)
+ceph osd crush rule create-replicated replicated_hdd_all hdd-root host
+```
+
+### Phase 3: Switch Replicated Pools
+
+```bash
+ceph osd pool set hdd_disk crush_rule replicated_hdd_all
+ceph osd pool set rbd crush_rule replicated_hdd_all
+ceph osd pool set tm_disks crush_rule replicated_hdd_all
+```
+
+### Phase 4: EC Pool Migration (Critical)
+
+EC pools cannot change their k/m profile. Must create a new pool and
+migrate data.
+
+#### RBD Online Migration (preferred — VMs keep running)
+
+```bash
+# 4a. Create new EC pool
+ceph osd pool create media_ec_new 128 128 erasure ec31-hdd-large
+ceph osd pool set media_ec_new crush_rule media_ec_large
+ceph osd pool set media_ec_new ec_overwrites
+ceph osd pool application enable media_ec_new rbd
+
+# 4b. Create new metadata pool
+ceph osd pool create media_meta_new 32
+ceph osd pool set media_meta_new crush_rule replicated_ssd
+
+# 4c. Migrate each RBD image (online, VM keeps running)
+for img in vm-114-disk-0 vm-136-disk-0 vm-137-disk-0 vm-138-disk-1 vm-143-disk-0; do
+  rbd migration prepare media_meta/${img} media_meta_new/${img} \
+    --data-pool media_ec_new
+  rbd migration execute media_meta_new/${img}
+  rbd migration commit media_meta_new/${img}
+done
+```
+
+Per-image timeline: `prepare` (instant, starts copy) → `execute` (copies
+data, 30-60 min per 500 GiB image) → `commit` (atomic switch, old image
+deleted). VMs continue running throughout — the migration is transparent.
+
+#### Offline Alternative (faster, VM stopped)
+
+Stop VM, `rbd copy --data-pool media_ec_new old_pool/img new_meta_pool/img`,
+update VM disk config, start VM. See `rbd-pool-migration-2026-07.md`.
+
+### Phase 5: Cleanup
+
+```bash
+ceph osd pool delete media_ec --yes-i-really-really-mean-it
+ceph osd crush rule rm media_ec
+ceph osd crush rule rm replicated_hdd
+ceph osd crush rm proxmox       # empty host bucket
+ceph osd crush rm px-tmp20      # empty host bucket
+ceph osd erasure-code-profile rm ec41-hdd
+ceph osd pool rename media_ec_new media_ec
+ceph osd pool rename media_meta_new media_meta
+```
+
+### Phase 6: Correct CRUSH Weights
+
+```bash
+# Now safe — small HDDs no longer in EC pool
+ceph osd crush reweight osd.7 0.96   # was 0.50, should be ~0.96 for 1 TB
+ceph osd crush reweight osd.10 0.96  # was 0.30, should be ~0.96 for 1 TB
+```
+
+## Capacity Analysis
+
+| Pool | Raw Capacity | Usable | Currently Used | Free |
+|------|-------------|--------|----------------|------|
+| media_ec (EC 3+1) | 4× ~2.9 TB = 11.6 TB | 8.7 TB | 764 GiB | ~7.9 TB |
+| hdd_disk (Repl 3) | 15 TB | 5.0 TB | 1.0 TiB | ~4.0 TB |
+
+## Safety-Net
+
+- Backup CRUSH map: `ceph osd getcrushmap -o /tmp/crushmap.backup`
+- RBD snapshots of critical images before migration
+- Rollback: restore CRUSH map, revert pool rules
+
+## Pitfalls
+
+### EC Pools Cannot Change k/m In Place
+
+An EC pool's erasure-code profile is fixed at creation. Changing k or m
+requires creating a new pool and migrating data. Do NOT attempt
+`ceph osd pool set media_ec erasure-code-profile ec31-hdd-large` — it
+will fail or corrupt data.
+
+### `rbd migration prepare` Needs Both Pools
+
+The `prepare` command takes `source_image dest_image` where dest is in
+the new metadata pool. The `--data-pool` flag specifies where the EC data
+chunks go. The metadata pool stores RBD headers (small, replicated).
+
+### Moving Hosts Between Roots Triggers Full Remap
+
+Every `ceph osd crush move` changes the CRUSH hash inputs for all PGs in
+that subtree. Moving 5 hosts at once causes massive simultaneous remapping.
+Move hosts individually with rebalance gaps for production clusters.
+
+### Device Class Must Match Reality
+
+`ceph osd crush set-device-class hdd-large osd.X` overrides the
+auto-detected class. If the OSD is later recreated, it reverts to
+auto-detected `hdd`. Re-apply the custom class after any OSD rebuild.
+
+## Gitea Issue Creation Pattern
+
+This plan was saved as Gitea issue #14 in `dominik/iac-homelab`. Pattern
+for large issue bodies:
+
+```bash
+# 1. Write JSON to temp file locally
+cat > /tmp/issue.json << 'EOF'
+{"title":"...", "body":"...markdown...", "labels":[]}
+EOF
+
+# 2. SCP to Gitea host (CT108 on proxmox6)
+scp -i ~/.ssh/id_ed25519_proxmox /tmp/issue.json root@10.0.30.105:/tmp/
+
+# 3. POST via curl on the Gitea host (avoids remote curl auth issues)
+ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.105 \
+  "curl -s -X POST -H 'Authorization: token TOKEN' \
+   -H 'Content-Type: application/json' \
+   -d @/tmp/issue.json \
+   http://localhost:3000/api/v1/repos/dominik/iac-homelab/issues"
+```
+
+Using a file (`-d @file`) avoids shell escaping issues with large markdown
+bodies containing backticks, quotes, and special characters.
+
+See also: `references/gitea-api-and-skills-versioning-2026-07.md` for
+token generation and full API patterns.
@@ -89,6 +89,43 @@ for img in vm-104-disk-0 vm-104-disk-1 vm-108-disk-0 ...; do
 done
 ```

+### ⚠️ Protected Snapshot Deletion Order (2026-07-12)
+
+Base images (`base-NNN-disk-X`) often have a protected `__base__` snapshot
+(leftover from PVE template cloning). `rbd rm` FAILS with:
+
+```
+rbd: image has snapshots - these must be deleted with 'rbd snap purge' before the image can be removed.
+```
+
+And `rbd snap purge` FAILS if the snapshot is protected:
+
+```
+rbd: cannot remove protected snapshot __base__
+```
+
+**Correct deletion order:**
+
+```bash
+# 1. Unprotect the snapshot FIRST
+rbd snap unprotect "hdd_disk/base-101-disk-0"@__base__
+
+# 2. Purge ALL snapshots
+rbd snap purge "hdd_disk/base-101-disk-0"
+# → "Removing all snapshots: 100% complete...done."
+
+# 3. Remove the image
+rbd rm "hdd_disk/base-101-disk-0" --no-progress
+```
+
+⚠️ There is NO `--force` flag for `rbd snap purge` in older Ceph versions.
+The unprotect step is mandatory and must succeed before purge.
+
+⚠️ Under Ceph backfill load, `rbd rm` may hang for 60s+ — the image header
+is removed immediately but object deletion happens asynchronously. Use
+`terminal(background=true, notify_on_complete=true)` for batch deletions
+to avoid SSH timeouts.
+
 ## Classification Rules

 | Image Pattern | Check | Safe to Delete? |
@@ -224,6 +261,70 @@ VMs 201, 202, 203 existed on n5pro (10.0.20.91, non-Ceph node) and VM 116 on pro
 - VM 116: 192 GB ext4 data disk, no partition table (likely MariaDB-01)
 - Total freed: ~920 GB thin-provisioned

+## Fast Orphan Verification: `/etc/pve/` Grep (O(1))
+
+When you need to check if a **specific** image (e.g. `vm-143-disk-0`) is
+referenced by ANY VM/CT config, do NOT iterate all 56 guests with `pvesh`.
+Instead, grep the PVE config filesystem directly:
+
+```bash
+# From any PVE node — /etc/pve is a shared FUSE filesystem (pmxcfs)
+grep -rl "vm-143" /etc/pve/qemu-server/ /etc/pve/lxc/ 2>/dev/null
+# Exit 0 + filename = referenced. Exit 1 = orphan (not referenced anywhere).
+```
+
+This is **dramatically faster** than the serial `pvesh` approach:
+- Serial `pvesh get /nodes/{node}/{type}/{vmid}/config` for 56 guests takes
+  >120s and often times out.
+- `grep -rl` on `/etc/pve/` returns in <2s because pmxcfs is local memory.
+
+Works for any image pattern: `vm-NNN`, `base-NNN`, `csi-vol-xxx`, `test-dummy`.
+
+## Full Cluster Disk-Storage Audit Technique
+
+To produce a complete table of all guests with their disk storage assignments:
+
+```bash
+ssh root@10.0.20.10 'bash -s' << 'SCRIPT'
+pvesh get /cluster/resources --type vm --output-format json 2>/dev/null | python3 -c "
+import sys, json, subprocess
+
+guests = json.load(sys.stdin)
+guests.sort(key=lambda g: g['vmid'])
+
+for g in guests:
+    vmid = g['vmid']
+    vtype = g.get('type','?')
+    node = g.get('node','?')
+
+    if vtype == 'qemu':
+        cmd = ['pvesh', 'get', f'/nodes/{node}/qemu/{vmid}/config', '--output-format', 'json']
+    else:
+        cmd = ['pvesh', 'get', f'/nodes/{node}/lxc/{vmid}/config', '--output-format', 'json']
+
+    try:
+        cfg_raw = subprocess.run(cmd, capture_output=True, text=True, timeout=5).stdout
+        cfg = json.loads(cfg_raw) if cfg_raw.strip() else {}
+    except:
+        cfg = {}
+
+    storages = []
+    for key, val in cfg.items():
+        if key.startswith(('scsi','virtio','ide','sata','nvme','rootfs','mp')):
+            if isinstance(val, str) and ':' in val:
+                volid = val.split(',')[0]
+                parts = volid.split(':')
+                if len(parts) >= 2 and parts[0] not in ('none','cloud-init'):
+                    storages.append(parts[0])
+    stores = list(dict.fromkeys(storages))
+    print(f'{vmid:>6} | {vtype:4} | {node:10} | {\", \".join(stores) or \"none\"}')
+"
+SCRIPT
+```
+
+⚠️ Needs `timeout=180` on the SSH call — 56 guests × 5s per-config timeout
+can exceed 120s. Split into batches of 20 if timeout is constrained.
+
 ## Pitfalls

 1. **`rbd du` timeouts** — On pools with many images, `rbd du` can hang. Use per-image `rbd info` instead, but batch to avoid SSH timeouts.
@@ -237,3 +338,10 @@ VMs 201, 202, 203 existed on n5pro (10.0.20.91, non-Ceph node) and VM 116 on pro
 9. **Stale RBD mappings block image deletion** — A previous `rbd map` that wasn't cleaned up (or a crashed QEMU) keeps watchers on the image. Check `rbd showmapped` on ALL nodes, force unmap, then retry removal.
 10. **Always unmap RBD images after inspection** — Mounted RBD images create watchers that prevent later deletion. `umount` + `rbd unmap` after every inspection session.
 11. **Mount can hang under heavy backfill** — When Ceph recovery I/O is high, `mount /dev/rbdN` can hang for 30s+. Abort and infer from disk structure if needed.
+12. **`pvesm list` fails when pool is backfillfull** (2026-07-12) — `pvesm list hdd_disk` returns empty with `rbd error: (2) No such file or directory` when the Ceph pool is in `backfillfull` state. Similarly, `pvesh get /nodes/{node}/storage/hdd_disk/content` also fails. In this scenario, fall back to direct `rbd ls <pool>` (which still works) for image listing, and use `pvesh get /cluster/resources --type vm` for VM/CT existence checks. The PVE storage layer cannot enumerate RBD images when Ceph blocks I/O.
+13. **Ghost RBD images with empty `rbd info`** (2026-07-12) — `vm-201-disk-0` and `vm-202-disk-0` appear in `rbd ls` but `rbd info` returns empty output (even with `--format json`). This is the "ghost RBD" state described in the main skill body — header is gone, data objects may remain. These need `rados rm` purge, not `rbd rm`.
+14. **`execute_code` blocked for SSH-heavy scripts** (2026-07-12) — When orphan detection requires many sequential SSH calls (one per image for `rbd info`), `execute_code` may be blocked by approval policy. Use `terminal` with heredoc scripts instead: `ssh root@host 'bash -s' << 'SCRIPT' ... SCRIPT`. Batch all `rbd info` calls in a single SSH session to avoid per-call overhead.
+15. **Serial `pvesh` config retrieval for 56 guests exceeds 120s** (2026-07-12) — Iterating all guests with `pvesh get /nodes/{node}/{type}/{vmid}/config` one-by-one (even with `timeout=5` per call) takes >120s for 56 guests and times out. Solutions: (a) increase SSH timeout to 180s+, (b) split into batches of 20, or (c) for specific-image checks use `grep -rl "vm-NNN" /etc/pve/qemu-server/ /etc/pve/lxc/` instead (O(1), <2s). See "Fast Orphan Verification" section above.
+16. **SSH key path typo** (2026-07-12) — `id_ed25519_prozmox` vs `id_ed25519_proxmox` — transposing `x` and `m` in the key filename causes a silent fallback to password auth which then fails with "Permission denied (publickey,password)". Always double-check the key path: `~/.ssh/id_ed25519_proxmox`.
+17. **CSI volume with stale watchers** (2026-07-12) — `csi-vol-e596e771-...` on hdd_disk had a watcher from IP 10.0.30.63 (K8s pod VLAN) but `kubectl get pv,pvc` showed no RBD PVCs. The watcher is a stale client from a crashed/evicted pod. `rbd rm` fails with "image still has watchers". Resolution: wait 30s for the watcher to timeout, or identify and kill the stale ceph client. Do NOT force-delete — may corrupt data if a pod reattaches.
+18. **`rbd rm` succeeds but image lingers during backfill** (2026-07-12) — When Ceph is in backfill state, `rbd rm` may return exit 0 but the image still appears in `rbd ls` for several seconds. The header is removed but object cleanup is deferred. Check with `rbd info` (returns "No such file or directory" once truly gone) rather than `rbd ls` for definitive verification.
@@ -0,0 +1,324 @@
+# Creating a Ceph OSD via Hot-Plug on Non-PVE Nodes (n5pro / Minisforum)
+
+## When to Use
+
+Adding a new HDD OSD to a Ceph cluster by physically hot-plugging a SATA
+drive into a running node that is not a standard PVE host (e.g. Minisforum
+N5 PRO with JMB58x SATA controller, CRUSH host `n5pro`).
+
+## Prerequisites Checklist
+
+- [ ] Drive is physically inserted (hot-plug or powered-off install)
+- [ ] Node has `ceph`, `ceph-volume`, `pveceph` installed
+- [ ] Node has `/etc/ceph/ceph.conf` and `ceph.client.admin.keyring`
+- [ ] `/etc/ceph/ceph.conf` is a **symlink to `/etc/pve/ceph.conf`**
+
+## Step-by-Step
+
+### 1. Detect the Hot-Plugged Drive
+
+If the drive was inserted while the node was running, the SATA bus may
+not auto-scan. Trigger a manual rescan:
+
+```bash
+for h in /sys/class/scsi_host/host{0,1,2,3,4}; do
+  echo "- - -" > ${h}/scan
+done
+sleep 2
+lsblk -d -o NAME,MODEL,SERIAL,SIZE,TRAN,ROTA,STATE | grep -v rbd | grep -v loop
+```
+
+Verify the drive appears (e.g. `/dev/sdb`, MODEL=WDC WD30EFZX).
+
+### 2. Copy Ceph Config to Non-PVE Node (if missing)
+
+If the node lacks `/etc/ceph/ceph.conf` or the admin keyring:
+
+```bash
+# From PVE master (10.0.20.10), copy to target node:
+scp /etc/ceph/ceph.conf root@<node>:/etc/ceph/
+scp /etc/ceph/ceph.client.admin.keyring root@<node>:/etc/ceph/
+ssh root@<node> 'chmod 644 /etc/ceph/ceph.conf && chmod 600 /etc/ceph/ceph.client.admin.keyring'
+```
+
+### 3. Fix ceph.conf Symlink (CRITICAL — pveceph requires it)
+
+`pveceph osd create` refuses to run if `/etc/ceph/ceph.conf` is a regular
+file rather than a symlink to `/etc/pve/ceph.conf`:
+
+```
+file '/etc/ceph/ceph.conf' already exists and is not a symlink to /etc/pve/ceph.conf
+```
+
+Fix:
+
+```bash
+rm /etc/ceph/ceph.conf
+ln -s /etc/pve/ceph.conf /etc/ceph/ceph.conf
+```
+
+### 4. Zap the Drive Clean
+
+Remove any leftover partitions/filesystem signatures from prior use:
+
+```bash
+sgdisk --zap-all /dev/sdX
+wipefs -a /dev/sdX
+parted /dev/sdX mklabel gpt
+lsblk /dev/sdX -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT  # verify clean
+```
+
+### 5. Create the OSD
+
+```bash
+pveceph osd create /dev/sdX
+```
+
+This runs `ceph-volume lvm create` internally: creates a VG + LV on the
+disk, formats BlueStore, registers with the MON, enables systemd services
+(`ceph-volume@lvm-<id>-<uuid>.service`, `ceph-osd@<id>.service`), and
+starts the OSD.
+
+### 6. Wait for OSD to Come Up (10-15 seconds)
+
+The newly created OSD will initially show as **down** in `ceph osd tree`.
+The `ceph-osd@<id>` service is running but waits for the OSD map to load.
+Wait ~15 seconds, then verify:
+
+```bash
+ceph osd tree | grep osd.<id>
+# Should show: up   1.00000
+```
+
+If still down after 30s, check:
+
+```bash
+journalctl -u ceph-osd@<id> --no-pager -n 30
+systemctl status ceph-osd@<id>
+```
+
+Common non-fatal log lines:
+- `failed to load OSD map for epoch N, got 0 bytes` — transient, resolves
+- `set_numa_affinity unable to identify public interface` — cosmetic
+
+### 7. Verify Cluster Integration
+
+```bash
+ceph osd crush get-device-class osd.<id>  # should report 'hdd'
+ceph -s                                   # OSD count increased
+ceph df                                   # raw capacity increased
+```
+
+Rebalancing begins automatically — expect a spike in `remapped_pgs` (normal).
+
+## Pitfalls
+
+### pveceph osd create fails: "already exists and is not a symlink"
+
+**Symptom**: `pveceph osd create /dev/sdb` exits with code 22 and the
+message `file '/etc/ceph/ceph.conf' already exists and is not a symlink`.
+
+**Cause**: On non-standard PVE nodes (e.g. n5pro), `/etc/ceph/ceph.conf`
+was manually placed as a regular file. `pveceph` insists on it being a
+symlink to `/etc/pve/ceph.conf`.
+
+**Fix**: Remove the file and create the symlink (step 3 above).
+
+### Hot-Plugged SATA Drive Not Detected
+
+**Symptom**: Drive physically inserted but `lsblk` doesn't show it.
+
+**Cause**: The JMB58x AHCI controller does not expose `hot_plug` sysfs
+attribute, and Linux may not auto-scan the port.
+
+**Fix**: Manual SCSI host rescan (step 1 above). The JMB58x does support
+AHCI hot-plug electrically even though the sysfs attribute is absent.
+
+### OSD Shows "down" Immediately After Creation
+
+**Symptom**: `ceph osd tree` shows the new OSD as `down` with weight 0.
+
+**Cause**: The `ceph-osd` daemon takes 10-15 seconds to load the OSD map
+and register with the MONs. This is normal startup latency, not a failure.
+
+**Fix**: Wait. Check `systemctl status ceph-osd@<id>` — if it says
+`active (running)`, the OSD will come up shortly.
+
+### Bootstrap Keyring Warning During ceph-volume
+
+**Symptom**: `unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring`
+
+**Cause**: The bootstrap-osd keyring lives elsewhere on some nodes. The
+warning is non-fatal — `ceph-volume` falls back to the admin keyring.
+
+**Fix**: None needed. OSD creation succeeds despite this warning.
+
+## Hardware Notes: Minisforum N5 PRO
+
+| Spec | Value |
+|------|-------|
+| SATA Controller | JMicron JMB58x AHCI, 5 ports |
+| AHCI Version | 1.0301, 32 command slots, 6 Gbps |
+| AHCI Flags | ncq sntf stag pm led clo pmp fbs pio slum part ccc apst boh |
+| Hot-Plug sysfs | `hot_plug` attribute NOT exposed |
+| Actual Hot-Plug | Works — bus rescan detects the drive |
+| CRUSH Host Name | `n5pro` (own bucket, NOT `ubuntu`) |
+| Existing OSDs | NONE — n5pro starts with zero OSDs |
+| NVMe | AirDisk 128GB SSD (boot + LVM VG `pve` with WAL/DB LVs) |
+
+> **CRITICAL — n5pro ≠ ubuntu**: Host `ubuntu` (IP 10.0.30.100) is a SEPARATE
+> machine with its own OSDs (osd.6, osd.8). n5pro (IP 10.0.20.91) must have
+> its own CRUSH host bucket named `n5pro`. Do NOT place n5pro OSDs under
+> `ubuntu` — this would falsely suggest geographic separation for replicas
+> that are actually on the same physical node.
+
+## Adding SSD WAL/DB to an OSD on Non-PVE Nodes
+
+When the OSD's HDD is on a node that also has an NVMe SSD, placing the
+BlueStore DB (and collocated WAL) on the SSD dramatically improves write
+performance. The n5pro has a 128 GB AirDisk NVMe (`/dev/nvme0n1`) with
+an LVM VG `pve` that can host WAL/DB LVs.
+
+### Creating WAL/DB LVs on the NVMe
+
+```bash
+# Check free space in the pve VG
+vgs pve   # look at VFree
+
+# Create 30 GB LV for WAL/DB (one per OSD)
+lvcreate -L 30G -n wal-db-osd-a pve
+# Optional second LV for a future OSD:
+lvcreate -L 30G -n wal-db-osd-b pve
+```
+
+### ⚠️ pveceph Cannot Use LVM Devices for WAL
+
+`pveceph osd create /dev/sdb --wal_dev /dev/pve/wal-db-osd-a` FAILS:
+
+```
+unable to get device info for '/dev/dm-2' for type wal_dev
+```
+
+pveceph expects a raw block device, not a device-mapper LV. Use
+`ceph-volume` directly instead.
+
+### ⚠️ BlueStore Cannot Share One LV For Both WAL and DB Separately
+
+Specifying `--block.wal` AND `--block.db` pointing to the SAME LV fails:
+
+```
+bdev(...) open got: (16) Device or resource busy
+bluestore _minimal_open_bluefs add block device(block.wal) returned: (16) Device or resource busy
+OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
+```
+
+BlueStore opens the WAL device exclusively — it cannot be the same
+underlying device as DB. The rollback also destroys the WAL LV.
+
+### ✅ Correct Method: ceph-volume with --block.db Only
+
+Specify ONLY `--block.db` — BlueStore collocates WAL on the DB device
+automatically. This is the standard approach for a single shared SSD:
+
+```bash
+# 1. Ensure the WAL/DB LV exists
+lvcreate -L 30G -n wal-db-osd-a pve
+
+# 2. Prepare the OSD with DB on NVMe (WAL collocated)
+ceph-volume lvm prepare --bluestore \
+  --data /dev/sdb \
+  --block.db /dev/pve/wal-db-osd-a
+
+# 3. Activate the OSD (replace UUID from prepare output)
+ceph-volume lvm activate <osd_id> <osd_uuid>
+
+# 4. Verify DB/WAL placement
+ceph-volume lvm list <osd_id>
+# Should show:
+#   [block]     /dev/ceph-<vg>/osd-block-<uuid>   (data on HDD)
+#   [db]        /dev/pve/wal-db-osd-a              (DB on NVMe)
+#   db device   /dev/pve/wal-db-osd-a
+#   devices     /dev/nvme0n1p3
+```
+
+### Destroy and Recreate an OSD With WAL
+
+If an OSD was already created WITHOUT WAL and needs to be rebuilt:
+
+```bash
+# 1. Stop the OSD service
+systemctl stop ceph-osd@<id>
+
+# 2. Mark down and destroy
+ceph osd down <id>
+ceph osd destroy <id> --yes-i-really-mean-it
+
+# 3. Zap the disk (removes VG, LV, BlueStore signatures)
+ceph-volume lvm zap /dev/sdX --destroy
+
+# 4. Recreate with WAL/DB as above
+```
+
+### CRUSH Host Assignment for Non-PVE Nodes
+
+When using `ceph-volume` (not `pveceph`), the OSD may land in the wrong
+CRUSH host or under a generic bucket. Verify and fix:
+
+```bash
+# Check which host the OSD landed in
+ceph osd tree | grep osd.<id>
+
+# If in wrong host, move it:
+ceph osd crush set osd.<id> <weight> root=default host=<correct_hostname>
+
+# Set device class if not auto-detected
+ceph osd crush set-device-class hdd osd.<id>
+
+# Remove old destroyed OSD from CRUSH
+ceph osd crush remove osd.<old_id>
+ceph osd rm <old_id>
+```
+
+## Session Log — 2026-07-13
+
+### Phase 1: Initial OSD Creation (osd.9, no WAL)
+
+Added osd.9 (WD Red Plus 3 TB, WD-WX22DA0D7EP3) to n5pro via hot-plug.
+Drive was removed from CT 10.0.30.100 and physically inserted into n5pro
+while running. Bus rescan detected it on ata5. Created as BlueStore OSD
+with `pveceph osd create /dev/sdb`. Cluster went from 10→11 OSDs.
+`backfill_toofull` PGs dropped from 9→1 immediately. Remapped PGs spiked
+to 317 (expected rebalance). Device class: hdd.
+
+### Phase 2: CRUSH Host Correction
+
+**Mistake**: Initially moved osd.6+osd.8 from host `ubuntu` to `n5pro`,
+assuming they were on the same machine. **User corrected**: osd.6 and
+osd.8 belong to host `ubuntu` (10.0.30.100), a separate physical machine.
+n5pro (10.0.20.91) only hosts osd.9 (later osd.11).
+
+Fixed by recreating the `ubuntu` bucket and moving osd.6+osd.8 back:
+```bash
+ceph osd crush add-bucket ubuntu host
+ceph osd crush move ubuntu root=default
+ceph osd crush set osd.6 2.75829 root=default host=ubuntu
+ceph osd crush set osd.8 2.72899 root=default host=ubuntu
+```
+
+### Phase 3: Rebuild with SSD WAL/DB (osd.9 → osd.11)
+
+Destroyed osd.9 (created without WAL) and rebuilt as osd.11 with DB on
+NVMe SSD. Key steps:
+1. `systemctl stop ceph-osd@9` → `ceph osd destroy 9` → `ceph-volume lvm zap /dev/sdb --destroy`
+2. Attempted `ceph-volume lvm prepare --bluestore --data /dev/sdb --block.wal /dev/pve/wal-db-osd-a --block.db /dev/pve/wal-db-osd-a` → FAILED (device busy, same LV for WAL+DB)
+3. Rollback deleted `wal-db-osd-a` LV — had to recreate with `lvcreate -L 30G -n wal-db-osd-a pve`
+4. Succeeded with `--block.db /dev/pve/wal-db-osd-a` only (WAL collocated on DB)
+5. Activated, verified DB on NVMe, set CRUSH host to `n5pro`, device class `hdd`
+6. Removed old osd.9 from CRUSH: `ceph osd crush remove osd.9` + `ceph osd rm 9`
+
+Final osd.11: data on /dev/sdb (3 TB HDD), DB+WAL on /dev/pve/wal-db-osd-a
+(30 GB NVMe LV), CRUSH host `n5pro`, weight 2.76, device class hdd.
+
+Related: `references/hdd-selection-for-osd-2026-07.md` (drive selection),
+`references/ceph-pg-osd-management-2026-07.md` (rebalancing),
+`references/ceph-recovery-acceleration-2026-07.md` (recovery tuning).
@@ -54,6 +54,25 @@ ceph daemon osd.7 config get osd_max_backfills
 # Must return "3" — if still "1", the config hasn't been applied.
 ```

+### UPDATE 2026-07-13: `ceph tell osd.* config set` DOES Work
+
+In a follow-up session, `ceph tell osd.* config set osd_max_backfills 3`
+DID successfully apply to running OSDs. Verification:
+
+```bash
+ceph tell osd.11 config get osd_max_backfills
+# Returns: {"osd_max_backfills": "3"}
+```
+
+The difference from the earlier session: `ceph config set osd osd_max_backfills 3`
+(MON-level persistent config) was set FIRST, then `ceph tell osd.* config set`
+was used to push it to running OSDs. The combination worked. Injectargs
+returned empty `{}` (appeared to fail) but the config took effect.
+
+**Recommendation**: Use BOTH `ceph config set` (persistent for future OSDs)
+AND `ceph tell osd.* config set` (runtime for current OSDs). Then verify
+with `ceph tell osd.N config get`.
+
 ### `osd_recovery_max_active` — NOT effectively tunable

 `ceph config set osd osd_recovery_max_active 3` accepts the value but OSDs report 0 (meaning auto/default). `injectargs` also shows empty. This appears to be a Ceph limitation — the parameter is managed internally. Do not waste time trying to force it.
@@ -145,6 +164,127 @@ Command: ssh to monitor node, run `ceph status`, report:
  - Announce "COMPLETE" when 0 remapped PGs
 ```

+## Pitfalls: Setting backfill_full_ratio (2026-07-12)
+
+Three commands FAIL when trying to raise the backfill full ratio — only
+ONE works:
+
+```bash
+# ✅ CORRECT — CLI command with hyphen (not underscore):
+ceph osd set-backfillfull-ratio 0.97
+# Verify: ceph osd dump | grep backfillfull_ratio → "backfillfull_ratio 0.97"
+
+# ❌ WRONG — underscore variant:
+ceph osd set-backfillfullratio 0.97
+# → "no valid command found" (EINVAL)
+
+# ❌ WRONG — MON config set:
+ceph config set mon mon_osd_backfillfull_ratio 0.97
+# → "mon_osd_backfillfull_ratio is special and cannot be stored by the mon"
+
+# ❌ WRONG — injectargs on specific OSD:
+ceph tell osd.10 injectargs "--osd_backfill_full_ratio 0.97"
+# → "failed to parse arguments: --osd_backfill_full_ratio,0.97" (EINVAL)
+# injectargs only accepts runtime-tunable OSD params, not full-ratio thresholds.
+```
+
+The ratio is a MON-level OSD map parameter, not a per-OSD config. It can
+ONLY be changed via `ceph osd set-backfillfull-ratio` (CLI command, not
+config set). Similarly: `ceph osd set-nearfull-ratio` and
+`ceph osd set-full-ratio` for the other thresholds.
+
+## Pitfalls: Pool-Level `backfillfull` Flag (2026-07-13)
+
+When pools show the `backfillfull` flag in `ceph osd dump`, attempting
+to unset it at the pool level FAILS:
+
+```bash
+# ❌ WRONG — no such pool flag:
+ceph osd pool unset 6 backfillfull
+# → "no valid command found; 1 closest matches: osd pool unset noautoscale"
+
+# ❌ WRONG — all pool IDs tried, all fail:
+for pool in 1 5 6 7 8 9 10; do ceph osd pool unset $pool backfillfull; done
+# All return EINVAL
+```
+
+The `backfillfull` flag on a pool is a DERIVED condition — it appears when
+any OSD in the pool's up/acting set is above the `backfillfull_ratio`.
+It is NOT a settable pool flag. To clear it, fix the underlying OSD
+condition: raise `backfillfull_ratio` or reweight the overfull OSD.
+
+```bash
+# ✅ CORRECT — raise the ratio (clears backfillfull on all pools):
+ceph osd set-backfillfull-ratio 0.99
+
+# ✅ ALSO — reweight the overfull OSD:
+ceph osd reweight 7 0.8
+```
+
+## Pitfalls: `osd_recovery_max_active` Behavior (2026-07-13)
+
+`ceph config set osd osd_recovery_max_active 3` accepts the value and
+`ceph config get osd osd_recovery_max_active` returns 3. However,
+`ceph tell osd.N config get osd_recovery_max_active` may return empty
+string — the OSD interprets 0 or empty as "auto/default". Despite this,
+setting it alongside `osd_max_backfills` appeared to help overall. Don't
+rely on this parameter alone — use `osd_max_backfills` as the primary lever.
+
+## Pitfalls: `full ratio(s) out of order` (2026-07-13)
+
+When adjusting full/nearfull/backfillfull ratios, you MUST preserve the
+ordering: `nearfull_ratio < backfillfull_ratio < full_ratio <= osd_failsafe_full_ratio`.
+
+The `osd_failsafe_full_ratio` is hardcoded at **0.97** and cannot be changed.
+If you set `full_ratio` above 0.97, the cluster goes to `HEALTH_ERR`:
+
+```
+[ERR] OSD_OUT_OF_ORDER_FULL: full ratio(s) out of order
+    osd_failsafe_full_ratio (0.97) < full_ratio (0.98), increased
+```
+
+**Safe ratio combinations for a cluster with an OSD at 95%:**
+
+```bash
+# ✅ CORRECT — all ratios below failsafe (0.97), properly ordered:
+ceph osd set-nearfull-ratio 0.94
+ceph osd set-backfillfull-ratio 0.96
+ceph osd set-full-ratio 0.97   # == failsafe, OK
+
+# ❌ WRONG — full_ratio above failsafe:
+ceph osd set-full-ratio 0.98   # → HEALTH_ERR: out of order
+```
+
+**With an OSD at 95.2% utilization:**
+- `nearfull_ratio` 0.94 → OSD at 95.2% shows `nearfull` (WARN, acceptable)
+- `backfillfull_ratio` 0.96 → OSD at 95.2% is NOT backfillfull → backfill proceeds
+- `full_ratio` 0.97 → OSD at 95.2% is NOT full → client I/O continues
+
+If the OSD drains below 0.94, all flags clear automatically.
+
+## Pitfalls: `ceph osd reweight` vs `ceph osd crush reweight` (2026-07-13)
+
+Two DIFFERENT commands that both change OSD weighting:
+
+```bash
+# Runtime reweight — temporary, lost on OSD restart:
+ceph osd reweight 7 0.8
+# Changes the `reweight` column in `ceph osd df`. Persists until OSD restart
+# or cluster-wide reweight-by-utilization. Does NOT change CRUSH weight.
+# Good for emergency drain of an overfull OSD.
+
+# CRUSH weight change — permanent, affects PG distribution:
+ceph osd crush reweight osd.7 0.96
+# Changes the CRUSH `weight` column. Survives restarts. Changes how many
+# PGs the OSD gets assigned going forward.
+# Use this to match weight to actual device size.
+```
+
+**Decision guide:**
+- Emergency: OSD is 96%+ and blocking recovery → `ceph osd reweight` (fast, temporary)
+- Permanent: OSD weight doesn't match device size → `ceph osd crush reweight` (slow, proper)
+- After recovery: Reset `reweight` to 1.0, set `crush weight` to true device size
+
 ## Post-Recovery Cleanup

 After all PGs return to active+clean:
@@ -47,8 +47,18 @@ After resetting, recovery resumes (67 MiB/s, 16 obj/s observed).

 - **Replicated pools** (size=3): reweighting is safe — CRUSH redirects
  to other OSDs in the failure domain.
- **Erasure-coded pools**: reweighting is only safe when there are MORE
-  OSDs than `k+m`. With exactly `k+m` OSDs, reweighting traps PGs.
+- **Erasure-coded pools**: `crush reweight` (permanent CRUSH weight change)
+  is only safe when there are MORE OSDs than `k+m`. With exactly `k+m`
+  OSDs, `crush reweight` traps PGs (2147483647 in up-set).
+- **⚠️ `osd reweight` (temporary) CAN drain EC OSDs** (2026-07-12):
+  Unlike `crush reweight`, `ceph osd reweight N 0.5` sets a temporary
+  runtime override that adjusts PG placement probability WITHOUT
+  changing CRUSH weights. This means CRUSH still considers all OSDs
+  available — no 2147483647 trap. Tested on osd.10 (EC4+1, exactly 5
+  HDD OSDs): reweight to 0.5 drained from 93%→91%, backfill started
+  at 30 MiB/s, 5 of 6 `backfill_toofull` PGs unblocked. **Prerequisite:**
+  `crush reweight` must be at 1.0 (default). If `crush reweight` was
+  previously changed, revert it to 1.0 FIRST, then use `osd reweight`.
 - **Long-term fix**: add more HDD OSDs (e.g. on the ubuntu host with
  2×2.8 TB spare) to give CRUSH room to redistribute.

@@ -95,14 +105,63 @@ flagged until its utilization drops below the new threshold naturally

 ### Mitigation

- **Patience**: Once the reweight trap is fixed and recovery proceeds,
+- **`osd reweight` (temporary) WORKS for EC4+1 drain** (2026-07-12):
+  `ceph osd reweight 10 0.5` successfully drained osd.10 from 93.14%→90.90%
+  and unblocked 5 of 6 `backfill_toofull` PGs. This is a runtime
+  override, NOT a CRUSH weight change — CRUSH still sees all OSDs,
+  preventing the 2147483647 trap. Recovery proceeded at 30 MiB/s.
+  Steps: (1) ensure `crush reweight` is 1.0, (2) `ceph osd reweight
+  10 0.5`, (3) `ceph osd set-backfillfull-ratio 0.95` to unblock
+  remaining PGs, (4) wait for recovery, (5) `ceph osd reweight 10 1.0`
+  to restore.
+- **Aggressive drain: reweight 0.3** (2026-07-12 session 2):
+  When 0.5 wasn't enough (osd.10 still at 89%, 1 PG stubbornly
+  `backfill_toofull`), dropping to `ceph osd reweight 10 0.3` increased
+  misplaced objects from 12%→16% (more PGs queued for relocation) and
+  accelerated the drain. Combined with `set-backfillfull-ratio 0.97`.
+  **Reweight progression: 0.5 → 0.3 if 0.5 doesn't drain fast enough.**
+  Remember to restore to 1.0 after recovery completes.
+
+### ⚠️ Correct Command Syntax for backfillfull_ratio (2026-07-12)
+
+Multiple command forms were tried — only ONE works. Save time by using
+the correct one first:
+
+```bash
+# ✅ CORRECT — hyphenated, no underscore:
+ceph osd set-backfillfull-ratio 0.97
+
+# ❌ WRONG — underscore variant (EINVAL):
+ceph osd set_backfill_full_ratio 0.97
+
+# ❌ WRONG — config set (rejected as "special"):
+ceph config set mon mon_osd_backfillfull_ratio 0.97
+# → "mon_osd_backfillfull_ratio is special and cannot be stored by the mon"
+
+# ❌ WRONG — injectargs with wrong key name:
+ceph tell osd.10 injectargs "--osd_backfill_full_ratio 0.97"
+# → "failed to parse arguments"
+
+# ❌ WRONG — osd set with wrong key:
+ceph osd set backfill_full_ratio 0.97
+# → "Invalid command: backfill_full_ratio not in full|pause|..."
+```
+
+Similarly for full_ratio and nearfull_ratio:
+```bash
+ceph osd set-full-ratio 0.97        # ✅ hyphenated
+ceph osd set-nearfull-ratio 0.95     # ✅ hyphenated
+```
+- **Patience**: Once the reweight takes effect and recovery proceeds,
  data redistributes and osd.10 utilization drops, eventually clearing
-  the flag.
+  the backfillfull flag.
 - **PG merge**: Reducing PG count (128→32) eliminates some remapped PGs
  entirely, reducing the number of PGs that need to backfill to the
  stuck OSD.
- **Do NOT reweight osd.10** to reduce its load — with EC4+1 and only 5
-  HDD OSDs, reweighting makes things worse (see above).
+- **Do NOT use `crush reweight`** to reduce osd.10's load — with EC4+1
+  and only 5 HDD OSDs, permanent CRUSH reweighting makes things worse
+  (see EC4+1 Reweight Trap above). Use `osd reweight` (temporary)
+  instead.

 ## PG Merge (128 → 32)

@@ -547,3 +606,27 @@ I/O-intensive operations that write to pools containing that OSD. Either:
  `seafile-api` skill → `references/seafile-immich-backup-strategy.md`.
 - CT111 (Seafile) backup on noris_s3 showed 0 MB (empty/failed) on 2026-07-07.
  Previous valid backup: 558 GB on 2026-07-06.
+
+## Session Log — 2026-07-12 (Continued)
+
+- **osd.10 drain via `osd reweight` SUCCESSFUL**: `ceph osd reweight 10 0.5`
+  drained osd.10 from 93.14%→90.90%. Combined with `set-backfillfull-ratio 0.95`,
+  unblocked 5 of 6 `backfill_toofull` PGs. Recovery at 30 MiB/s, 13% misplaced.
+  This CONTRADICTS the earlier advice "Do NOT reweight osd.10" — that was
+  correct for `crush reweight` (permanent, triggers 2147483647 trap) but
+  WRONG for `osd reweight` (temporary runtime override, safe with EC4+1).
+  Reference patched accordingly.
+- **Orphaned RBD cleanup completed**: vm-143-disk-0 (50G), base-101-disk-0
+  (20G), base-102-disk-0 (20G), vm-902-disk-0 (40G), vm-201-disk-0,
+  vm-202-disk-0, test-dummy all deleted. base-101/102 required
+  `rbd snap unprotect @__base__` before `rbd snap purge` + `rbd rm`.
+  csi-vol-e596 has stale watcher (10.0.30.63, no K8s PVC found) —
+  pending natural watcher timeout. ~161 GiB freed.
+- **K8s worker anti-colocation migration completed**: VM 132→proxmox1,
+  VM 131→proxmox4, VM 128→n5pro via `qm migrate --online` (HA-managed,
+  async). Required expanding `k8s-prefer-n5pro` node-affinity rule to
+  include all 8 nodes first. All 6 K8s VMs now on 6 separate nodes.
+- **hdd_disk→vm_disks migration feasibility**: vm_disks has only 86 GiB
+  free (75% full). Small CTs (vm-127 4G, vm-121 4G, base-9000 3G,
+  vm-9001 10G = ~21G total) are candidates. Large K8s worker disks
+  (80G×3) stay on hdd_disk (correct HDD-tier placement).
@@ -195,3 +195,69 @@ n5pro is NOT reachable by hostname from proxmox1 — use IP:
 ```bash
 ssh -o StrictHostKeyChecking=no 10.0.20.91 'commands'
 ```
+
+### `pvesm list` Fails During Ceph Backfillfull (2026-07-12)
+
+When Ceph pool `hdd_disk` is in `backfillfull` state (osd.10 at 96.72%),
+`pvesm list hdd_disk` returns empty with error:
+```
+rbd error: rbd: listing images failed: (2) No such file or directory
+```
+Similarly `pvesh get /nodes/{node}/storage/hdd_disk/content` fails.
+**Fallback**: Use `rbd ls hdd_disk` directly (works even when PVE storage
+layer cannot enumerate). Cross-reference with `pvesh get /cluster/resources
+--type vm` for orphan detection.
+
+### HA Rule Created: k8s-prefer-n5pro (2026-07-12)
+
+Successfully created node-affinity rule for K8s VMs:
+- Rule name: `k8s-prefer-n5pro`
+- Type: `node-affinity`
+- Resources: `vm:118,vm:128,vm:129,vm:130,vm:131,vm:132`
+- Nodes: `n5pro,proxmox5,proxmox3,proxmox2` (ordered preference)
+- All 6 VMs briefly went through "deleting" → "starting" → "started" transition
+- Existing rules untouched (ct:104/ct:99999 → n5pro, vm:310/311 + vm:301/302 negative affinity)
+
+See `references/ha-manager-and-custom-dashboards-2026-07.md` for full syntax.
+
+## K8s VM Anti-Affinity Problem (2026-07-12)
+
+### Current Distribution (BAD)
+
+| Node | CP | Worker | Risk |
+|------|----|--------|------|
+| proxmox2 | cp-03 (129) | worker-02 (132) | 2 VMs lost on node failure |
+| proxmox3 | cp-02 (130) | worker-03 (131) | 2 VMs lost on node failure |
+| proxmox5 | cp-01 (118) | worker-01 (128) | 2 VMs lost on node failure |
+
+Each of the 3 K8s nodes hosts BOTH a CP and a Worker. A single node
+failure drops 1 CP + 1 Worker simultaneously. Quorum survives (2/3 CPs)
+but workload capacity drops 33%.
+
+### K8s VM Anti-Colocation: EXECUTED (2026-07-12) ✅
+
+The proposed distribution was successfully implemented via live migration.
+All 3 worker VMs migrated to separate nodes (no node hosts >1 K8s VM):
+
+| Node | CP | Worker |
+|------|----|--------|
+| proxmox1 | — | worker-02 (132) ← migrated FROM proxmox2 |
+| proxmox2 | cp-03 (129) | — |
+| proxmox3 | cp-02 (130) | — |
+| proxmox4 | — | worker-03 (131) ← migrated FROM proxmox3 |
+| proxmox5 | cp-01 (118) | — |
+| n5pro | — | worker-01 (128) ← migrated FROM proxmox5 |
+
+**Migration commands** (ran from source node, not coordinator):
+```bash
+ssh root@10.0.20.20 'qm migrate 132 proxmox1 --online'  # proxmox2 → proxmox1
+ssh root@10.0.20.30 'qm migrate 131 proxmox4 --online'  # proxmox3 → proxmox4
+ssh root@10.0.20.50 'qm migrate 128 n5pro --online'     # proxmox5 → n5pro
+```
+
+HA rule `k8s-prefer-n5pro` was expanded to include ALL 8 nodes before
+migration (was missing proxmox1, proxmox4 — caused migration failure).
+See `references/ha-manager-and-custom-dashboards-2026-07.md` for details.
+
+Result: Any single node failure now loses at most 1 K8s VM (was 2).
+Quorum survives any single CP node failure (2/3 CPs remain).
@@ -0,0 +1,53 @@
+# CT Deletion: `pct destroy` Leaving Stale Config — 2026-07-14
+
+## When to Use
+
+When `pct destroy NNN --purge` fails because the RBD disk was already
+deleted (partial destroy, manual disk removal, or disk lost during Ceph
+recovery), but the CT config remains in `/etc/pve/lxc/NNN.conf`.
+
+## Symptoms
+
+```bash
+pct destroy 112 --purge --destroy-unreferenced-disks
+# rbd error: rbd: error opening image vm-112-disk-0: (2) No such file or directory
+# exit code: 255
+
+pct status 112
+# status: stopped  ← CT still appears!
+
+pct list | grep 112
+# 112   stopped   authelia  ← still listed!
+```
+
+## Cause
+
+`pct destroy` operates in two phases:
+1. Delete the storage (RBD images)
+2. Remove the CT config (`/etc/pve/lxc/NNN.conf`)
+
+If phase 1 fails (disk already gone → `rbd: error opening image`), the
+command aborts with exit 255 **before reaching phase 2**. The config
+remains in place.
+
+## Fix
+
+```bash
+# On the PVE node hosting the CT (discover via ha-manager status):
+rm -f /etc/pve/lxc/NNN.conf
+
+# Verify:
+pct list | grep NNN  # should return nothing
+```
+
+The config removal is safe — the disk is already gone, so there's
+nothing to corrupt. `/etc/pve/lxc/` is a shared FUSE filesystem (pmxcfs),
+so the removal propagates cluster-wide instantly.
+
+## Related
+
+- `references/ceph-orphaned-rbd-cleanup-2026-07.md` — "VM Destruction
+  on Non-Ceph Nodes" covers the `qm destroy` equivalent (CFS lock
+  timeout, stale locks, manual config + RBD cleanup).
+- The LXC case is simpler than the QEMU case: no CFS lock issues, no
+  watcher problems — just a stale config file to remove.
@@ -0,0 +1,45 @@
+# CT Deletion with Stale RBD Config — 2026-07-14
+
+## Problem
+
+`pct destroy <CTID> --purge` fails with exit 255 when the RBD disk image
+was already deleted by a prior operation:
+
+```
+rbd error: rbd: error opening image vm-112-disk-0: (2) No such file or directory
+```
+
+The CT config file remains in pmxcfs (`/etc/pve/lxc/NNN.conf`), so
+`pct list` still shows the CT as `stopped`.
+
+## Fix
+
+```bash
+# On the PVE node hosting the CT:
+rm -f /etc/pve/lxc/NNN.conf
+# Verify:
+pct list | grep NNN  # Should return nothing
+```
+
+## Reliable CT Deletion Sequence
+
+1. `pct stop NNN` (ensure stopped)
+2. `pct destroy NNN --purge --destroy-unreferenced-disks`
+3. If it fails with RBD error but config remains:
+   `rm -f /etc/pve/lxc/NNN.conf`
+4. Verify: `pct list | grep NNN` → empty
+
+## Pitfall
+
+`--destroy-unreferenced-disks` helps clean up orphaned RBD images not
+referenced in the CT config, but if the RBD image is already gone, the
+flag doesn't suppress the error — it just tries and fails on the missing
+image. The config file is the leftover that needs manual cleanup.
+
+## Session Context
+
+CT112 (Authelia) was stopped and then deleted with `pct destroy 112
+--purge`. The RBD image was apparently already removed (possibly during
+a prior storage migration or manual cleanup), so the destroy command
+failed on the missing image. The config was manually removed with
+`rm -f /etc/pve/lxc/112.conf`.
@@ -0,0 +1,294 @@
+# Gitea API, Project Boards & Skills Versioning
+
+## Link Format Convention (USER PREFERENCE)
+
+**Always use `git.familie-schoen.com` for user-facing links in chat responses.**
+Never paste the internal IP `10.0.30.105:3000` as a clickable link to the user.
+
+| Context | Format | Example |
+|---------|--------|---------|
+| User-facing links in chat | `git.familie-schoen.com` | `http://git.familie-schoen.com/dominik/iac-homelab/issues/14` |
+| API calls (curl) | `10.0.30.105:3000` | `curl http://10.0.30.105:3000/api/v1/repos/...` |
+| SSH to Gitea host | proxmox6 IP `10.0.20.60` | `ssh root@10.0.20.60 'pct exec 108 -- ...'` |
+
+The domain `git.familie-schoen.com` resolves externally; the internal IP is
+needed for API calls from within the network. When referencing Gitea issues,
+PRs, or boards in any user-visible output, always use the domain form.
+
+## Gitea Server Access
+
+- **Internal URL (API only)**: http://10.0.30.105:3000 (CT108 on proxmox6)
+- **External URL (user-facing links)**: http://git.familie-schoen.com
+- **User**: dominik
+- **Token**: Generate via CLI on CT108 (1Password `op` may fail on VM200):
+  ```bash
+  ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.105 \
+    "su -s /bin/sh gitea -c 'gitea admin user generate-access-token -u dominik -t <name> --scopes all --raw --config /etc/gitea/app.ini'"
+  ```
+- **Existing repos**: dominik/iac-homelab (IaC), dominik/hermes-skills (skills)
+
+## Gitea API Patterns
+
+```bash
+TOKEN="<generated-token>"
+BASE="http://10.0.30.105:3000/api/v1"
+REPO="dominik/iac-homelab"
+
+# Create issue
+curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
+  -d '{"title":"Issue title","body":"Description","labels":[1,5],"assignees":["dominik"]}' \
+  "$BASE/repos/$REPO/issues"
+
+# Comment on issue
+curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
+  -d '{"body":"Comment text"}' \
+  "$BASE/repos/$REPO/issues/ISSUE_NUMBER/comments"
+
+# List all issues
+curl -s -H "Authorization: token $TOKEN" "$BASE/repos/$REPO/issues?limit=50&type=issues"
+
+# Close issue
+curl -s -X PATCH -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
+  -d '{"state":"closed"}' "$BASE/repos/$REPO/issues/ISSUE_NUMBER"
+
+# Create labels (returns ID needed for issue creation)
+curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
+  -d '{"name":"priority:high","color":"#b60205"}' "$BASE/repos/$REPO/labels"
+```
+
+## Pitfall: Gitea 1.25.5 Project Board API Missing
+
+Gitea 1.25.5 (dev build `go1.25.8, bindata/sqlite`) has project tables in DB
+and repo unit enabled, but **REST API routes for projects are not registered**.
+All project endpoints return 404:
+- `/repos/{owner}/{repo}/projects`
+- `/repos/{owner}/{repo}/projects/` (trailing slash doesn't help)
+- Org-level (`/orgs/{org}/projects`) and user-level project endpoints
+
+### Accessing CT108 (Gitea)
+
+CT108 runs on **proxmox6** (10.0.20.60), NOT directly reachable at 10.0.30.105
+for SSH. The 10.0.30.105 address is the CT's network IP for HTTP/API only.
+
+```bash
+# CORRECT: SSH to proxmox6, then pct exec
+ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.60 'pct exec 108 -- bash -c "..."'
+
+# WRONG: SSH to 10.0.30.105 (this is the CT's DHCP IP, not a hypervisor)
+# ssh root@10.0.30.105 → fails or goes to wrong host
+```
+
+### Generating a Gitea Token
+
+The `gitea` binary is at `/usr/local/bin/gitea` — NOT in the default PATH
+for the `gitea` system user. Must use the full path:
+
+```bash
+# CORRECT (note full binary path):
+ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.60 'pct exec 108 -- bash -c "
+  su -s /bin/sh gitea -c \"/usr/local/bin/gitea admin user generate-access-token -u dominik -t <name> --scopes all --raw --config /etc/gitea/app.ini\"
+"'
+
+# WRONG (gitea not in PATH):
+# su -s /bin/sh gitea -c 'gitea admin ...' → "gitea: not found" (exit 127)
+```
+
+### Workaround: Direct SQLite manipulation
+
+```bash
+# SSH to proxmox6, then pct exec into CT108
+ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.60
+
+# Enter CT108
+pct exec 108 -- bash
+
+# Create project board directly in SQLite
+sqlite3 /var/lib/gitea/data/gitea.db \
+  "INSERT INTO project (title, description, repo_id, type, board_type, card_type, creator_id, is_closed, created_unix, updated_unix) \
+   VALUES ('Board Title', 'Description', REPO_ID, 2, 0, 0, 1, 0, strftime('%s','now'), strftime('%s','now'));"
+# type=2 for repo-level project, board_type=0 for kanban
+# Get the inserted project ID: SELECT last_insert_rowid();
+
+# Create board columns (Backlog, Todo, In Progress, Done)
+sqlite3 /var/lib/gitea/data/gitea.db << 'SQL'
+INSERT INTO project_board (title, sorting, project_id, creator_id, created_unix, updated_unix) VALUES
+('Backlog', 0, PROJECT_ID, 1, strftime('%s','now'), strftime('%s','now')),
+('Todo', 1, PROJECT_ID, 1, strftime('%s','now'), strftime('%s','now')),
+('In Progress', 2, PROJECT_ID, 1, strftime('%s','now'), strftime('%s','now')),
+('Done', 3, PROJECT_ID, 1, strftime('%s','now'), strftime('%s','now'));
+SQL
+
+# Map issues to board columns
+sqlite3 /var/lib/gitea/data/gitea.db << 'SQL'
+INSERT INTO project_issue (issue_id, project_id, project_board_id, sorting) VALUES
+(ISSUE_DB_ID, PROJECT_ID, BOARD_COLUMN_ID, SORT_ORDER);
+SQL
+```
+
+### ⚠️ SQLite Schema Quirks
+
+1. **`index` is a SQLite reserved word** — the `issue` table has a column
+   named `index` (the issue number displayed in UI). Must always quote it:
+   ```sql
+   SELECT id, "index", name FROM issue WHERE repo_id=34;  -- CORRECT
+   SELECT id, index, name FROM issue WHERE repo_id=34;    -- SYNTAX ERROR
+   ```
+
+2. **`project_issue` table has NO timestamp columns** — unlike most Gitea
+   tables, it only has: `id`, `issue_id`, `project_id`, `project_board_id`,
+   `sorting`. Do NOT include `created_unix`/`updated_unix` in INSERTs.
+
+3. **Issue `id` ≠ `index`** — the API returns `number` (which maps to
+   `index` in DB), but `project_issue` references the DB primary key `id`.
+   Always look up both: `SELECT id, "index", name FROM issue WHERE repo_id=N;`
+
+### Label Color Format
+
+Gitea API rejects colors with `#` prefix. Use bare hex:
+```bash
+# CORRECT:
+curl -d '{"name":"ceph","color":"7050ff"}'  # → 200 OK
+
+# WRONG:
+curl -d '{"name":"ceph","color":"#7050ff"}' # → 404 / error
+```
+
+### Label IDs are numeric in API
+
+When creating issues via API, the `labels` field takes **numeric IDs** (from
+label creation response), not label names. Passing names silently ignores them.
+Look up IDs first: `GET /repos/{owner}/{repo}/labels` → use `id` field.
+
+### Complete PM Setup Recipe (End-to-End)
+
+```bash
+TOKEN="<generated-token>"
+BASE="http://10.0.30.105:3000/api/v1"
+AUTH="Authorization: token $TOKEN"
+
+# 1. Create org
+curl -s -X POST -H "$AUTH" -H "Content-Type: application/json" \
+  "$BASE/orgs" -d '{"username":"pm-infra","description":"...","visibility":"private"}'
+
+# 2. Create repo under org
+curl -s -X POST -H "$AUTH" -H "Content-Type: application/json" \
+  "$BASE/orgs/pm-infra/repos" \
+  -d '{"name":"homelab-board","private":true,"has_issues":true,"has_projects":true,"default_branch":"main"}'
+
+# 3. Create labels (colors WITHOUT #)
+for label in "ceph:7050ff:Ceph storage" "k8s:326ce5:Kubernetes" ...; do
+  IFS=: read name color desc <<< "$label"
+  curl -s -X POST -H "$AUTH" -H "Content-Type: application/json" \
+    "$BASE/repos/pm-infra/homelab-board/labels" \
+    -d "{\"name\":\"$name\",\"color\":\"$color\",\"description\":\"$desc\"}"
+done
+
+# 4. Create milestones
+curl -s -X POST -H "$AUTH" -H "Content-Type: application/json" \
+  "$BASE/repos/pm-infra/homelab-board/milestones" \
+  -d '{"title":"Infra Audit & Cleanup","description":"..."}'
+
+# 5. Create issues (labels=numeric IDs, milestone=numeric ID)
+curl -s -X POST -H "$AUTH" -H "Content-Type: application/json" \
+  "$BASE/repos/pm-infra/homelab-board/issues" \
+  -d '{"title":"...","body":"...","labels":[8,17],"milestone":1}'
+
+# 6. Close done issues
+curl -s -X PATCH -H "$AUTH" -H "Content-Type: application/json" \
+  "$BASE/repos/pm-infra/homelab-board/issues/3" -d '{"state":"closed"}'
+
+# 7. Create project board via SQLite (API returns 404)
+#    → See "Direct SQLite manipulation" section above
+
+# 8. Map issues to board columns via SQLite
+#    → See "SQLite Schema Quirks" for gotchas
+```
+
+### ⚠️ execute_code Blocked by Cron Approval Mode
+
+`execute_code` may be blocked with "cron approval mode" error even in
+interactive sessions. When it fails, use `terminal` + `curl` + `python3 -c`
+instead. Avoid `execute_code` for Gitea API scripting — use inline Python
+in terminal commands for JSON construction.
+
+## Skills Versioning + Gitea Sync (2026-07-12)
+
+### Setup
+
+```bash
+cd ~/.hermes/skills/
+
+# .gitignore (exclude archives + bundled manifest)
+cat > .gitignore << 'EOF'
+.archive/
+.bundled_manifest
+*.pyc
+__pycache__/
+EOF
+
+# Init + commit + push
+git init
+git add -A
+git commit -m "Initial commit: Hermes Agent Skills collection"
+git branch -M main  # rename master → main
+git remote add origin http://10.0.30.105:3000/dominik/hermes-skills.git
+git push -u "http://dominik:<token>@10.0.30.105:3000/dominik/hermes-skills.git" main
+
+# Clean token from stored remote URL
+git remote set-url origin http://10.0.30.105:3000/dominik/hermes-skills.git
+```
+
+### Result
+- 789 files, 233,126 lines committed
+- Repo: http://10.0.30.105:3000/dominik/hermes-skills
+- Private repo, owned by dominik
+
+### Pitfalls
+1. **Default branch is `master`** — `git push main` fails with "src refspec main does not match any". Fix: `git branch -M main` before pushing.
+2. **Token in remote URL** — `git push` needs auth. Embed token in push URL: `http://dominik:<token>@host/...`. After successful push, reset URL without token: `git remote set-url origin http://host/...` to avoid storing credentials in `.git/config`.
+3. **Submodule warnings** — Some skill directories (e.g. `productivity/noris-pptx`) may be git submodules. `git add -A` warns but still commits them as regular files. Check `git rm --cached` if needed.
+4. **1Password CLI may fail on VM200** — `op item get "Gitea Token" --vault "Kubernetes ESO" --reveal` fails intermittently. Fall back to generating a fresh token via `gitea admin user generate-access-token` on CT108 directly.
+
+## PM Tool Decision: Gitea over Telegram Topics (2026-07-12)
+
+User explicitly chose **Gitea Issues + Project Boards** over Telegram
+Topics as the PM tool for infrastructure work. The Telegram Topics
+approach (below) was abandoned because:
+- Bots cannot create supergroups (requires manual user action)
+- Telegram topics lack structured state management (labels, milestones,
+  assignees, dependencies)
+- Gitea integrates with the existing IaC repo workflow
+
+The complete Gitea PM setup (org → repo → labels → milestones → issues
+→ project board via SQLite) is documented in the sections above and was
+fully implemented on 2026-07-12. The board lives at:
+`http://10.0.30.105:3000/pm-infra/homelab-board`
+
+## Telegram Topics as PM Tool (ABANDONED — kept for reference)
+
+### Limitation: Bots cannot create supergroups
+Telegram Bot API does not allow bots to create groups or supergroups. The user
+must manually:
+1. Create a new group in Telegram
+2. Add `@hermesdominik_bot` as **Admin** (with "Add New Members" + "Manage Topics" rights)
+3. Enable **Topics** in group settings ("Topics" toggle)
+4. Share the Chat ID with the agent
+
+Once the bot is admin in a topics-enabled supergroup, the agent can create
+topic threads via the Bot API:
+
+```bash
+# Create a topic thread (requires forum_topic icon color)
+curl -s -X POST "https://api.telegram.org/bot<TOKEN>/createForumTopic" \
+  -d "chat_id=-100xxxx&name=Ceph Crisis&icon_color=0xff6b6b"
+
+# Send message to a specific topic
+curl -s -X POST "https://api.telegram.org/bot<TOKEN>/sendMessage" \
+  -d "chat_id=-100xxxx&message_thread_id=TOPIC_ID&text=Status update"
+```
+
+### Bot Token Location
+- Stored in `~/.hermes/.env` as `TELEGRAM_BOT_TOKEN`
+- Source with: `source ~/.hermes/.env && export TELEGRAM_BOT_TOKEN`
+- Bot: `@hermesdominik_bot` (ID: 8482479728)
+- Home chat: 223926918 (DM, not a group)
@@ -10,15 +10,273 @@ No `--group` parameter needed (groups migrated to rules in newer PVE).
 Specifying `--group 1` fails with: `invalid configuration ID '1'`.
 Verify: `ha-manager status | grep 127` → `service ct:127 (proxmox6, started)`.

-### Pitfall: ha-manager groups → rules migration
-`ha-manager groupconfig` fails with "ha groups have been migrated to rules".
-Use `ha-manager rules list` instead. The `add` command works without a group.
+### Pitfall: ha-manager groups → rules migration (PVE 9.x)
+`ha-manager groupadd` and `ha-manager groupconfig` fail with "ha groups have been migrated to rules".
+Use `ha-manager rules list` / `ha-manager rules config` instead.
+
+**PVE 9.2.3 Rules Syntax** (discovered 2026-07-12, took multiple failed attempts):
+
+```bash
+# Create node-affinity rule (replaces HA groups)
+ha-manager rules add node-affinity <rule-name> \
+  --nodes "n5pro,proxmox5,proxmox3,proxmox2" \
+  --resources "vm:118,vm:128,vm:129,vm:130,vm:131,vm:132"
+
+# List all rules (table format)
+ha-manager rules config
+
+# Raw config file
+cat /etc/pve/ha/rules.cfg
+```
+
+Rule types: `node-affinity` (prefer certain nodes) and `resource-affinity` (keep resources together/apart).
+
+**Failed approaches** (do NOT use):
+- `ha-manager groupadd k8s-nodes --nodes "..."` → "cannot create group: ha groups have been migrated to rules"
+- `ha-manager add vm:118 --group k8s-nodes` → "invalid parameter 'group'"
+- `ha-manager rules add location "node==n5pro"` → type must be `node-affinity` or `resource-affinity`
+- `ha-manager rules add node-affinity "n5pro"` without `--nodes` → "missing value for required option 'nodes'"
+
+**Removing + re-adding HA services** (needed when migrating from old group config):
+```bash
+ha-manager remove vm:118          # VM keeps running, loses HA protection, shows "deleting"
+ha-manager add vm:118 --state started   # Re-add immediately
+ha-manager status | grep "service vm:118"  # Verify: "started"
+```
+
+⚠️ When `ha-manager remove` runs, the service transitions through "deleting" state. The VM continues running. Re-add immediately to restore HA protection.

 ### CT 127 Added to HA (2026-07-05)
 CT 127 (avahi-reflector, proxmox6) added to ha-manager with max_restart=1,
 max_relocate=1. Was previously NOT HA-managed. Verified: `service ct:127
 (proxmox6, queued)` → `started`.

+### K8s VMs Node-Affinity Rule (2026-07-12)
+Created rule `k8s-prefer-n5pro` (type: node-affinity) for VMs 118,128-132.
+Initial nodes: n5pro,proxmox5,proxmox3,proxmox2 (n5pro = preferred, others = failover).
+All 6 K8s VMs re-added to HA after brief "deleting" transition.
+Existing rules in cluster: `ha-rule-9a834247-53bf` (ct:104,ct:99999 → n5pro),
+`ha-rule-ea9e852a-8d82` (vm:310,vm:311 negative affinity), `ha-rule-d61699eb-2baf` (vm:301,vm:302 negative affinity).
+
+### ⚠️ PITFALL: HA Node-Affinity Blocks Live Migration (2026-07-12)
+When K8s VMs are HA-managed with a `node-affinity` rule restricting them to
+certain nodes, `qm migrate <vmid> <target> --online` FAILS if the target node
+is not in the rule's `--nodes` list:
+
+```
+cannot migrate resource 'vm:132' to node 'proxmox1':
+- resource 'vm:132' not allowed on target node 'proxmox1'
+```
+
+**Fix:** Expand the rule's node list BEFORE attempting migration. Edit
+`/etc/pve/ha/rules.cfg` directly (it's a shared FUSE filesystem, edits
+propagate instantly):
+
+```bash
+# Add all cluster nodes to the rule
+sed -i '/^node-affinity: k8s-prefer-n5pro$/,/^$/{
+  s/nodes n5pro,proxmox2,proxmox3,proxmox5/nodes n5pro,proxmox1,proxmox2,proxmox3,proxmox4,proxmox5,proxmox6,proxmox7/
+}' /etc/pve/ha/rules.cfg
+```
+
+Then retry `qm migrate` from the SOURCE node (not the coordinator). The
+migration command must run on the node where the VM currently lives:
+
+```bash
+# CORRECT: SSH to source node, then migrate
+ssh root@10.0.20.20 'qm migrate 132 proxmox1 --online'   # proxmox2 → proxmox1
+ssh root@10.0.20.30 'qm migrate 131 proxmox4 --online'   # proxmox3 → proxmox4
+ssh root@10.0.20.50 'qm migrate 128 n5pro --online'      # proxmox5 → n5pro
+
+# WRONG: Running qm migrate from the coordinator (proxmox1) for a VM on proxmox2
+# → "400 Parameter verification failed. target: target is local node."
+```
+
+HA migrations are asynchronous — `qm migrate` returns immediately with
+"Requesting HA migration for VM NNN to node XXX". The HA manager processes
+the migration in the background. Check progress with:
+
+```bash
+ha-manager status | grep "vm:NNN"
+# States: migrate → starting → started
+```
+
+Poll every 15-30s until all show `started`. Typical duration: 30-90s per VM
+(longer if Ceph is under backfill load).
+
+### K8s VM Anti-Colocation Strategy (2026-07-12)
+
+**Problem:** Initially each K8s node hosted BOTH a CP AND a Worker:
+
+| Node | CP | Worker | Risk |
+|------|-----|--------|------|
+| proxmox2 | cp-03 (129) | worker-02 (132) | ⚠️ Node loss = 2 VMs gone |
+| proxmox3 | cp-02 (130) | worker-03 (131) | ⚠️ Node loss = 2 VMs gone |
+| proxmox5 | cp-01 (118) | worker-01 (128) | ⚠️ Node loss = 2 VMs gone |
+
+**Solution:** Spread 6 VMs across 6 different nodes (1 per node):
+
+| Node | CP | Worker |
+|------|-----|--------|
+| proxmox1 | — | worker-02 (132) |
+| proxmox2 | cp-03 (129) | — |
+| proxmox3 | cp-02 (130) | — |
+| proxmox4 | — | worker-03 (131) |
+| proxmox5 | cp-01 (118) | — |
+| n5pro | — | worker-01 (128) |
+
+Since all disks are on shared Ceph storage, live migration is pure RAM
+transfer — no storage move needed. Quorum survives any single node failure
+(2/3 CPs remain), and only 1 worker is lost per node failure (33% capacity
+reduction, not 66%).
+
+**Corosync node IP reference** (from `/etc/pve/corosync.conf`):
+
+| Node | Ring0 IP |
+|------|----------|
+| proxmox1 | 10.0.20.10 |
+| proxmox2 | 10.0.20.20 |
+| proxmox3 | 10.0.20.30 |
+| proxmox4 | 10.0.20.40 |
+| proxmox5 | 10.0.20.50 |
+| proxmox6 | 10.0.20.60 |
+| proxmox7 | 10.0.20.70 |
+| n5pro | 10.0.20.91 |
+
+### DB Nodes Strict Affinity Rule (2026-07-13, updated for quorum safety)
+
+**Problem:** Galera + MaxScale VMs were co-located with K8s CP/worker VMs on 15 GB
+PVE nodes, causing RAM overcommit >147%. An OOM-killer event on proxmox2 killed
+the KVM process of VM300 (mariadb-01), dropping the Galera cluster from 3→2 nodes.
+
+**Initial fix (2 hosts — had quorum risk):** Restricted DB VMs to n5pro + proxmox3.
+But this placed 2 of 3 Galera nodes on proxmox3 — if proxmox3 fails, only 1 Galera
+node survives → **no quorum, entire cluster down.**
+
+**⚠️ CRITICAL Quorum Rule for Galera Placement:**
+Each Galera node MUST be on a different physical PVE host. With 3 nodes, losing
+any single host must leave ≥2 alive (= quorum). 2 nodes on the same host =
+single point of failure for the entire database cluster.
+
+**Final fix (3 hosts, quorum-safe):** Added proxmox6 as third allowed host:
+
+```bash
+# /etc/pve/ha/rules.cfg (shared FUSE FS, edits propagate instantly)
+node-affinity: db-nodes
+	nodes n5pro,proxmox3,proxmox6
+	resources vm:300,vm:301,vm:302,vm:310,vm:311
+	strict 1
+```
+
+**Resulting placement (3 different hardware hosts):**
+| VM | Node | Quorum impact if host fails |
+|----|------|---------------------------|
+| 300 (mariadb-01) | n5pro | 2/3 survive on proxmox3+proxmox6 ✅ |
+| 301 (mariadb-02) | proxmox3 | 2/3 survive on n5pro+proxmox6 ✅ |
+| 302 (mariadb-03) | proxmox6 | 2/3 survive on n5pro+proxmox3 ✅ |
+
+With `strict 1`, the HA manager will NEVER place these VMs on other nodes,
+even during failover. If all 3 allowed nodes are down, the VMs stay down
+rather than starting on an overloaded 15 GB node and risking another OOM.
+
+**⚠️ PITFALL: Live Migration Can Kill Recovering Galera Nodes**
+Never live-migrate a Galera VM that is still in `activating` state (SST in
+progress) or was recently OOM-killed. The VM can freeze during migration,
+drop from the cluster, and become unreachable (guest agent down, SSH refused).
+Wait until `wsrep_local_state_comment = Synced` before migrating.
+
+**Existing anti-affinity rules** (keep Galera/MaxScale nodes apart):
+- `vm:310,vm:311` negative resource-affinity (MaxScale auseinander)
+- `vm:301,vm:302` negative resource-affinity (Galera auseinander)
+
+⚠️ Anti-affinity can block migration: VM 311 couldn't migrate to n5pro because
+VM 310 was already there and they have negative affinity. Had to use proxmox3
+instead. Always check `rules.cfg` before choosing a migration target.
+
+### OOM-Killer on PVE: Diagnosis Pattern (2026-07-13)
+
+When a VM is unreachable (SSH closed, Guest Agent down) but QEMU shows `running`,
+check the PVE host's kernel log for OOM-killer events:
+
+```bash
+# On the PVE host where the VM runs:
+dmesg -T | grep -iE "oom|killed|out of memory" | tail -20
+# Look for: "Out of memory: Killed process <PID> (kvm)" with task_memcg=/qemu.slice/<VMID>.scope
+```
+
+**Root cause pattern:** PVE RAM overcommit = sum(VM maxmem) / node physical RAM.
+When this exceeds 100% and a VM demands more memory, the host kernel OOM-killer
+selects the largest memory consumer (usually a KVM process) and kills it.
+
+**Why `balloon: 0` on DB VMs is correct:** MySQL/Galera uses InnoDB buffer pool
+(hot pages constantly accessed). If the balloon shrinks, the guest kernel swaps
+hot buffer pool pages → massive latency → Galera flow control throttles the ENTIRE
+cluster. Ballooning is appropriate for idle/web VMs, NOT for database workloads.
+
+**Mitigation hierarchy:**
+1. Migrate DB VMs to nodes with RAM headroom (n5pro: 91 GB, proxmox3: 31 GB)
+2. Set strict HA node-affinity rules to prevent failover to small nodes
+3. Deploy RAM-based rebalancer cron job (see below)
+4. Long-term: upgrade RAM on 15 GB nodes or redistribute K8s VMs
+
+### RAM-Based Rebalancer (2026-07-13)
+
+PVE HA manager does NOT do RAM-based rebalancing — `ha-manager rebalance` counts
+VMs per node, not their resource consumption. A custom script + cron job fills
+this gap:
+
+**Script:** `~/.hermes/scripts/pve-ram-rebalancer.sh`
+- Queries `pvesh get /cluster/resources` for VM allocations and node RAM
+- Calculates overcommit = sum(running VM maxmem on node) / node physical RAM
+- If any node > 100%, finds the largest HA-managed VM and migrates it to the
+  node with the most free RAM (that won't exceed 100% after adding the VM)
+- Respects HA node-affinity rules (strict rules constrain target selection)
+- Only migrates HA-managed VMs (non-HA VMs can't be safely migrated via ha-manager)
+- **Silent when balanced** (empty stdout = nothing to do)
+- Outputs `MIGRATE <vmid> <from> <to>` lines when action needed
+
+**Cron job:** Job ID `a41717349482`, every 10 minutes, delivers to SRE agent.
+The agent reviews proposed migrations, executes them via `qm migrate --online`,
+and reports a summary. Only triggers when the script produces output (changes).
+
+**Known limitation:** If ALL nodes with free RAM are constrained by HA rules
+(e.g., db-nodes strict rule limits DB VMs to n5pro+proxmox3, and both are full),
+the script correctly reports "balanced" — it can't violate HA rules. The solution
+is to relax the rule or add more RAM-capable nodes to the rule.
+
+### Cluster RAM Inventory (2026-07-13)
+
+| Node | Physical RAM | Cores | Role |
+|------|-------------|-------|------|
+| n5pro | 91 GB | 24 | Big node — DB VMs, K8s worker |
+| proxmox3 | 31 GB | 4 | Medium — DB failover, K8s CP |
+| proxmox1 | 15 GB | 8 | Small — K8s worker, HA, embedding |
+| proxmox2 | 15 GB | 4 | Small — K8s CP (was OOM source) |
+| proxmox4 | 15 GB | 4 | Small — K8s worker, openwebui |
+| proxmox5 | 15 GB | 4 | Small — K8s CP |
+| proxmox6 | 15 GB | 4 | Small — various CTs |
+| proxmox7 | 15 GB | 4 | Small — Hermes, monitoring, SMB |
+
+⚠️ 6 of 8 nodes have only 15 GB RAM. K8s VMs (12 GB each) + any other VM
+on the same node = overcommit. Distribute carefully.
+
+### RKE2 CP Nodes Are Untainted (2026-07-13)
+
+RKE2 does NOT set `NoSchedule` taints on control-plane nodes by default.
+All 6 K8s nodes (3 CP + 3 Worker) schedule pods freely:
+
+| Node | Role | Pods |
+|------|------|------|
+| rke2-cp-01 | CP | 21 (most!) |
+| rke2-cp-03 | CP | 13 |
+| rke2-cp-02 | CP | 11 |
+| rke2-worker-03 | Worker | 14 |
+| rke2-worker-02 | Worker | 6 |
+| rke2-worker-01 | Worker | 5 |
+
+This means CP nodes carry real workload — don't assume they're "just etcd+API".
+When planning RAM allocation, count CP nodes as full workload nodes.
+
 ## Custom Grafana Dashboard Creation via REST API

 Custom dashboards can be created directly via the Grafana REST API without