feat: add home-assistant-dashboard-conventions skill + update multiple skills

- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
2026-07-14 18:35:16 +00:00
parent e9cc106625
commit 01bd921ced
37 changed files with 6133 additions and 126 deletions
@@ -0,0 +1,546 @@
+---
+name: ceph-cluster-administration
+description: "Class-level skill for Ceph cluster administration on Proxmox VE. Covers OSD lifecycle (create/destroy/WAL-DB), CRUSH topology (device classes, roots, rules, weights), EC pools (profiles, mixed disk sizes, migration), recovery & rebalancing (tuning, ratios, monitoring), RBD management (orphan cleanup, migration, inspection), pool management (PG count, full recovery, autoscale), and health diagnosis."
+version: 1.0.0
+author: Hermes Agent
+tags: [ceph, proxmox, bluestore, crush, erasure-coding, rbd, recovery, osd, storage]
+metadata:
+  hermes:
+    tags: [ceph, proxmox, bluestore, crush, erasure-coding, rbd, recovery, osd, storage]
+    related_skills: [proxmox-ve-administration, rke2-cluster-administration]
+---
+
+# Ceph Cluster Administration
+
+## Scope
+
+**This skill covers the Homelab (self-hosted) Ceph cluster.** Not noris infrastructure. At noris only remote PBS + S3 Storage exist — neither involves Ceph. Do not confuse the two environments.
+
+## Environment
+
+| Component | Detail |
+|-----------|--------|
+| Environment | **Homelab (self-hosted)** — NOT noris |
+| PVE Master | 10.0.20.10 (SSH: `root@10.0.20.10`, key `~/.ssh/id_ed25519_proxmox`) |
+| PVE Nodes (SSH) | All PVE hosts reachable via `10.0.20.x` (storage VLAN), NOT `10.0.30.x` (management VLAN — SSH key rejected). px1=.10, px2=.20, px3=.30, px4=.40, px5=.50, px6=.60, px7(n5pro)=.70 or .91 |
+| OSDs | 11 total (6 HDD + 5 SSD) |
+| HDD mix | 4× ~3 TB (osd.1, 6, 8, 11) + 2× ~1 TB (osd.7, 10) |
+| EC Profile | `ec41-hdd` (k=4, m=1) → migrating to `ec31-hdd-large` (k=3, m=1) |
+| Pools | `media_ec` (EC), `hdd_disk` (Repl 3), `rbd` (Repl 3), `tm_disks` (Repl 2), `media_meta` (Repl 3), `vm_disks` (Repl 3) |
+| Ratios | nearfull=0.94, backfillfull=0.96, full=0.97 (failsafe hardcoded 0.97) |
+| Monitoring | Grafana CT141 (DS UID=PBFA97CFB590B2093), Prometheus, Alertmanager→Telegram |
+| Ceph Mon Network | `10.0.20.x` (storage VLAN) — NOT `10.0.30.x` (management) |
+
+All Ceph commands run from PVE master (**10.0.20.10** — storage network) unless noted. Non-Ceph nodes (e.g. n5pro) need `ceph.conf` + admin keyring copied and `rbd`/`ceph` commands run from a monitor node.
+
+### SSH Access: 10.0.20.x Works, 10.0.30.x Does NOT
+
+PVE hosts have **two networks**: storage (10.0.20.x) and management (10.0.30.x). SSH with `~/.ssh/id_ed25519_proxmox` works on **10.0.20.x** but is rejected on **10.0.30.x** (key not in authorized_keys). Always use 10.0.20.x IPs for PVE host SSH:
+
+```bash
+# CORRECT — storage network
+ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 "ceph -s"
+
+# WRONG — management network (key rejected)
+ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.10 "ceph -s"  # Permission denied
+```
+
+PVE host IP mapping (storage network): px1=.10, px2=.20, px3=.30, px4=.40, px5=.50, px6=.60, px7=.70, n5pro=.91.
+
+### Fallback: Ceph CLI via K8s ceph-csi Pods
+
+When PVE SSH is unavailable, `ceph` commands can be run from inside K8s `ceph-csi-rbd-nodeplugin` pods (the `csi-rbdplugin` container has `/usr/bin/ceph`). Requires a ceph.conf with monitor addresses and the `csi-rbd-secret` credentials:
+
+```bash
+# On mgmt-runner (10.0.30.124):
+POD=$(kubectl get pods -n kube-system --no-headers | grep ceph-csi-rbd-nodeplugin | head -1 | awk '{print $1}')
+
+# Get cephx key from K8s secret
+USER_KEY=$(kubectl get secret csi-rbd-secret -n kube-system -o jsonpath='{.data.userKey}' | base64 -d)
+
+# Write ceph.conf into the pod and query
+kubectl exec -n kube-system $POD -c csi-rbdplugin -- bash -c "
+cat > /tmp/ceph.conf << 'EOF'
+[global]
+  auth_cluster_required = cephx
+  auth_service_required = cephx
+  auth_client_required = cephx
+  mon host = 10.0.20.50:6789,10.0.20.70:6789,10.0.20.40:6789
+[client.kubernetes]
+  key = $USER_KEY
+EOF
+ceph -c /tmp/ceph.conf --name client.kubernetes status
+"
+```
+
+Monitor addresses are in the `ceph-csi-config` ConfigMap: `kubectl get cm ceph-csi-config -n kube-system -o jsonpath='{.data.config\.json}'`.
+
+**Pitfall:** Not all monitors are always reachable. Check connectivity first (`bash -c "echo > /dev/tcp/IP/6789"` from a Galera VM or K8s node). Only use reachable monitors in the `mon host` list.
+
+**Dual-network pitfall:** PVE hosts have SSH on `10.0.20.x` (storage
+network, port 22 open) but NOT on `10.0.30.x` (management network,
+SSH key rejected). Always use `10.0.20.10` for `ssh root@10.0.20.10`.
+The `10.0.30.x` IPs are for VM bridges, not PVE host management.
+
+**Fallback when PVE SSH is unavailable:** Run `ceph` commands from
+inside a `ceph-csi-rbd-nodeplugin` pod in K8s (see Monitoring section
+below for the full procedure).
+
+⚠️ **Pitfall: PVE hosts are on 10.0.20.x (storage net), NOT 10.0.30.x (management net).** The 10.0.30.x addresses are for VM bridges, not PVE host SSH. Always use `root@10.0.20.10` for Ceph commands. If SSH to 10.0.30.x PVE hosts fails with "Permission denied", you're on the wrong network — check this skill first, not the SSH key.
+
+### Fallback: Ceph CLI via K8s ceph-csi Pods
+
+When PVE host SSH is unavailable (key rotation, host down, network change), `ceph` CLI is available inside the `csi-rbdplugin` container of `ceph-csi-rbd-nodeplugin` pods in K8s. Steps:
+
+1. Find monitor addresses from the `ceph-csi-config` ConfigMap:
+   ```bash
+   kubectl get cm ceph-csi-config -n kube-system -o jsonpath='{.data.config\.json}'
+   # Parse JSON for "monitors" array (e.g. ["10.0.20.10:6789","10.0.20.20:6789",...])
+   ```
+2. Get cephx credentials from the `csi-rbd-secret`:
+   ```bash
+   kubectl get secret csi-rbd-secret -n kube-system -o jsonpath='{.data.userKey}' | base64 -d
+   ```
+3. Write a temporary ceph.conf into the pod and query:
+   ```bash
+   POD=$(kubectl get pods -n kube-system --no-headers | grep ceph-csi-rbd-nodeplugin | head -1 | awk '{print $1}')
+   # Create /tmp/ceph.conf with [global] mon host = <reachable monitors> and [client.kubernetes] key = <userKey>
+   # Then: kubectl exec -n kube-system $POD -c csi-rbdplugin -- ceph -c /tmp/ceph.conf --name client.kubernetes status
+   ```
+
+⚠️ Not all monitors may be reachable from K8s nodes — test connectivity first. Only use monitors that respond on port 6789.
+
+## Decision Trees
+
+### OSD Creation
+
+```
+Is target a standard PVE node?
+├─ YES → pveceph osd create /dev/sdX [--wal_dev /dev/nvme0n1pN]
+│         ⚠️ pveceph CANNOT use LVM device-mapper paths (/dev/dm-*) as WAL/DB
+│         If WAL/DB is on an LV: use ceph-volume directly (see below)
+└─ NO (non-PVE node like n5pro)
+   ├─ Has /etc/ceph/ceph.conf symlink to /etc/pve/ceph.conf?
+   │  ├─ NO → rm /etc/ceph/ceph.conf && ln -s /etc/pve/ceph.conf /etc/ceph/ceph.conf
+   │  └─ YES → proceed
+   ├─ WAL/DB on raw partition? → pveceph osd create /dev/sdX --wal_dev /dev/nvme0n1pN
+   └─ WAL/DB on LVM LV? → ceph-volume lvm prepare --bluestore --data /dev/sdX --block.db /dev/pve/wal-db-osd-X
+                           (WAL auto-collocates on DB — do NOT specify --block.wal separately)
+```
+
+### Recovery / Rebalance
+
+```
+ceph -s shows degraded/misplaced/backfill?
+├─ Any OSD > full_ratio (0.97)?
+│  └─ EMERGENCY: ceph osd set-full-ratio 0.99 → reweight overfull OSD → fix root cause
+├─ backfill_toofull PGs > 0?
+│  └─ Target OSDs above backfillfull_ratio → raise ratio (keep ordering!) or reweight
+├─ Recovery speed too slow?
+│  └─ Tune: osd_max_backfills, osd_recovery_op_priority, osd_recovery_sleep
+│     ⚠️ ceph config set osd X does NOT propagate to running OSDs
+│     Use: ceph tell osd.* injectargs "--osd_max_backfills 3" + verify with ceph daemon osd.N config get
+├─ Recovery stuck at 0 MiB/s with remapped PGs?
+│  └─ Check for sentinel 2147483647 in up-sets → EC with OSD count == k+m → reset all weights to 1.0
+└─ upmap balancer not helping?
+   └─ Balancer skips when >5% misplaced → wait for recovery, then balancer resumes
+```
+
+### EC Pool with Mixed Disk Sizes
+
+```
+EC pool has OSDs of different capacities?
+├─ Small OSDs filling disproportionately?
+│  └─ ROOT CAUSE: EC stores equal-sized chunks regardless of OSD capacity
+│     ceph osd reweight has marginal effect when k+m ≈ total OSD count
+│  SOLUTION:
+│  1. Define device classes: hdd-large, hdd-small
+│  2. Create separate CRUSH root (hdd-root) for all HDDs
+│  3. EC rule: take hdd-root~hdd-large (only large OSDs)
+│  4. Replicated rule: take hdd-root (all HDDs, no class filter)
+│  5. New EC profile with k adjusted to large-OSD count
+│  6. Migrate RBD images to new pool (online via rbd migration)
+└─ Artificially low CRUSH weights on small OSDs?
+   └─ ANTI-PATTERN: causes vicious cycle → reset to true device size after separating classes
+```
+
+## Core Operations
+
+### 1. OSD Lifecycle
+
+#### Create OSD with WAL/DB on NVMe (LVM-based)
+
+```bash
+# 1. Create WAL/DB LV on NVMe
+lvcreate -L 30G -n wal-db-osd-X pve
+
+# 2. Prepare OSD — ONLY --block.db, WAL auto-collocates
+ceph-volume lvm prepare --bluestore \
+  --data /dev/sdX \
+  --block.db /dev/pve/wal-db-osd-X
+
+# 3. Activate (use osd_id + fsid from prepare output)
+ceph-volume lvm activate <osd_id> <osd_uuid>
+
+# 4. Start service
+systemctl enable --now ceph-osd@<osd_id>
+
+# 5. Set CRUSH host + device class
+ceph osd crush set osd.<id> <weight> root=default host=<hostname>
+ceph osd crush set-device-class hdd osd.<id>
+```
+
+⚠️ **NEVER** specify `--block.wal` AND `--block.db` on the same LV — BlueStore opens the device exclusively → "Device or resource busy".
+
+#### Destroy and Rebuild OSD
+
+```bash
+systemctl stop ceph-osd@<id>
+ceph osd down <id>
+ceph osd destroy <id> --yes-i-really-mean-it
+ceph-volume lvm zap /dev/sdX --destroy
+# Then recreate as above
+```
+
+#### Hot-Plug SATA Drive Detection
+
+```bash
+# JMB58x controller doesn't auto-scan
+for h in /sys/class/scsi_host/host{0,1,2,3,4}; do
+  echo "- - -" > ${h}/scan
+done
+sleep 2
+lsblk -d -o NAME,MODEL,SERIAL,SIZE,TRAN,ROTA,STATE | grep -v rbd | grep -v loop
+```
+
+#### Expand BlueFS DB (Online LV Extend + Offline BlueStore Resize)
+
+When BlueFS spillover occurs (RocksDB grows beyond DB LV onto HDD), expand the DB LV:
+
+```bash
+# 1. Extend the LV (online, no OSD stop needed for LVM)
+lvextend -L +20G pve/ceph-db
+
+# 2. Stop the OSD — bluefs-bdev-expand CANNOT run while OSD is active
+#    Error if attempted online: "Device or resource busy" + assert crash
+systemctl stop ceph-osd@<id>
+
+# 3. Expand BlueFS awareness of the new DB size
+ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-<id>
+# Output: "1 : Expanding to 0x... (... GiB)" + "size updated"
+
+# 4. Restart OSD
+systemctl start ceph-osd@<id>
+
+# 5. Verify BlueFS sees the new size
+ceph daemon osd.<id> bluefs stats | head -3
+# Should show: "1 : device size 0x...(XX GiB)"
+
+# 6. Verify OSD is back up in cluster
+ceph osd tree | grep "osd.<id>"
+```
+
+⚠️ **`bluefs-bdev-expand` MUST be run with OSD stopped.** Running it online crashes with `FAILED ceph_assert(r == 0)` — the block device is locked by the running OSD. The `ceph daemon osd.N` admin socket does NOT have an expand command (unlike config get/set). The only path is: stop OSD → bluestore-tool → start OSD.
+
+⚠️ **Spillover warning clears asynchronously.** After expansion, `BLUEFS_SPILLOVER` health warning persists until BlueFS compacts RocksDB back onto the expanded DB device. This happens in the background over minutes to hours.
+
+#### When NOT to Expand (Cosmetic Spillover)
+
+**Spillover < 1 MiB with >50% DB free → cosmetic, do NOT expand.** Tiny spillovers (e.g. 128 KiB) are RocksDB metadata overflow that self-resolves on next compaction. Stopping an OSD for `bluefs-bdev-expand` triggers PG degradation and recovery I/O — worse than the warning itself.
+
+**Expand only when:** spillover > 1 GiB OR DB device >80% full. This indicates sustained RocksDB growth that won't self-resolve.
+
+### 2. CRUSH Topology
+
+#### Device Class Separation
+
+```bash
+# Assign device classes
+ceph osd crush set-device-class hdd-large osd.1 osd.6 osd.8 osd.11
+ceph osd crush set-device-class hdd-small osd.7 osd.10
+
+# Create separate root
+ceph osd crush add-bucket hdd-root root
+
+# Move HDD hosts to new root (⚠️ TRIGGERS REMAPPING!)
+ceph osd crush move <hostname> root=hdd-root
+```
+
+#### CRUSH Rules
+
+```bash
+# EC rule — only large HDDs
+ceph osd erasure-code-profile set ec31-hdd-large \
+  k=3 m=1 crush-root=hdd-root crush-device-class=hdd-large \
+  crush-failure-domain=osd plugin=jerasure technique=reed_sol_van
+ceph osd crush rule create-erasure media_ec_large ec31-hdd-large
+
+# Replicated rule — ALL HDDs (no class filter)
+ceph osd crush rule create-replicated replicated_hdd_all hdd-root host
+```
+
+#### Weight Management
+
+| Command | Scope | Persistence | Use Case |
+|---------|-------|-------------|----------|
+| `ceph osd reweight N X` | Runtime reweight | Lost on restart | Emergency drain of overfull OSD |
+| `ceph osd crush reweight osd.N X` | CRUSH weight | Permanent | Match weight to device size |
+
+**Correct CRUSH weights by device size:**
+
+| Device | Weight |
+|--------|--------|
+| 1 TB | ~0.96 |
+| 2 TB | ~1.82 |
+| 3 TB | ~2.73 |
+| 3.6 TiB | ~3.64 |
+
+⚠️ **Anti-pattern**: Lowering CRUSH weight to "protect" a full OSD. This prevents new PGs but doesn't move existing data → vicious cycle.
+
+### 3. EC Pool Migration
+
+EC pools cannot change k/m in-place. Must create new pool and migrate.
+
+```bash
+# 1. Create new EC pool
+ceph osd pool create media_ec_new 128 128 erasure ec31-hdd-large
+ceph osd pool set media_ec_new ec_overwrites
+ceph osd pool application enable media_ec_new rbd
+
+# 2. Create new metadata pool
+ceph osd pool create media_meta_new 32
+
+# 3. Online RBD migration (VMs keep running!)
+for img in $(rbd ls media_meta); do
+  rbd migration prepare media_meta/${img} media_meta_new/${img} --data-pool media_ec_new
+  rbd migration execute media_meta_new/${img}
+  rbd migration commit media_meta_new/${img}
+done
+
+# 4. Cleanup old pool
+ceph osd pool delete media_ec --yes-i-really-really-mean-it
+ceph osd pool rename media_ec_new media_ec
+```
+
+Timeline: `prepare` (instant) → `execute` (30-60 min per 500 GiB) → `commit` (atomic switch).
+
+### 4. Recovery Tuning
+
+#### Full Recovery Recipe
+
+```bash
+# 1. Unblock backfill (keep ratio ordering: nearfull < backfillfull < full ≤ 0.97)
+ceph osd set-nearfull-ratio 0.94
+ceph osd set-backfillfull-ratio 0.96
+ceph osd set-full-ratio 0.97
+
+# 2. Reweight overfull OSD (emergency, temporary)
+ceph osd reweight <osd_id> 0.8
+
+# 3. Set recovery params (persistent for future OSDs)
+ceph config set osd osd_max_backfills 3
+ceph config set osd osd_recovery_op_priority 10
+ceph config set osd osd_recovery_sleep 0
+
+# 4. Push to running OSDs (config set alone does NOT propagate!)
+ceph tell osd.* injectargs "--osd_max_backfills 3"
+
+# 5. Verify EACH OSD picked up the config
+ceph daemon osd.<N> config get osd_max_backfills
+# Must return "3" — if "1", config not applied
+```
+
+#### Post-Recovery Cleanup
+
+```bash
+# Restore defaults
+ceph config set osd osd_max_backfills 1
+ceph tell osd.* injectargs "--osd_max_backfills 1"
+ceph osd reweight <osd_id> 1.0
+# Ratios: keep at 0.94/0.96/0.97 (safe for mixed-size cluster)
+```
+
+### 5. RBD Orphan Cleanup
+
+```bash
+# 1. Get cluster-wide VM inventory (NOT per-node qm config!)
+pvesh get /cluster/resources --type vm --output-format json | python3 -c "
+import sys, json
+for g in json.load(sys.stdin): print(g['vmid'])
+" | sort -n | tr '\n' ' '
+
+# 2. Fast orphan check for specific image (O(1))
+grep -rl "vm-NNN" /etc/pve/qemu-server/ /etc/pve/lxc/ 2>/dev/null
+# Exit 0 = referenced. Exit 1 = orphan.
+
+# 3. Delete with protected snapshot
+rbd snap unprotect "hdd_disk/base-NNN-disk-0"@__base__
+rbd snap purge "hdd_disk/base-NNN-disk-0"
+rbd rm "hdd_disk/base-NNN-disk-0"
+```
+
+⚠️ **Always use `pvesh get /cluster/resources --type vm`** for VM existence — per-node `qm config` only sees local VMs → false positives.
+
+⚠️ **Non-Ceph nodes cannot run `rbd` commands** — run from a monitor node (e.g. proxmox7 = 10.0.20.70).
+
+### 6. PG Management
+
+```bash
+# Reduce PG count (must disable autoscale first!)
+ceph osd pool set <pool> pg_autoscale_mode off
+ceph osd pool set <pool> pgp_num 32
+ceph osd pool set <pool> pg_num 32
+# Merge happens AFTER all PGs reach active+clean
+
+# Raise mon_max_pg_per_osd (temporary relief)
+ceph config set mon mon_max_pg_per_osd 400
+ceph tell mon.* injectargs --mon_max_pg_per_osd=400
+```
+
+## Pitfalls
+
+### Ratio Ordering (HEALTH_ERR)
+
+`nearfull < backfillfull < full ≤ osd_failsafe_full_ratio (0.97 hardcoded)`. Violation → `HEALTH_ERR: OSD_OUT_OF_ORDER_FULL`. Always set all three ratios together.
+
+### injectargs Reliability
+
+`ceph tell osd.* injectargs` returns empty `{}` which looks like failure but may have succeeded. **Always verify** with `ceph daemon osd.N config get <param>`. Most reliable: `ceph config set` (persistent) + `injectargs` (runtime) + verify.
+
+### EC Reweight with Minimal OSDs
+
+**Never reweight OSDs in EC pools when OSD count == k+m.** CRUSH places sentinel `2147483647` ("no OSD available") in up-sets → remapped PGs can never recover. Fix: reset all weights to 1.0. General rule: only reweight when OSD count > k+m.
+
+### pveceph LVM WAL/DB Limitation
+
+`pveceph osd create --wal_dev /dev/pve/wal-db-osd-X` fails with "unable to get device info for '/dev/dm-2'". pveceph cannot resolve LVM device-mapper symlinks. Use `ceph-volume lvm prepare` directly with `--block.db`.
+
+### BlueStore WAL/DB Same Device
+
+Specifying `--block.wal` AND `--block.db` on the same LV → "Device or resource busy". BlueStore opens WAL exclusively. Solution: specify ONLY `--block.db` — WAL collocates automatically.
+
+### ceph config set Doesn't Propagate to Running OSDs
+
+`ceph config set osd osd_max_backfills 3` updates MON config store but running OSDs keep old value. Must use `ceph tell osd.* injectargs` for runtime change, or restart OSDs. Always verify with `ceph daemon osd.N config get`.
+
+### backfillfull Is Derived, Not Settable
+
+Pool-level `backfillfull` flag is derived from OSD conditions, not a settable pool flag. `ceph osd pool unset <pool> backfillfull` → EINVAL. Fix: raise `backfillfull_ratio` or reweight overfull OSDs.
+
+### Ghost RBD Images
+
+Images appearing in `rbd ls` but `rbd info` returns empty → header gone, data objects remain. Need `rados rm` purge, not `rbd rm`.
+
+### pvesm list Fails When Pool Backfillfull
+
+`pvesm list hdd_disk` returns empty when Ceph pool is in backfillfull state. Fall back to `rbd ls <pool>` for image listing.
+
+## Monitoring
+
+```bash
+# Quick health
+ceph -s
+
+# Per-OSD utilization
+ceph osd df
+
+# PG state summary
+ceph pg dump pgs_brief | awk '{print $2}' | sort | uniq -c | sort -rn
+
+# Recovery progress
+ceph -s | grep -E "recovery:|misplaced"
+
+# Balancer status
+ceph balancer status
+
+# Slow ops
+ceph health detail | grep -A5 "slow"
+```
+
+### Ceph Status via ceph-csi Pod (Fallback When PVE SSH Unavailable)
+
+When PVE hosts are unreachable via SSH (key rotation, network issue,
+host down), but K8s nodes can reach the Ceph monitors, run `ceph`
+commands from inside a `ceph-csi-rbd-nodeplugin` pod:
+
+```bash
+# 1. Find the ceph-csi pod (on mgmt-runner or any K8s node with kubectl)
+POD=$(kubectl get pods -n kube-system --no-headers | \
+  grep ceph-csi-rbd-nodeplugin | head -1 | awk '{print $1}')
+
+# 2. Get monitor addresses from the ceph-csi-config ConfigMap
+MONS=$(kubectl get cm ceph-csi-config -n kube-system \
+  -o jsonpath='{.data.config\.json}' | \
+  python3 -c "import sys,json; print(','.join(json.load(sys.stdin)[0]['monitors']))")
+
+# 3. Get the cephx key from the csi-rbd-secret
+KEY=$(kubectl get secret csi-rbd-secret -n kube-system \
+  -o jsonpath='{.data.userKey}' | base64 -d)
+
+# 4. Write ceph.conf + keyring into the pod, then run ceph commands
+kubectl exec -n kube-system $POD -c csi-rbdplugin -- bash -c "
+cat > /tmp/ceph.conf << EOF
+[global]
+  auth_cluster_required = cephx
+  auth_service_required = cephx
+  auth_client_required = cephx
+  mon host = $MONS
+[client.kubernetes]
+  key = $KEY
+"
+ceph -c /tmp/ceph.conf --name client.kubernetes status
+ceph -c /tmp/ceph.conf --name client.kubernetes osd df
+"
+```
+
+**Key details:**
+- The `csi-rbd-plugin` container has `/usr/bin/ceph` installed
+- Monitor addresses are in the `ceph-csi-config` ConfigMap (may be on
+  a different subnet like `10.0.20.x` — the storage VLAN)
+- The cephx key is in the `csi-rbd-secret` Secret (field `userKey`)
+- Not all monitors need to be up — 3 of 7 was sufficient in practice
+- `client.kubernetes` has limited permissions (reads status, OSD df,
+  pool df) but NOT admin operations (no `ceph osd set-full-ratio` etc.)
+
+**Pitfall:** Monitor addresses in `ceph-csi-config` may be on a
+different subnet (e.g. `10.0.20.x` storage VLAN) than the management
+network (`10.0.30.x`). K8s pods can usually reach both, but Galera VMs
+may only see `10.0.30.x`.
+
+**Pitfall:** The `ceph -c /tmp/ceph.conf` approach requires writing
+the config file into the pod. If `kubectl cp` is blocked, use
+`kubectl exec -- bash -c "cat > /tmp/ceph.conf << EOF ..."` via a
+heredoc through SSH to the mgmt-runner.
+
+## Recovery Monitoring Cron Pattern
+
+```
+Schedule: every 30 minutes
+Report: PGs remapped/backfilling, % misplaced, OSD utilization, recovery speed
+Alert: any OSD down or health ERR
+Announce "COMPLETE" when 0 remapped PGs
+```
+
+## Cross-References
+
+Detailed session logs and edge cases in `proxmox-ve-administration` skill references:
+- `references/ceph-osd-create-hotplug-2026-07.md` — Full OSD hot-plug walkthrough
+- `references/ceph-ec-mixed-size-optimization-2026-07.md` — Complete EC migration plan
+- `references/ceph-recovery-acceleration-2026-07.md` — Advanced recovery tuning
+- `references/ceph-crush-weight-optimization-2026-07.md` — Weight alignment for mixed sizes
+- `references/ceph-pg-management-2026-07.md` — PG count reduction + EC reweight pitfall
+- `references/ceph-pool-full-recovery-2026-07.md` — Pool-full diagnosis + OSD preparation
+- `references/ceph-orphaned-rbd-cleanup-2026-07.md` — Full orphan audit methodology
+- `references/ceph-recovery-ec-reweight-2026-07.md` — EC reweight stagnation fix
+- `references/ceph-pg-osd-management-2026-07.md` — PG/OSD rebalancing
+- `references/ceph-ec-pool-and-service-migration-2026-07.md` — EC pool + service migration
+- `references/ceph-pg-reduction-and-osd-rebalance-2026-07.md` — PG reduction + rebalance
+- `references/rbd-pool-migration-2026-07.md` — Offline RBD pool migration
+- `references/hdd-selection-for-osd-2026-07.md` — Drive selection criteria
+- `references/bluefs-db-expansion-2026-07.md` — BlueFS DB LV expansion + cosmetic spillover assessment
+
+Compound-learning solution docs in `~/docs/solutions/`:
+- `bug-fixes/2026-07-13-pveceph-wal-lvm-device-mapper-limitation.md`
+- `bug-fixes/2026-07-13-ceph-ratio-ordering-constraint.md`
+- `architecture/2026-07-13-ec-mixed-disk-sizes-optimization.md`
+- `tooling/2026-07-13-injectargs-vs-config-set-reliability.md`