feat: add home-assistant-dashboard-conventions skill + update multiple skills

- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards)
- Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills
- New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
This commit is contained in:
Debian
2026-07-14 18:35:16 +00:00
parent e9cc106625
commit 01bd921ced
37 changed files with 6133 additions and 126 deletions
@@ -47,8 +47,18 @@ After resetting, recovery resumes (67 MiB/s, 16 obj/s observed).
- **Replicated pools** (size=3): reweighting is safe — CRUSH redirects
to other OSDs in the failure domain.
- **Erasure-coded pools**: reweighting is only safe when there are MORE
OSDs than `k+m`. With exactly `k+m` OSDs, reweighting traps PGs.
- **Erasure-coded pools**: `crush reweight` (permanent CRUSH weight change)
is only safe when there are MORE OSDs than `k+m`. With exactly `k+m`
OSDs, `crush reweight` traps PGs (2147483647 in up-set).
- **⚠️ `osd reweight` (temporary) CAN drain EC OSDs** (2026-07-12):
Unlike `crush reweight`, `ceph osd reweight N 0.5` sets a temporary
runtime override that adjusts PG placement probability WITHOUT
changing CRUSH weights. This means CRUSH still considers all OSDs
available — no 2147483647 trap. Tested on osd.10 (EC4+1, exactly 5
HDD OSDs): reweight to 0.5 drained from 93%→91%, backfill started
at 30 MiB/s, 5 of 6 `backfill_toofull` PGs unblocked. **Prerequisite:**
`crush reweight` must be at 1.0 (default). If `crush reweight` was
previously changed, revert it to 1.0 FIRST, then use `osd reweight`.
- **Long-term fix**: add more HDD OSDs (e.g. on the ubuntu host with
2×2.8 TB spare) to give CRUSH room to redistribute.
@@ -95,14 +105,63 @@ flagged until its utilization drops below the new threshold naturally
### Mitigation
- **Patience**: Once the reweight trap is fixed and recovery proceeds,
- **`osd reweight` (temporary) WORKS for EC4+1 drain** (2026-07-12):
`ceph osd reweight 10 0.5` successfully drained osd.10 from 93.14%→90.90%
and unblocked 5 of 6 `backfill_toofull` PGs. This is a runtime
override, NOT a CRUSH weight change — CRUSH still sees all OSDs,
preventing the 2147483647 trap. Recovery proceeded at 30 MiB/s.
Steps: (1) ensure `crush reweight` is 1.0, (2) `ceph osd reweight
10 0.5`, (3) `ceph osd set-backfillfull-ratio 0.95` to unblock
remaining PGs, (4) wait for recovery, (5) `ceph osd reweight 10 1.0`
to restore.
- **Aggressive drain: reweight 0.3** (2026-07-12 session 2):
When 0.5 wasn't enough (osd.10 still at 89%, 1 PG stubbornly
`backfill_toofull`), dropping to `ceph osd reweight 10 0.3` increased
misplaced objects from 12%→16% (more PGs queued for relocation) and
accelerated the drain. Combined with `set-backfillfull-ratio 0.97`.
**Reweight progression: 0.5 → 0.3 if 0.5 doesn't drain fast enough.**
Remember to restore to 1.0 after recovery completes.
### ⚠️ Correct Command Syntax for backfillfull_ratio (2026-07-12)
Multiple command forms were tried — only ONE works. Save time by using
the correct one first:
```bash
# ✅ CORRECT — hyphenated, no underscore:
ceph osd set-backfillfull-ratio 0.97
# ❌ WRONG — underscore variant (EINVAL):
ceph osd set_backfill_full_ratio 0.97
# ❌ WRONG — config set (rejected as "special"):
ceph config set mon mon_osd_backfillfull_ratio 0.97
# → "mon_osd_backfillfull_ratio is special and cannot be stored by the mon"
# ❌ WRONG — injectargs with wrong key name:
ceph tell osd.10 injectargs "--osd_backfill_full_ratio 0.97"
# → "failed to parse arguments"
# ❌ WRONG — osd set with wrong key:
ceph osd set backfill_full_ratio 0.97
# → "Invalid command: backfill_full_ratio not in full|pause|..."
```
Similarly for full_ratio and nearfull_ratio:
```bash
ceph osd set-full-ratio 0.97 # ✅ hyphenated
ceph osd set-nearfull-ratio 0.95 # ✅ hyphenated
```
- **Patience**: Once the reweight takes effect and recovery proceeds,
data redistributes and osd.10 utilization drops, eventually clearing
the flag.
the backfillfull flag.
- **PG merge**: Reducing PG count (128→32) eliminates some remapped PGs
entirely, reducing the number of PGs that need to backfill to the
stuck OSD.
- **Do NOT reweight osd.10** to reduce its load — with EC4+1 and only 5
HDD OSDs, reweighting makes things worse (see above).
- **Do NOT use `crush reweight`** to reduce osd.10's load — with EC4+1
and only 5 HDD OSDs, permanent CRUSH reweighting makes things worse
(see EC4+1 Reweight Trap above). Use `osd reweight` (temporary)
instead.
## PG Merge (128 → 32)
@@ -547,3 +606,27 @@ I/O-intensive operations that write to pools containing that OSD. Either:
`seafile-api` skill → `references/seafile-immich-backup-strategy.md`.
- CT111 (Seafile) backup on noris_s3 showed 0 MB (empty/failed) on 2026-07-07.
Previous valid backup: 558 GB on 2026-07-06.
## Session Log — 2026-07-12 (Continued)
- **osd.10 drain via `osd reweight` SUCCESSFUL**: `ceph osd reweight 10 0.5`
drained osd.10 from 93.14%→90.90%. Combined with `set-backfillfull-ratio 0.95`,
unblocked 5 of 6 `backfill_toofull` PGs. Recovery at 30 MiB/s, 13% misplaced.
This CONTRADICTS the earlier advice "Do NOT reweight osd.10" — that was
correct for `crush reweight` (permanent, triggers 2147483647 trap) but
WRONG for `osd reweight` (temporary runtime override, safe with EC4+1).
Reference patched accordingly.
- **Orphaned RBD cleanup completed**: vm-143-disk-0 (50G), base-101-disk-0
(20G), base-102-disk-0 (20G), vm-902-disk-0 (40G), vm-201-disk-0,
vm-202-disk-0, test-dummy all deleted. base-101/102 required
`rbd snap unprotect @__base__` before `rbd snap purge` + `rbd rm`.
csi-vol-e596 has stale watcher (10.0.30.63, no K8s PVC found) —
pending natural watcher timeout. ~161 GiB freed.
- **K8s worker anti-colocation migration completed**: VM 132→proxmox1,
VM 131→proxmox4, VM 128→n5pro via `qm migrate --online` (HA-managed,
async). Required expanding `k8s-prefer-n5pro` node-affinity rule to
include all 8 nodes first. All 6 K8s VMs now on 6 separate nodes.
- **hdd_disk→vm_disks migration feasibility**: vm_disks has only 86 GiB
free (75% full). Small CTs (vm-127 4G, vm-121 4G, base-9000 3G,
vm-9001 10G = ~21G total) are candidates. Large K8s worker disks
(80G×3) stay on hdd_disk (correct HDD-tier placement).