feat: add home-assistant-dashboard-conventions skill + update multiple skills
- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
This commit is contained in:
@@ -47,8 +47,18 @@ After resetting, recovery resumes (67 MiB/s, 16 obj/s observed).
|
||||
|
||||
- **Replicated pools** (size=3): reweighting is safe — CRUSH redirects
|
||||
to other OSDs in the failure domain.
|
||||
- **Erasure-coded pools**: reweighting is only safe when there are MORE
|
||||
OSDs than `k+m`. With exactly `k+m` OSDs, reweighting traps PGs.
|
||||
- **Erasure-coded pools**: `crush reweight` (permanent CRUSH weight change)
|
||||
is only safe when there are MORE OSDs than `k+m`. With exactly `k+m`
|
||||
OSDs, `crush reweight` traps PGs (2147483647 in up-set).
|
||||
- **⚠️ `osd reweight` (temporary) CAN drain EC OSDs** (2026-07-12):
|
||||
Unlike `crush reweight`, `ceph osd reweight N 0.5` sets a temporary
|
||||
runtime override that adjusts PG placement probability WITHOUT
|
||||
changing CRUSH weights. This means CRUSH still considers all OSDs
|
||||
available — no 2147483647 trap. Tested on osd.10 (EC4+1, exactly 5
|
||||
HDD OSDs): reweight to 0.5 drained from 93%→91%, backfill started
|
||||
at 30 MiB/s, 5 of 6 `backfill_toofull` PGs unblocked. **Prerequisite:**
|
||||
`crush reweight` must be at 1.0 (default). If `crush reweight` was
|
||||
previously changed, revert it to 1.0 FIRST, then use `osd reweight`.
|
||||
- **Long-term fix**: add more HDD OSDs (e.g. on the ubuntu host with
|
||||
2×2.8 TB spare) to give CRUSH room to redistribute.
|
||||
|
||||
@@ -95,14 +105,63 @@ flagged until its utilization drops below the new threshold naturally
|
||||
|
||||
### Mitigation
|
||||
|
||||
- **Patience**: Once the reweight trap is fixed and recovery proceeds,
|
||||
- **`osd reweight` (temporary) WORKS for EC4+1 drain** (2026-07-12):
|
||||
`ceph osd reweight 10 0.5` successfully drained osd.10 from 93.14%→90.90%
|
||||
and unblocked 5 of 6 `backfill_toofull` PGs. This is a runtime
|
||||
override, NOT a CRUSH weight change — CRUSH still sees all OSDs,
|
||||
preventing the 2147483647 trap. Recovery proceeded at 30 MiB/s.
|
||||
Steps: (1) ensure `crush reweight` is 1.0, (2) `ceph osd reweight
|
||||
10 0.5`, (3) `ceph osd set-backfillfull-ratio 0.95` to unblock
|
||||
remaining PGs, (4) wait for recovery, (5) `ceph osd reweight 10 1.0`
|
||||
to restore.
|
||||
- **Aggressive drain: reweight 0.3** (2026-07-12 session 2):
|
||||
When 0.5 wasn't enough (osd.10 still at 89%, 1 PG stubbornly
|
||||
`backfill_toofull`), dropping to `ceph osd reweight 10 0.3` increased
|
||||
misplaced objects from 12%→16% (more PGs queued for relocation) and
|
||||
accelerated the drain. Combined with `set-backfillfull-ratio 0.97`.
|
||||
**Reweight progression: 0.5 → 0.3 if 0.5 doesn't drain fast enough.**
|
||||
Remember to restore to 1.0 after recovery completes.
|
||||
|
||||
### ⚠️ Correct Command Syntax for backfillfull_ratio (2026-07-12)
|
||||
|
||||
Multiple command forms were tried — only ONE works. Save time by using
|
||||
the correct one first:
|
||||
|
||||
```bash
|
||||
# ✅ CORRECT — hyphenated, no underscore:
|
||||
ceph osd set-backfillfull-ratio 0.97
|
||||
|
||||
# ❌ WRONG — underscore variant (EINVAL):
|
||||
ceph osd set_backfill_full_ratio 0.97
|
||||
|
||||
# ❌ WRONG — config set (rejected as "special"):
|
||||
ceph config set mon mon_osd_backfillfull_ratio 0.97
|
||||
# → "mon_osd_backfillfull_ratio is special and cannot be stored by the mon"
|
||||
|
||||
# ❌ WRONG — injectargs with wrong key name:
|
||||
ceph tell osd.10 injectargs "--osd_backfill_full_ratio 0.97"
|
||||
# → "failed to parse arguments"
|
||||
|
||||
# ❌ WRONG — osd set with wrong key:
|
||||
ceph osd set backfill_full_ratio 0.97
|
||||
# → "Invalid command: backfill_full_ratio not in full|pause|..."
|
||||
```
|
||||
|
||||
Similarly for full_ratio and nearfull_ratio:
|
||||
```bash
|
||||
ceph osd set-full-ratio 0.97 # ✅ hyphenated
|
||||
ceph osd set-nearfull-ratio 0.95 # ✅ hyphenated
|
||||
```
|
||||
- **Patience**: Once the reweight takes effect and recovery proceeds,
|
||||
data redistributes and osd.10 utilization drops, eventually clearing
|
||||
the flag.
|
||||
the backfillfull flag.
|
||||
- **PG merge**: Reducing PG count (128→32) eliminates some remapped PGs
|
||||
entirely, reducing the number of PGs that need to backfill to the
|
||||
stuck OSD.
|
||||
- **Do NOT reweight osd.10** to reduce its load — with EC4+1 and only 5
|
||||
HDD OSDs, reweighting makes things worse (see above).
|
||||
- **Do NOT use `crush reweight`** to reduce osd.10's load — with EC4+1
|
||||
and only 5 HDD OSDs, permanent CRUSH reweighting makes things worse
|
||||
(see EC4+1 Reweight Trap above). Use `osd reweight` (temporary)
|
||||
instead.
|
||||
|
||||
## PG Merge (128 → 32)
|
||||
|
||||
@@ -547,3 +606,27 @@ I/O-intensive operations that write to pools containing that OSD. Either:
|
||||
`seafile-api` skill → `references/seafile-immich-backup-strategy.md`.
|
||||
- CT111 (Seafile) backup on noris_s3 showed 0 MB (empty/failed) on 2026-07-07.
|
||||
Previous valid backup: 558 GB on 2026-07-06.
|
||||
|
||||
## Session Log — 2026-07-12 (Continued)
|
||||
|
||||
- **osd.10 drain via `osd reweight` SUCCESSFUL**: `ceph osd reweight 10 0.5`
|
||||
drained osd.10 from 93.14%→90.90%. Combined with `set-backfillfull-ratio 0.95`,
|
||||
unblocked 5 of 6 `backfill_toofull` PGs. Recovery at 30 MiB/s, 13% misplaced.
|
||||
This CONTRADICTS the earlier advice "Do NOT reweight osd.10" — that was
|
||||
correct for `crush reweight` (permanent, triggers 2147483647 trap) but
|
||||
WRONG for `osd reweight` (temporary runtime override, safe with EC4+1).
|
||||
Reference patched accordingly.
|
||||
- **Orphaned RBD cleanup completed**: vm-143-disk-0 (50G), base-101-disk-0
|
||||
(20G), base-102-disk-0 (20G), vm-902-disk-0 (40G), vm-201-disk-0,
|
||||
vm-202-disk-0, test-dummy all deleted. base-101/102 required
|
||||
`rbd snap unprotect @__base__` before `rbd snap purge` + `rbd rm`.
|
||||
csi-vol-e596 has stale watcher (10.0.30.63, no K8s PVC found) —
|
||||
pending natural watcher timeout. ~161 GiB freed.
|
||||
- **K8s worker anti-colocation migration completed**: VM 132→proxmox1,
|
||||
VM 131→proxmox4, VM 128→n5pro via `qm migrate --online` (HA-managed,
|
||||
async). Required expanding `k8s-prefer-n5pro` node-affinity rule to
|
||||
include all 8 nodes first. All 6 K8s VMs now on 6 separate nodes.
|
||||
- **hdd_disk→vm_disks migration feasibility**: vm_disks has only 86 GiB
|
||||
free (75% full). Small CTs (vm-127 4G, vm-121 4G, base-9000 3G,
|
||||
vm-9001 10G = ~21G total) are candidates. Large K8s worker disks
|
||||
(80G×3) stay on hdd_disk (correct HDD-tier placement).
|
||||
|
||||
Reference in New Issue
Block a user