feat: add home-assistant-dashboard-conventions skill + update multiple skills

- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
2026-07-14 18:35:16 +00:00
parent e9cc106625
commit 01bd921ced
37 changed files with 6133 additions and 126 deletions
@@ -0,0 +1,103 @@
+# BlueFS DB Expansion — Session Log (2026-07-13)
+
+## Scenario
+
+osd.7 (982 GiB HDD, proxmox7) triggered `BLUEFS_SPILLOVER` warning — RocksDB grew beyond its 50 GiB DB LV on NVMe and spilled onto the slow HDD device. DB was 50 GiB, only 2.2 GiB actually used, but BlueFS spillover is about the RocksDB *logical* size exceeding the DB device, not physical usage.
+
+## Diagnosis
+
+```bash
+# Check spillover target
+ceph health detail | grep -A2 SPILLOVER
+# Shows which OSD spilled and how much
+
+# Check DB device size and usage
+ceph daemon osd.7 bluefs stats
+# Line 1: DB device size + usage
+# Line 2: Slow (HDD) device size + usage
+# If DB device is full → spillover to slow device
+
+# Check DB LV and VG free space
+ssh root@<osd-host> 'lvs --units g | grep ceph-db && vgs --units g | grep pve'
+```
+
+## Assessment Heuristic
+
+**Spillover < 1 MiB with >50% DB free → cosmetic, do NOT expand.**
+The spillover is often a tiny RocksDB metadata overflow that will self-resolve on compaction. Stopping an OSD for `bluefs-bdev-expand` causes unnecessary downtime (PGs degraded, recovery triggered).
+
+**Spillover > 1 GiB OR DB device >80% full → expand.**
+This indicates sustained RocksDB growth that won't self-resolve.
+
+## Expansion Procedure (Verified Working)
+
+osd.7 on proxmox7 (10.0.20.70), DB LV = `pve/ceph-db`:
+
+```bash
+# 1. Extend LV (online, no OSD stop needed for LVM)
+lvextend -L +20G pve/ceph-db
+# Output: "Size of logical volume pve/ceph-db changed from 50.00 GiB to 70.00 GiB"
+
+# 2. Stop OSD — bluefs-bdev-expand CANNOT run while OSD is active
+systemctl stop ceph-osd@7
+sleep 2
+
+# 3. Expand BlueFS awareness
+ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-7
+# Output:
+# "1 : device size 0x1180000000(70 GiB) : using 0x89000000(2.1 GiB)"
+# "Expanding DB/WAL..."
+# "1 : Expanding to 0x1180000000(70 GiB)"
+# "1 : size updated to 0x1180000000(70 GiB)"
+
+# 4. Restart OSD
+systemctl start ceph-osd@7
+
+# 5. Wait for OSD to come up in cluster (10-15 seconds)
+ceph osd tree | grep "osd.7"
+# Should show "up"
+
+# 6. Verify BlueFS sees new size
+ceph daemon osd.7 bluefs stats | head -3
+# Should show: "1 : device size 0x1180000000(70 GiB)"
+```
+
+## What Happened After
+
+- **osd.7 spillover cleared** — warning moved to osd.8 (128 KiB, cosmetic)
+- **osd.7 utilization dropped** from 86.1% → 78.9% (backfill continued, OSD healthier)
+- **osd.8 spillover assessed**: 128 KiB spill, 28 GiB DB free → NO action needed
+- **BlueFS spillover warning persists** in `ceph health` until RocksDB compacts back onto DB device — background process, minutes to hours
+
+## Pitfall: Online bluefs-bdev-expand Crashes
+
+Running `ceph-bluestore-tool bluefs-bdev-expand` while OSD is active:
+```
+bdev(...) open open got: (16) Device or resource busy
+bluestore(...) _lock_fsid failed to lock fsid (is another ceph-osd still running?)
+FAILED ceph_assert(r == 0)
+```
+The tool tries to open the block device exclusively, which the running OSD already holds. Must stop OSD first.
+
+## Pitfall: ceph daemon admin socket Has No Expand Command
+
+`ceph daemon osd.7 help` shows bluefs commands but NO `expand` or `bdev-expand` option. Available bluefs commands: `bluefs stats`, `bluefs files list`, `bluefs debug_inject_read_zeros`. The only path to expand is the offline `ceph-bluestore-tool`.
+
+## osd.8 Case Study — When NOT to Expand
+
+osd.8 on host `ubuntu` (10.0.30.100):
+- DB LV: `ceph-db-vg/osd-8-db` = 30 GiB on Crucial MX300 SSD (`sda3`)
+- DB usage: 2.0 GiB of 30 GiB (93% free)
+- Spillover: 128 KiB (trivial)
+- VG free: 35.67 GiB across two PVs (`sda3` 1.89 GiB free, `sda4` 33.78 GiB free)
+
+**Decision: NO expansion.** 128 KiB spillover with 28 GiB DB free is cosmetic. Stopping the OSD would trigger PG degradation and recovery I/O for zero practical benefit. The spillover will likely self-resolve on next RocksDB compaction.
+
+## Cross-Host SSH Pattern
+
+OSDs span multiple PVE hosts. The Ceph commands (`ceph -s`, `ceph osd df`) run from any monitor node (10.0.20.10). But `ceph-bluestore-tool` and `systemctl` must run on the OSD's host. Use `ceph osd metadata N | grep hostname` to find the host, then SSH there:
+
+```bash
+HOST=$(ssh root@10.0.20.10 "ceph osd metadata 7 | python3 -c \"import sys,json; print(json.load(sys.stdin)['hostname'])\"")
+ssh root@$HOST 'systemctl stop ceph-osd@7 && ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-7 && systemctl start ceph-osd@7'
+```