feat: add home-assistant-dashboard-conventions skill + update multiple skills
- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
This commit is contained in:
Debian
@@ -54,6 +54,25 @@ ceph daemon osd.7 config get osd_max_backfills
|
||||
# Must return "3" — if still "1", the config hasn't been applied.
|
||||
```
|
||||
|
||||
### UPDATE 2026-07-13: `ceph tell osd.* config set` DOES Work
|
||||
|
||||
In a follow-up session, `ceph tell osd.* config set osd_max_backfills 3`
|
||||
DID successfully apply to running OSDs. Verification:
|
||||
|
||||
```bash
|
||||
ceph tell osd.11 config get osd_max_backfills
|
||||
# Returns: {"osd_max_backfills": "3"}
|
||||
```
|
||||
|
||||
The difference from the earlier session: `ceph config set osd osd_max_backfills 3`
|
||||
(MON-level persistent config) was set FIRST, then `ceph tell osd.* config set`
|
||||
was used to push it to running OSDs. The combination worked. Injectargs
|
||||
returned empty `{}` (appeared to fail) but the config took effect.
|
||||
|
||||
**Recommendation**: Use BOTH `ceph config set` (persistent for future OSDs)
|
||||
AND `ceph tell osd.* config set` (runtime for current OSDs). Then verify
|
||||
with `ceph tell osd.N config get`.
|
||||
|
||||
### `osd_recovery_max_active` — NOT effectively tunable
|
||||
|
||||
`ceph config set osd osd_recovery_max_active 3` accepts the value but OSDs report 0 (meaning auto/default). `injectargs` also shows empty. This appears to be a Ceph limitation — the parameter is managed internally. Do not waste time trying to force it.
|
||||
@@ -145,6 +164,127 @@ Command: ssh to monitor node, run `ceph status`, report:
|
||||
- Announce "COMPLETE" when 0 remapped PGs
|
||||
```
|
||||
|
||||
## Pitfalls: Setting backfill_full_ratio (2026-07-12)
|
||||
|
||||
Three commands FAIL when trying to raise the backfill full ratio — only
|
||||
ONE works:
|
||||
|
||||
```bash
|
||||
# ✅ CORRECT — CLI command with hyphen (not underscore):
|
||||
ceph osd set-backfillfull-ratio 0.97
|
||||
# Verify: ceph osd dump | grep backfillfull_ratio → "backfillfull_ratio 0.97"
|
||||
|
||||
# ❌ WRONG — underscore variant:
|
||||
ceph osd set-backfillfullratio 0.97
|
||||
# → "no valid command found" (EINVAL)
|
||||
|
||||
# ❌ WRONG — MON config set:
|
||||
ceph config set mon mon_osd_backfillfull_ratio 0.97
|
||||
# → "mon_osd_backfillfull_ratio is special and cannot be stored by the mon"
|
||||
|
||||
# ❌ WRONG — injectargs on specific OSD:
|
||||
ceph tell osd.10 injectargs "--osd_backfill_full_ratio 0.97"
|
||||
# → "failed to parse arguments: --osd_backfill_full_ratio,0.97" (EINVAL)
|
||||
# injectargs only accepts runtime-tunable OSD params, not full-ratio thresholds.
|
||||
```
|
||||
|
||||
The ratio is a MON-level OSD map parameter, not a per-OSD config. It can
|
||||
ONLY be changed via `ceph osd set-backfillfull-ratio` (CLI command, not
|
||||
config set). Similarly: `ceph osd set-nearfull-ratio` and
|
||||
`ceph osd set-full-ratio` for the other thresholds.
|
||||
|
||||
## Pitfalls: Pool-Level `backfillfull` Flag (2026-07-13)
|
||||
|
||||
When pools show the `backfillfull` flag in `ceph osd dump`, attempting
|
||||
to unset it at the pool level FAILS:
|
||||
|
||||
```bash
|
||||
# ❌ WRONG — no such pool flag:
|
||||
ceph osd pool unset 6 backfillfull
|
||||
# → "no valid command found; 1 closest matches: osd pool unset noautoscale"
|
||||
|
||||
# ❌ WRONG — all pool IDs tried, all fail:
|
||||
for pool in 1 5 6 7 8 9 10; do ceph osd pool unset $pool backfillfull; done
|
||||
# All return EINVAL
|
||||
```
|
||||
|
||||
The `backfillfull` flag on a pool is a DERIVED condition — it appears when
|
||||
any OSD in the pool's up/acting set is above the `backfillfull_ratio`.
|
||||
It is NOT a settable pool flag. To clear it, fix the underlying OSD
|
||||
condition: raise `backfillfull_ratio` or reweight the overfull OSD.
|
||||
|
||||
```bash
|
||||
# ✅ CORRECT — raise the ratio (clears backfillfull on all pools):
|
||||
ceph osd set-backfillfull-ratio 0.99
|
||||
|
||||
# ✅ ALSO — reweight the overfull OSD:
|
||||
ceph osd reweight 7 0.8
|
||||
```
|
||||
|
||||
## Pitfalls: `osd_recovery_max_active` Behavior (2026-07-13)
|
||||
|
||||
`ceph config set osd osd_recovery_max_active 3` accepts the value and
|
||||
`ceph config get osd osd_recovery_max_active` returns 3. However,
|
||||
`ceph tell osd.N config get osd_recovery_max_active` may return empty
|
||||
string — the OSD interprets 0 or empty as "auto/default". Despite this,
|
||||
setting it alongside `osd_max_backfills` appeared to help overall. Don't
|
||||
rely on this parameter alone — use `osd_max_backfills` as the primary lever.
|
||||
|
||||
## Pitfalls: `full ratio(s) out of order` (2026-07-13)
|
||||
|
||||
When adjusting full/nearfull/backfillfull ratios, you MUST preserve the
|
||||
ordering: `nearfull_ratio < backfillfull_ratio < full_ratio <= osd_failsafe_full_ratio`.
|
||||
|
||||
The `osd_failsafe_full_ratio` is hardcoded at **0.97** and cannot be changed.
|
||||
If you set `full_ratio` above 0.97, the cluster goes to `HEALTH_ERR`:
|
||||
|
||||
```
|
||||
[ERR] OSD_OUT_OF_ORDER_FULL: full ratio(s) out of order
|
||||
osd_failsafe_full_ratio (0.97) < full_ratio (0.98), increased
|
||||
```
|
||||
|
||||
**Safe ratio combinations for a cluster with an OSD at 95%:**
|
||||
|
||||
```bash
|
||||
# ✅ CORRECT — all ratios below failsafe (0.97), properly ordered:
|
||||
ceph osd set-nearfull-ratio 0.94
|
||||
ceph osd set-backfillfull-ratio 0.96
|
||||
ceph osd set-full-ratio 0.97 # == failsafe, OK
|
||||
|
||||
# ❌ WRONG — full_ratio above failsafe:
|
||||
ceph osd set-full-ratio 0.98 # → HEALTH_ERR: out of order
|
||||
```
|
||||
|
||||
**With an OSD at 95.2% utilization:**
|
||||
- `nearfull_ratio` 0.94 → OSD at 95.2% shows `nearfull` (WARN, acceptable)
|
||||
- `backfillfull_ratio` 0.96 → OSD at 95.2% is NOT backfillfull → backfill proceeds
|
||||
- `full_ratio` 0.97 → OSD at 95.2% is NOT full → client I/O continues
|
||||
|
||||
If the OSD drains below 0.94, all flags clear automatically.
|
||||
|
||||
## Pitfalls: `ceph osd reweight` vs `ceph osd crush reweight` (2026-07-13)
|
||||
|
||||
Two DIFFERENT commands that both change OSD weighting:
|
||||
|
||||
```bash
|
||||
# Runtime reweight — temporary, lost on OSD restart:
|
||||
ceph osd reweight 7 0.8
|
||||
# Changes the `reweight` column in `ceph osd df`. Persists until OSD restart
|
||||
# or cluster-wide reweight-by-utilization. Does NOT change CRUSH weight.
|
||||
# Good for emergency drain of an overfull OSD.
|
||||
|
||||
# CRUSH weight change — permanent, affects PG distribution:
|
||||
ceph osd crush reweight osd.7 0.96
|
||||
# Changes the CRUSH `weight` column. Survives restarts. Changes how many
|
||||
# PGs the OSD gets assigned going forward.
|
||||
# Use this to match weight to actual device size.
|
||||
```
|
||||
|
||||
**Decision guide:**
|
||||
- Emergency: OSD is 96%+ and blocking recovery → `ceph osd reweight` (fast, temporary)
|
||||
- Permanent: OSD weight doesn't match device size → `ceph osd crush reweight` (slow, proper)
|
||||
- After recovery: Reset `reweight` to 1.0, set `crush weight` to true device size
|
||||
|
||||
## Post-Recovery Cleanup
|
||||
|
||||
After all PGs return to active+clean:
|
||||
|
||||
Reference in New Issue
Block a user