hermes-skills/devops/proxmox-ve-administration/references/ceph-recovery-acceleration-2026-07.md

Ceph Recovery Acceleration — Advanced Tuning — 2026-07-05

Context

Follow-up to ceph-pool-full-recovery-2026-07.md. After unblocking backfill by raising backfillfull_ratio and reweighting OSD 7, recovery was progressing but slowly (22 MiB/s, 5 obj/s, only 2 PGs backfilling). This session applied additional tuning to accelerate recovery.

Additional Recovery Parameters

osd_recovery_op_priority (effective)

# Set globally via MON config
ceph config set osd osd_recovery_op_priority 10
# Default: 3. Range: 1-63. Higher = recovery ops preempt client ops.

# Verify on running OSD:
ceph daemon osd.7 config get osd_recovery_op_priority
# Returns: {"osd_recovery_op_priority": "10"}

Unlike osd_max_backfills, this one DOES apply at runtime via ceph config set — no restart needed. OSDs pick it up within seconds.

osd_recovery_sleep (set to 0)

ceph config set osd osd_recovery_sleep 0
# Default: 0.001 (1ms pause between recovery ops). Set to 0 for maximum throughput.

osd_recovery_max_chunk

ceph config set osd osd_recovery_max_chunk 104857600
# Default: 8MiB. Set to 100MiB for larger transfer windows.

ceph tell osd.N config set — Alternative to injectargs

Newer Ceph (Reef/Squid) supports ceph tell osd.N config set as a cleaner alternative to injectargs:

# Instead of: ceph tell osd.7 injectargs "--osd_max_backfills 3"
# Use:
ceph tell osd.7 config set osd_max_backfills 3

However, in testing, ceph config set osd osd_max_backfills 3 (MON-level) did NOT propagate to running OSDs — the OSDs kept using value 1. The ceph tell osd.N config set also didn't reliably stick. The most reliable method remains:

# Most reliable runtime method:
ceph tell osd.7 injectargs "--osd_max_backfills 3"
# Verify:
ceph daemon osd.7 config get osd_max_backfills
# Must return "3" — if still "1", the config hasn't been applied.

UPDATE 2026-07-13: ceph tell osd.* config set DOES Work

In a follow-up session, ceph tell osd.* config set osd_max_backfills 3 DID successfully apply to running OSDs. Verification:

ceph tell osd.11 config get osd_max_backfills
# Returns: {"osd_max_backfills": "3"}

The difference from the earlier session: ceph config set osd osd_max_backfills 3 (MON-level persistent config) was set FIRST, then ceph tell osd.* config set was used to push it to running OSDs. The combination worked. Injectargs returned empty {} (appeared to fail) but the config took effect.

Recommendation: Use BOTH ceph config set (persistent for future OSDs) AND ceph tell osd.* config set (runtime for current OSDs). Then verify with ceph tell osd.N config get.

osd_recovery_max_active — NOT effectively tunable

ceph config set osd osd_recovery_max_active 3 accepts the value but OSDs report 0 (meaning auto/default). injectargs also shows empty. This appears to be a Ceph limitation — the parameter is managed internally. Do not waste time trying to force it.

Full Recovery Tuning Recipe

# 1. Unblock backfill (if backfillfull)
ceph osd set-backfillfull-ratio 0.97
ceph osd set-nearfull-ratio 0.97

# 2. Reweight overfull OSD
ceph osd crush reweight osd.7 0.50

# 3. Set recovery parameters via MON config (applies to new/restarted OSDs)
ceph config set osd osd_max_backfills 3
ceph config set osd osd_recovery_op_priority 10
ceph config set osd osd_recovery_sleep 0
ceph config set osd osd_recovery_max_chunk 104857600

# 4. Force runtime application on each HDD OSD (MON config doesn't propagate to running OSDs)
for osd in 1 6 7 8 10; do
  ceph tell osd.$osd injectargs "--osd_max_backfills 3"
  ceph tell osd.$osd injectargs "--osd_recovery_op_priority 10"
done

# 5. Verify each OSD picked up the config
for osd in 1 6 7 8 10; do
  echo "=== osd.$osd ==="
  ceph daemon osd.$osd config get osd_max_backfills
  ceph daemon osd.$osd config get osd_recovery_op_priority
done

Recovery Speed Progression

Stage	Recovery Speed	Concurrent Backfills	Notes
Before intervention	0 MiB/s (stuck)	0	backfillfull blocks all
After ratio + reweight	22-23 MiB/s, 5 obj/s	1-2	Backfill unblocked
After injectargs (max_backfills=3)	34 MiB/s, 8 obj/s	2-3	Even with max=3, Ceph schedules conservatively
After op_priority=10 + sleep=0	~34 MiB/s, 8 obj/s	2-3	Priority helps under client I/O load

Even with osd_max_backfills=3, expect only 2-3 PGs backfilling simultaneously — Ceph schedules conservatively. The throughput improvement comes from larger transfer windows and reduced pauses, not dramatically more parallel PGs.

OSD Restart Considerations

If injectargs doesn't work, restarting the OSD daemon forces it to pick up the MON config:

# ⚠️ MUST restart on the correct node! Use ceph osd find to locate:
ceph osd find 7
# Returns JSON with host name and IP

# SSH to THAT node and restart:
ssh root@<correct_node_ip> "systemctl restart ceph-osd@7"
# ❌ WRONG: restarting on a node that doesn't host the OSD →
#   "OSD data directory /var/lib/ceph/osd/ceph-7 does not exist; bailing out"

# After restart, OSD goes through peering — may take 30-60s
# During peering, PGs may show as "peering" or "activating"

Monitoring Recovery

# Quick status
ceph -s | grep -E "osd:|pgs:|recovery:|misplaced"

# Detailed PG states
ceph -s | grep -A15 "pgs:"

# Per-OSD utilization (track if overfull OSD is draining)
ceph osd df | grep "^ 7 "

# Estimated duration: misplaced_objects / objects_per_second

Recovery Monitoring Cron Pattern

Schedule: every 30 minutes
Command: ssh to monitor node, run `ceph status`, report:
  - PGs remapped/backfilling count
  - % misplaced objects
  - OSD 7 utilization
  - Recovery speed
  - Alert if any OSD down or health ERR
  - Announce "COMPLETE" when 0 remapped PGs

Pitfalls: Setting backfill_full_ratio (2026-07-12)

Three commands FAIL when trying to raise the backfill full ratio — only ONE works:

# ✅ CORRECT — CLI command with hyphen (not underscore):
ceph osd set-backfillfull-ratio 0.97
# Verify: ceph osd dump | grep backfillfull_ratio → "backfillfull_ratio 0.97"

# ❌ WRONG — underscore variant:
ceph osd set-backfillfullratio 0.97
# → "no valid command found" (EINVAL)

# ❌ WRONG — MON config set:
ceph config set mon mon_osd_backfillfull_ratio 0.97
# → "mon_osd_backfillfull_ratio is special and cannot be stored by the mon"

# ❌ WRONG — injectargs on specific OSD:
ceph tell osd.10 injectargs "--osd_backfill_full_ratio 0.97"
# → "failed to parse arguments: --osd_backfill_full_ratio,0.97" (EINVAL)
# injectargs only accepts runtime-tunable OSD params, not full-ratio thresholds.

The ratio is a MON-level OSD map parameter, not a per-OSD config. It can ONLY be changed via ceph osd set-backfillfull-ratio (CLI command, not config set). Similarly: ceph osd set-nearfull-ratio and ceph osd set-full-ratio for the other thresholds.

Pitfalls: Pool-Level backfillfull Flag (2026-07-13)

When pools show the backfillfull flag in ceph osd dump, attempting to unset it at the pool level FAILS:

# ❌ WRONG — no such pool flag:
ceph osd pool unset 6 backfillfull
# → "no valid command found; 1 closest matches: osd pool unset noautoscale"

# ❌ WRONG — all pool IDs tried, all fail:
for pool in 1 5 6 7 8 9 10; do ceph osd pool unset $pool backfillfull; done
# All return EINVAL

The backfillfull flag on a pool is a DERIVED condition — it appears when any OSD in the pool's up/acting set is above the backfillfull_ratio. It is NOT a settable pool flag. To clear it, fix the underlying OSD condition: raise backfillfull_ratio or reweight the overfull OSD.

# ✅ CORRECT — raise the ratio (clears backfillfull on all pools):
ceph osd set-backfillfull-ratio 0.99

# ✅ ALSO — reweight the overfull OSD:
ceph osd reweight 7 0.8

Pitfalls: osd_recovery_max_active Behavior (2026-07-13)

ceph config set osd osd_recovery_max_active 3 accepts the value and ceph config get osd osd_recovery_max_active returns 3. However, ceph tell osd.N config get osd_recovery_max_active may return empty string — the OSD interprets 0 or empty as "auto/default". Despite this, setting it alongside osd_max_backfills appeared to help overall. Don't rely on this parameter alone — use osd_max_backfills as the primary lever.

Pitfalls: full ratio(s) out of order (2026-07-13)

When adjusting full/nearfull/backfillfull ratios, you MUST preserve the ordering: nearfull_ratio < backfillfull_ratio < full_ratio <= osd_failsafe_full_ratio.

The osd_failsafe_full_ratio is hardcoded at 0.97 and cannot be changed. If you set full_ratio above 0.97, the cluster goes to HEALTH_ERR:

[ERR] OSD_OUT_OF_ORDER_FULL: full ratio(s) out of order
    osd_failsafe_full_ratio (0.97) < full_ratio (0.98), increased

Safe ratio combinations for a cluster with an OSD at 95%:

# ✅ CORRECT — all ratios below failsafe (0.97), properly ordered:
ceph osd set-nearfull-ratio 0.94
ceph osd set-backfillfull-ratio 0.96
ceph osd set-full-ratio 0.97   # == failsafe, OK

# ❌ WRONG — full_ratio above failsafe:
ceph osd set-full-ratio 0.98   # → HEALTH_ERR: out of order

With an OSD at 95.2% utilization:

• nearfull_ratio 0.94 → OSD at 95.2% shows nearfull (WARN, acceptable)
• backfillfull_ratio 0.96 → OSD at 95.2% is NOT backfillfull → backfill proceeds
• full_ratio 0.97 → OSD at 95.2% is NOT full → client I/O continues

If the OSD drains below 0.94, all flags clear automatically.

Pitfalls: ceph osd reweight vs ceph osd crush reweight (2026-07-13)

Two DIFFERENT commands that both change OSD weighting:

# Runtime reweight — temporary, lost on OSD restart:
ceph osd reweight 7 0.8
# Changes the `reweight` column in `ceph osd df`. Persists until OSD restart
# or cluster-wide reweight-by-utilization. Does NOT change CRUSH weight.
# Good for emergency drain of an overfull OSD.

# CRUSH weight change — permanent, affects PG distribution:
ceph osd crush reweight osd.7 0.96
# Changes the CRUSH `weight` column. Survives restarts. Changes how many
# PGs the OSD gets assigned going forward.
# Use this to match weight to actual device size.

Decision guide:

• Emergency: OSD is 96%+ and blocking recovery → ceph osd reweight (fast, temporary)
• Permanent: OSD weight doesn't match device size → ceph osd crush reweight (slow, proper)
• After recovery: Reset reweight to 1.0, set crush weight to true device size

Post-Recovery Cleanup

After all PGs return to active+clean:

# Restore default ratios
ceph osd set-backfillfull-ratio 0.95
ceph osd set-nearfull-ratio 0.93

# Restore default backfill speed
ceph config set osd osd_max_backfills 1
ceph tell osd.* injectargs "--osd_max_backfills 1"

# Restore OSD weight (if it was reduced)
ceph osd crush reweight osd.7 1.0  # or original weight

# Verify cluster health
ceph health
ceph -s