5.5 KiB
Ceph Recovery Acceleration — Advanced Tuning — 2026-07-05
Context
Follow-up to ceph-pool-full-recovery-2026-07.md. After unblocking backfill by raising backfillfull_ratio and reweighting OSD 7, recovery was progressing but slowly (22 MiB/s, 5 obj/s, only 2 PGs backfilling). This session applied additional tuning to accelerate recovery.
Additional Recovery Parameters
osd_recovery_op_priority (effective)
# Set globally via MON config
ceph config set osd osd_recovery_op_priority 10
# Default: 3. Range: 1-63. Higher = recovery ops preempt client ops.
# Verify on running OSD:
ceph daemon osd.7 config get osd_recovery_op_priority
# Returns: {"osd_recovery_op_priority": "10"}
Unlike osd_max_backfills, this one DOES apply at runtime via ceph config set — no restart needed. OSDs pick it up within seconds.
osd_recovery_sleep (set to 0)
ceph config set osd osd_recovery_sleep 0
# Default: 0.001 (1ms pause between recovery ops). Set to 0 for maximum throughput.
osd_recovery_max_chunk
ceph config set osd osd_recovery_max_chunk 104857600
# Default: 8MiB. Set to 100MiB for larger transfer windows.
ceph tell osd.N config set — Alternative to injectargs
Newer Ceph (Reef/Squid) supports ceph tell osd.N config set as a cleaner alternative to injectargs:
# Instead of: ceph tell osd.7 injectargs "--osd_max_backfills 3"
# Use:
ceph tell osd.7 config set osd_max_backfills 3
However, in testing, ceph config set osd osd_max_backfills 3 (MON-level) did NOT propagate to running OSDs — the OSDs kept using value 1. The ceph tell osd.N config set also didn't reliably stick. The most reliable method remains:
# Most reliable runtime method:
ceph tell osd.7 injectargs "--osd_max_backfills 3"
# Verify:
ceph daemon osd.7 config get osd_max_backfills
# Must return "3" — if still "1", the config hasn't been applied.
osd_recovery_max_active — NOT effectively tunable
ceph config set osd osd_recovery_max_active 3 accepts the value but OSDs report 0 (meaning auto/default). injectargs also shows empty. This appears to be a Ceph limitation — the parameter is managed internally. Do not waste time trying to force it.
Full Recovery Tuning Recipe
# 1. Unblock backfill (if backfillfull)
ceph osd set-backfillfull-ratio 0.97
ceph osd set-nearfull-ratio 0.97
# 2. Reweight overfull OSD
ceph osd crush reweight osd.7 0.50
# 3. Set recovery parameters via MON config (applies to new/restarted OSDs)
ceph config set osd osd_max_backfills 3
ceph config set osd osd_recovery_op_priority 10
ceph config set osd osd_recovery_sleep 0
ceph config set osd osd_recovery_max_chunk 104857600
# 4. Force runtime application on each HDD OSD (MON config doesn't propagate to running OSDs)
for osd in 1 6 7 8 10; do
ceph tell osd.$osd injectargs "--osd_max_backfills 3"
ceph tell osd.$osd injectargs "--osd_recovery_op_priority 10"
done
# 5. Verify each OSD picked up the config
for osd in 1 6 7 8 10; do
echo "=== osd.$osd ==="
ceph daemon osd.$osd config get osd_max_backfills
ceph daemon osd.$osd config get osd_recovery_op_priority
done
Recovery Speed Progression
| Stage | Recovery Speed | Concurrent Backfills | Notes |
|---|---|---|---|
| Before intervention | 0 MiB/s (stuck) | 0 | backfillfull blocks all |
| After ratio + reweight | 22-23 MiB/s, 5 obj/s | 1-2 | Backfill unblocked |
| After injectargs (max_backfills=3) | 34 MiB/s, 8 obj/s | 2-3 | Even with max=3, Ceph schedules conservatively |
| After op_priority=10 + sleep=0 | ~34 MiB/s, 8 obj/s | 2-3 | Priority helps under client I/O load |
Even with osd_max_backfills=3, expect only 2-3 PGs backfilling simultaneously — Ceph schedules conservatively. The throughput improvement comes from larger transfer windows and reduced pauses, not dramatically more parallel PGs.
OSD Restart Considerations
If injectargs doesn't work, restarting the OSD daemon forces it to pick up the MON config:
# ⚠️ MUST restart on the correct node! Use ceph osd find to locate:
ceph osd find 7
# Returns JSON with host name and IP
# SSH to THAT node and restart:
ssh root@<correct_node_ip> "systemctl restart ceph-osd@7"
# ❌ WRONG: restarting on a node that doesn't host the OSD →
# "OSD data directory /var/lib/ceph/osd/ceph-7 does not exist; bailing out"
# After restart, OSD goes through peering — may take 30-60s
# During peering, PGs may show as "peering" or "activating"
Monitoring Recovery
# Quick status
ceph -s | grep -E "osd:|pgs:|recovery:|misplaced"
# Detailed PG states
ceph -s | grep -A15 "pgs:"
# Per-OSD utilization (track if overfull OSD is draining)
ceph osd df | grep "^ 7 "
# Estimated duration: misplaced_objects / objects_per_second
Recovery Monitoring Cron Pattern
Schedule: every 30 minutes
Command: ssh to monitor node, run `ceph status`, report:
- PGs remapped/backfilling count
- % misplaced objects
- OSD 7 utilization
- Recovery speed
- Alert if any OSD down or health ERR
- Announce "COMPLETE" when 0 remapped PGs
Post-Recovery Cleanup
After all PGs return to active+clean:
# Restore default ratios
ceph osd set-backfillfull-ratio 0.95
ceph osd set-nearfull-ratio 0.93
# Restore default backfill speed
ceph config set osd osd_max_backfills 1
ceph tell osd.* injectargs "--osd_max_backfills 1"
# Restore OSD weight (if it was reduced)
ceph osd crush reweight osd.7 1.0 # or original weight
# Verify cluster health
ceph health
ceph -s