# Ceph Recovery Acceleration — Advanced Tuning — 2026-07-05

## Context

Follow-up to `ceph-pool-full-recovery-2026-07.md`. After unblocking backfill by raising `backfillfull_ratio` and reweighting OSD 7, recovery was progressing but slowly (22 MiB/s, 5 obj/s, only 2 PGs backfilling). This session applied additional tuning to accelerate recovery.

## Additional Recovery Parameters

### `osd_recovery_op_priority` (effective)

```bash
# Set globally via MON config
ceph config set osd osd_recovery_op_priority 10
# Default: 3. Range: 1-63. Higher = recovery ops preempt client ops.

# Verify on running OSD:
ceph daemon osd.7 config get osd_recovery_op_priority
# Returns: {"osd_recovery_op_priority": "10"}
```

Unlike `osd_max_backfills`, this one DOES apply at runtime via `ceph config set` — no restart needed. OSDs pick it up within seconds.

### `osd_recovery_sleep` (set to 0)

```bash
ceph config set osd osd_recovery_sleep 0
# Default: 0.001 (1ms pause between recovery ops). Set to 0 for maximum throughput.
```

### `osd_recovery_max_chunk`

```bash
ceph config set osd osd_recovery_max_chunk 104857600
# Default: 8MiB. Set to 100MiB for larger transfer windows.
```

### `ceph tell osd.N config set` — Alternative to `injectargs`

Newer Ceph (Reef/Squid) supports `ceph tell osd.N config set` as a cleaner alternative to `injectargs`:

```bash
# Instead of: ceph tell osd.7 injectargs "--osd_max_backfills 3"
# Use:
ceph tell osd.7 config set osd_max_backfills 3
```

However, in testing, `ceph config set osd osd_max_backfills 3` (MON-level) did NOT propagate to running OSDs — the OSDs kept using value 1. The `ceph tell osd.N config set` also didn't reliably stick. The most reliable method remains:

```bash
# Most reliable runtime method:
ceph tell osd.7 injectargs "--osd_max_backfills 3"
# Verify:
ceph daemon osd.7 config get osd_max_backfills
# Must return "3" — if still "1", the config hasn't been applied.
```

### `osd_recovery_max_active` — NOT effectively tunable

`ceph config set osd osd_recovery_max_active 3` accepts the value but OSDs report 0 (meaning auto/default). `injectargs` also shows empty. This appears to be a Ceph limitation — the parameter is managed internally. Do not waste time trying to force it.

## Full Recovery Tuning Recipe

```bash
# 1. Unblock backfill (if backfillfull)
ceph osd set-backfillfull-ratio 0.97
ceph osd set-nearfull-ratio 0.97

# 2. Reweight overfull OSD
ceph osd crush reweight osd.7 0.50

# 3. Set recovery parameters via MON config (applies to new/restarted OSDs)
ceph config set osd osd_max_backfills 3
ceph config set osd osd_recovery_op_priority 10
ceph config set osd osd_recovery_sleep 0
ceph config set osd osd_recovery_max_chunk 104857600

# 4. Force runtime application on each HDD OSD (MON config doesn't propagate to running OSDs)
for osd in 1 6 7 8 10; do
  ceph tell osd.$osd injectargs "--osd_max_backfills 3"
  ceph tell osd.$osd injectargs "--osd_recovery_op_priority 10"
done

# 5. Verify each OSD picked up the config
for osd in 1 6 7 8 10; do
  echo "=== osd.$osd ==="
  ceph daemon osd.$osd config get osd_max_backfills
  ceph daemon osd.$osd config get osd_recovery_op_priority
done
```

## Recovery Speed Progression

| Stage | Recovery Speed | Concurrent Backfills | Notes |
|-------|---------------|---------------------|-------|
| Before intervention | 0 MiB/s (stuck) | 0 | backfillfull blocks all |
| After ratio + reweight | 22-23 MiB/s, 5 obj/s | 1-2 | Backfill unblocked |
| After injectargs (max_backfills=3) | 34 MiB/s, 8 obj/s | 2-3 | Even with max=3, Ceph schedules conservatively |
| After op_priority=10 + sleep=0 | ~34 MiB/s, 8 obj/s | 2-3 | Priority helps under client I/O load |

Even with `osd_max_backfills=3`, expect only 2-3 PGs backfilling simultaneously — Ceph schedules conservatively. The throughput improvement comes from larger transfer windows and reduced pauses, not dramatically more parallel PGs.

## OSD Restart Considerations

If `injectargs` doesn't work, restarting the OSD daemon forces it to pick up the MON config:

```bash
# ⚠️ MUST restart on the correct node! Use ceph osd find to locate:
ceph osd find 7
# Returns JSON with host name and IP

# SSH to THAT node and restart:
ssh root@<correct_node_ip> "systemctl restart ceph-osd@7"
# ❌ WRONG: restarting on a node that doesn't host the OSD →
#   "OSD data directory /var/lib/ceph/osd/ceph-7 does not exist; bailing out"

# After restart, OSD goes through peering — may take 30-60s
# During peering, PGs may show as "peering" or "activating"
```

## Monitoring Recovery

```bash
# Quick status
ceph -s | grep -E "osd:|pgs:|recovery:|misplaced"

# Detailed PG states
ceph -s | grep -A15 "pgs:"

# Per-OSD utilization (track if overfull OSD is draining)
ceph osd df | grep "^ 7 "

# Estimated duration: misplaced_objects / objects_per_second
```

## Recovery Monitoring Cron Pattern

```
Schedule: every 30 minutes
Command: ssh to monitor node, run `ceph status`, report:
  - PGs remapped/backfilling count
  - % misplaced objects
  - OSD 7 utilization
  - Recovery speed
  - Alert if any OSD down or health ERR
  - Announce "COMPLETE" when 0 remapped PGs
```

## Post-Recovery Cleanup

After all PGs return to active+clean:

```bash
# Restore default ratios
ceph osd set-backfillfull-ratio 0.95
ceph osd set-nearfull-ratio 0.93

# Restore default backfill speed
ceph config set osd osd_max_backfills 1
ceph tell osd.* injectargs "--osd_max_backfills 1"

# Restore OSD weight (if it was reduced)
ceph osd crush reweight osd.7 1.0  # or original weight

# Verify cluster health
ceph health
ceph -s
```