# Ceph PG Management & OSD Balancing

## Reducing PG Count (pg_num) — Autoscale Mode Pitfall

### Problem

`ceph health` warns `too many PGs per OSD (259 > max 250)`. Reducing
`pg_num` on pools with little or no data (e.g. `cephfs_data` with 0 B,
`cephfs_metadata` with 85 KiB) should be straightforward, but the PG
autoscaler silently reverts manual changes.

### What Happens

```bash
# Attempt to reduce PGs:
ceph osd pool set cephfs_data pg_num 32
# → appears to succeed (exit 0)

# Check:
ceph osd pool get cephfs_data pg_num
# → pg_num: 128  (UNCHANGED!)
```

The autoscaler (`pg_autoscale_mode on`) overrides manual `pg_num` changes.
Even after setting `pg_autoscale_mode off`, the `pg_num` reduction may
not take effect immediately — it only merges when all PGs are in
`active+clean` state with no recovery/backfill in progress.

### Correct Procedure

```bash
# 1. Disable autoscale for the pool
ceph osd pool set cephfs_data pg_autoscale_mode off
ceph osd pool set cephfs_metadata pg_autoscale_mode off

# 2. Set pgp_num FIRST (controls placement), then pg_num
ceph osd pool set cephfs_data pgp_num 32
ceph osd pool set cephfs_data pg_num 32
# PVE may accept pgp_num=32 but show pg_num_target=32 (deferred merge)

# 3. Verify the target is set
ceph osd pool ls detail | grep cephfs_data
# Look for: pg_num_target 32

# 4. The actual PG merge happens AFTER all recovery/backfill completes
# Monitor: ceph pg dump pgs_brief | tail -n +2 | awk '{print $NF}' | sort | uniq -c
# All PGs must be "active+clean" for the merge to proceed
```

### Increasing mon_max_pg_per_osd (Temporary Relief)

If the PG merge is deferred (waiting for recovery), increase the
threshold to suppress the warning:

```bash
# Set via config (may not propagate immediately to MONs)
ceph config set mon mon_max_pg_per_osd 400

# Force MONs to reload config (CRITICAL — config set alone doesn't always work)
ceph tell mon.* injectargs --mon_max_pg_per_osd=400

# Verify
ceph config get mon mon_max_pg_per_osd
ceph health
```

**Pitfall**: `ceph config set mon mon_max_pg_per_osd 400` alone may NOT
propagate to running MONs. The health check still shows the old threshold.
Use `ceph tell mon.* injectargs` to force a live config reload.

## OSD Reweight vs Concurrent Writes

### Scenario

OSD.7 at 88% utilization. Applied `ceph osd reweight osd.7 0.65` to
gradually redistribute data. Simultaneously, an RBD copy to an EC pool
(`media_ec`) was running, writing new data to the same HDD OSDs.

### Observation

OSD.7 initially dropped to ~80% (reweight working), then climbed back
to ~85% (new writes from RBD copy landing on OSD.7 via CRUSH).

### Explanation

Two opposing forces:
1. **Reweight** moves existing PGs away from OSD.7 → utilization drops
2. **New writes** to EC pools distribute chunks to all HDD OSDs including
   OSD.7 → utilization rises

The reweight only affects PG placement for EXISTING data. New writes
follow CRUSH rules with the reduced weight, but the EC pool has its own
crush rule and may still place chunks on OSD.7.

### Takeaway

When rebalancing an OSD, avoid concurrent heavy writes to pools that
use the same OSD device class. The reweight will win AFTER the writes
stop. Don't expect monotonic decrease during concurrent I/O.

## Calculating PG Impact

```bash
# Total PG instances across all pools
for pool in $(ceph osd pool ls); do
  pgnum=$(ceph osd pool get $pool pg_num 2>/dev/null | awk '{print $2}')
  size=$(ceph osd pool get $pool size 2>/dev/null | awk '{print $2}')
  stored=$(ceph df | grep "^$pool " | awk '{print $3}')
  echo "$pool: $pgnum PGs × size $size, stored: $stored"
done

# Approximate PGs per OSD: sum(pg_num × size) / num_osds
# For 9 pools on 10 OSDs with mixed sizes:
# 2627 PG instances / 10 OSDs ≈ 263 per OSD (>250 threshold)
```

Reducing empty/near-empty pools from 128→32 PGs saves:
`(128-32) × size × 2 pools = 576 PG instances` → ~58 per OSD reduction.

## EC4+1 Reweight Pitfall — CRITICAL

### Never Reweight OSDs in EC Pools with Minimal OSD Count

An EC4+1 pool requires exactly 5 OSDs (k=4, m=1). If the cluster has
exactly 5 HDD OSDs, reweighting any of them causes CRUSH to place
`2147483647` (sentinel = "no OSD available") in the PG up-set. This
creates remapped PGs that can **never recover** — CRUSH has no
alternative OSD to move data to.

### Symptoms

- `ceph status` shows ~11% objects misplaced, recovery at 0 MiB/s
- Recovery settings tuned (`osd_max_backfills=5`, `osd_recovery_sleep=0`)
  but recovery doesn't progress
- `ceph pg dump pgs_brief` shows 100+ PGs in `active+clean+remapped`
- Remapped PGs show `2147483647` in their up-set:
  ```
  8.30 active+clean+remapped [8,10,1,2147483647,6] 8
  8.31 active+clean+remapped [6,1,8,2147483647,2147483647] 6
  ```

### Fix

Reset all HDD OSD weights to 1.0:
```bash
ceph osd reweight osd.7 1.0
ceph osd reweight osd.10 1.0
```
Within 60 seconds, remapped PGs begin recovering.

### General Rule

**Never reweight OSDs in EC pools unless OSD count > k+m.** EC(k=4, m=1)
needs 5 OSDs — with exactly 5, all must participate at full weight. To
balance an overly-full OSD:
1. Add a 6th HDD OSD (gives CRUSH redistribution flexibility)
2. Move data to a different pool
3. Accept the imbalance — EC with minimal OSDs has no rebalancing slack

### Diagnostic Flow for Stagnant Recovery

1. `ceph pg dump pgs_brief | awk '{print $2}' | sort | uniq -c | sort -rn`
   — look for `active+clean+remapped` count
2. Inspect remapped PG up-sets — `2147483647` = CRUSH can't place
3. `ceph osd erasure-code-profile get <profile>` — check k+m
4. `ceph osd tree | grep hdd | wc -l` — compare OSD count vs k+m
5. If OSD count == k+m: reset all weights to 1.0

## Session Log — 2026-07-06 (cont.)

- Reweighted osd.7 0.80→0.65 to relieve 88% utilization → caused 129
  remapped PGs in `media_ec` (EC4+1, only 5 HDD OSDs)
- Recovery stagnated at 11.5% misplaced for 7+ hours (0 MiB/s)
- Fixed by resetting osd.7 and osd.10 to weight 1.0
- Recovery resumed at 67 MiB/s after reset
- `mon_max_pg_per_osd` raised to 400 (didn't propagate to MONs without
  restart — left as-is, PG reduction is the proper fix)
- PG merge (cephfs_data/metadata 128→32) still queued, awaits recovery
