179 lines
6.1 KiB
Markdown
179 lines
6.1 KiB
Markdown
# Ceph PG Management & OSD Balancing
|
||
|
||
## Reducing PG Count (pg_num) — Autoscale Mode Pitfall
|
||
|
||
### Problem
|
||
|
||
`ceph health` warns `too many PGs per OSD (259 > max 250)`. Reducing
|
||
`pg_num` on pools with little or no data (e.g. `cephfs_data` with 0 B,
|
||
`cephfs_metadata` with 85 KiB) should be straightforward, but the PG
|
||
autoscaler silently reverts manual changes.
|
||
|
||
### What Happens
|
||
|
||
```bash
|
||
# Attempt to reduce PGs:
|
||
ceph osd pool set cephfs_data pg_num 32
|
||
# → appears to succeed (exit 0)
|
||
|
||
# Check:
|
||
ceph osd pool get cephfs_data pg_num
|
||
# → pg_num: 128 (UNCHANGED!)
|
||
```
|
||
|
||
The autoscaler (`pg_autoscale_mode on`) overrides manual `pg_num` changes.
|
||
Even after setting `pg_autoscale_mode off`, the `pg_num` reduction may
|
||
not take effect immediately — it only merges when all PGs are in
|
||
`active+clean` state with no recovery/backfill in progress.
|
||
|
||
### Correct Procedure
|
||
|
||
```bash
|
||
# 1. Disable autoscale for the pool
|
||
ceph osd pool set cephfs_data pg_autoscale_mode off
|
||
ceph osd pool set cephfs_metadata pg_autoscale_mode off
|
||
|
||
# 2. Set pgp_num FIRST (controls placement), then pg_num
|
||
ceph osd pool set cephfs_data pgp_num 32
|
||
ceph osd pool set cephfs_data pg_num 32
|
||
# PVE may accept pgp_num=32 but show pg_num_target=32 (deferred merge)
|
||
|
||
# 3. Verify the target is set
|
||
ceph osd pool ls detail | grep cephfs_data
|
||
# Look for: pg_num_target 32
|
||
|
||
# 4. The actual PG merge happens AFTER all recovery/backfill completes
|
||
# Monitor: ceph pg dump pgs_brief | tail -n +2 | awk '{print $NF}' | sort | uniq -c
|
||
# All PGs must be "active+clean" for the merge to proceed
|
||
```
|
||
|
||
### Increasing mon_max_pg_per_osd (Temporary Relief)
|
||
|
||
If the PG merge is deferred (waiting for recovery), increase the
|
||
threshold to suppress the warning:
|
||
|
||
```bash
|
||
# Set via config (may not propagate immediately to MONs)
|
||
ceph config set mon mon_max_pg_per_osd 400
|
||
|
||
# Force MONs to reload config (CRITICAL — config set alone doesn't always work)
|
||
ceph tell mon.* injectargs --mon_max_pg_per_osd=400
|
||
|
||
# Verify
|
||
ceph config get mon mon_max_pg_per_osd
|
||
ceph health
|
||
```
|
||
|
||
**Pitfall**: `ceph config set mon mon_max_pg_per_osd 400` alone may NOT
|
||
propagate to running MONs. The health check still shows the old threshold.
|
||
Use `ceph tell mon.* injectargs` to force a live config reload.
|
||
|
||
## OSD Reweight vs Concurrent Writes
|
||
|
||
### Scenario
|
||
|
||
OSD.7 at 88% utilization. Applied `ceph osd reweight osd.7 0.65` to
|
||
gradually redistribute data. Simultaneously, an RBD copy to an EC pool
|
||
(`media_ec`) was running, writing new data to the same HDD OSDs.
|
||
|
||
### Observation
|
||
|
||
OSD.7 initially dropped to ~80% (reweight working), then climbed back
|
||
to ~85% (new writes from RBD copy landing on OSD.7 via CRUSH).
|
||
|
||
### Explanation
|
||
|
||
Two opposing forces:
|
||
1. **Reweight** moves existing PGs away from OSD.7 → utilization drops
|
||
2. **New writes** to EC pools distribute chunks to all HDD OSDs including
|
||
OSD.7 → utilization rises
|
||
|
||
The reweight only affects PG placement for EXISTING data. New writes
|
||
follow CRUSH rules with the reduced weight, but the EC pool has its own
|
||
crush rule and may still place chunks on OSD.7.
|
||
|
||
### Takeaway
|
||
|
||
When rebalancing an OSD, avoid concurrent heavy writes to pools that
|
||
use the same OSD device class. The reweight will win AFTER the writes
|
||
stop. Don't expect monotonic decrease during concurrent I/O.
|
||
|
||
## Calculating PG Impact
|
||
|
||
```bash
|
||
# Total PG instances across all pools
|
||
for pool in $(ceph osd pool ls); do
|
||
pgnum=$(ceph osd pool get $pool pg_num 2>/dev/null | awk '{print $2}')
|
||
size=$(ceph osd pool get $pool size 2>/dev/null | awk '{print $2}')
|
||
stored=$(ceph df | grep "^$pool " | awk '{print $3}')
|
||
echo "$pool: $pgnum PGs × size $size, stored: $stored"
|
||
done
|
||
|
||
# Approximate PGs per OSD: sum(pg_num × size) / num_osds
|
||
# For 9 pools on 10 OSDs with mixed sizes:
|
||
# 2627 PG instances / 10 OSDs ≈ 263 per OSD (>250 threshold)
|
||
```
|
||
|
||
Reducing empty/near-empty pools from 128→32 PGs saves:
|
||
`(128-32) × size × 2 pools = 576 PG instances` → ~58 per OSD reduction.
|
||
|
||
## EC4+1 Reweight Pitfall — CRITICAL
|
||
|
||
### Never Reweight OSDs in EC Pools with Minimal OSD Count
|
||
|
||
An EC4+1 pool requires exactly 5 OSDs (k=4, m=1). If the cluster has
|
||
exactly 5 HDD OSDs, reweighting any of them causes CRUSH to place
|
||
`2147483647` (sentinel = "no OSD available") in the PG up-set. This
|
||
creates remapped PGs that can **never recover** — CRUSH has no
|
||
alternative OSD to move data to.
|
||
|
||
### Symptoms
|
||
|
||
- `ceph status` shows ~11% objects misplaced, recovery at 0 MiB/s
|
||
- Recovery settings tuned (`osd_max_backfills=5`, `osd_recovery_sleep=0`)
|
||
but recovery doesn't progress
|
||
- `ceph pg dump pgs_brief` shows 100+ PGs in `active+clean+remapped`
|
||
- Remapped PGs show `2147483647` in their up-set:
|
||
```
|
||
8.30 active+clean+remapped [8,10,1,2147483647,6] 8
|
||
8.31 active+clean+remapped [6,1,8,2147483647,2147483647] 6
|
||
```
|
||
|
||
### Fix
|
||
|
||
Reset all HDD OSD weights to 1.0:
|
||
```bash
|
||
ceph osd reweight osd.7 1.0
|
||
ceph osd reweight osd.10 1.0
|
||
```
|
||
Within 60 seconds, remapped PGs begin recovering.
|
||
|
||
### General Rule
|
||
|
||
**Never reweight OSDs in EC pools unless OSD count > k+m.** EC(k=4, m=1)
|
||
needs 5 OSDs — with exactly 5, all must participate at full weight. To
|
||
balance an overly-full OSD:
|
||
1. Add a 6th HDD OSD (gives CRUSH redistribution flexibility)
|
||
2. Move data to a different pool
|
||
3. Accept the imbalance — EC with minimal OSDs has no rebalancing slack
|
||
|
||
### Diagnostic Flow for Stagnant Recovery
|
||
|
||
1. `ceph pg dump pgs_brief | awk '{print $2}' | sort | uniq -c | sort -rn`
|
||
— look for `active+clean+remapped` count
|
||
2. Inspect remapped PG up-sets — `2147483647` = CRUSH can't place
|
||
3. `ceph osd erasure-code-profile get <profile>` — check k+m
|
||
4. `ceph osd tree | grep hdd | wc -l` — compare OSD count vs k+m
|
||
5. If OSD count == k+m: reset all weights to 1.0
|
||
|
||
## Session Log — 2026-07-06 (cont.)
|
||
|
||
- Reweighted osd.7 0.80→0.65 to relieve 88% utilization → caused 129
|
||
remapped PGs in `media_ec` (EC4+1, only 5 HDD OSDs)
|
||
- Recovery stagnated at 11.5% misplaced for 7+ hours (0 MiB/s)
|
||
- Fixed by resetting osd.7 and osd.10 to weight 1.0
|
||
- Recovery resumed at 67 MiB/s after reset
|
||
- `mon_max_pg_per_osd` raised to 400 (didn't propagate to MONs without
|
||
restart — left as-is, PG reduction is the proper fix)
|
||
- PG merge (cephfs_data/metadata 128→32) still queued, awaits recovery
|