hermes-skills/devops/proxmox-ve-administration/references/ceph-pg-management-2026-07.md

Ceph PG Management & OSD Balancing

Reducing PG Count (pg_num) — Autoscale Mode Pitfall

Problem

ceph health warns too many PGs per OSD (259 > max 250). Reducing pg_num on pools with little or no data (e.g. cephfs_data with 0 B, cephfs_metadata with 85 KiB) should be straightforward, but the PG autoscaler silently reverts manual changes.

What Happens

# Attempt to reduce PGs:
ceph osd pool set cephfs_data pg_num 32
# → appears to succeed (exit 0)

# Check:
ceph osd pool get cephfs_data pg_num
# → pg_num: 128  (UNCHANGED!)

The autoscaler (pg_autoscale_mode on) overrides manual pg_num changes. Even after setting pg_autoscale_mode off, the pg_num reduction may not take effect immediately — it only merges when all PGs are in active+clean state with no recovery/backfill in progress.

Correct Procedure

# 1. Disable autoscale for the pool
ceph osd pool set cephfs_data pg_autoscale_mode off
ceph osd pool set cephfs_metadata pg_autoscale_mode off

# 2. Set pgp_num FIRST (controls placement), then pg_num
ceph osd pool set cephfs_data pgp_num 32
ceph osd pool set cephfs_data pg_num 32
# PVE may accept pgp_num=32 but show pg_num_target=32 (deferred merge)

# 3. Verify the target is set
ceph osd pool ls detail | grep cephfs_data
# Look for: pg_num_target 32

# 4. The actual PG merge happens AFTER all recovery/backfill completes
# Monitor: ceph pg dump pgs_brief | tail -n +2 | awk '{print $NF}' | sort | uniq -c
# All PGs must be "active+clean" for the merge to proceed

Increasing mon_max_pg_per_osd (Temporary Relief)

If the PG merge is deferred (waiting for recovery), increase the threshold to suppress the warning:

# Set via config (may not propagate immediately to MONs)
ceph config set mon mon_max_pg_per_osd 400

# Force MONs to reload config (CRITICAL — config set alone doesn't always work)
ceph tell mon.* injectargs --mon_max_pg_per_osd=400

# Verify
ceph config get mon mon_max_pg_per_osd
ceph health

Pitfall: ceph config set mon mon_max_pg_per_osd 400 alone may NOT propagate to running MONs. The health check still shows the old threshold. Use ceph tell mon.* injectargs to force a live config reload.

OSD Reweight vs Concurrent Writes

Scenario

OSD.7 at 88% utilization. Applied ceph osd reweight osd.7 0.65 to gradually redistribute data. Simultaneously, an RBD copy to an EC pool (media_ec) was running, writing new data to the same HDD OSDs.

Observation

OSD.7 initially dropped to ~80% (reweight working), then climbed back to ~85% (new writes from RBD copy landing on OSD.7 via CRUSH).

Explanation

Two opposing forces:

1. Reweight moves existing PGs away from OSD.7 → utilization drops
2. New writes to EC pools distribute chunks to all HDD OSDs including OSD.7 → utilization rises

The reweight only affects PG placement for EXISTING data. New writes follow CRUSH rules with the reduced weight, but the EC pool has its own crush rule and may still place chunks on OSD.7.

Takeaway

When rebalancing an OSD, avoid concurrent heavy writes to pools that use the same OSD device class. The reweight will win AFTER the writes stop. Don't expect monotonic decrease during concurrent I/O.

Calculating PG Impact

# Total PG instances across all pools
for pool in $(ceph osd pool ls); do
  pgnum=$(ceph osd pool get $pool pg_num 2>/dev/null | awk '{print $2}')
  size=$(ceph osd pool get $pool size 2>/dev/null | awk '{print $2}')
  stored=$(ceph df | grep "^$pool " | awk '{print $3}')
  echo "$pool: $pgnum PGs × size $size, stored: $stored"
done

# Approximate PGs per OSD: sum(pg_num × size) / num_osds
# For 9 pools on 10 OSDs with mixed sizes:
# 2627 PG instances / 10 OSDs ≈ 263 per OSD (>250 threshold)

Reducing empty/near-empty pools from 128→32 PGs saves: (128-32) × size × 2 pools = 576 PG instances → ~58 per OSD reduction.

EC4+1 Reweight Pitfall — CRITICAL

Never Reweight OSDs in EC Pools with Minimal OSD Count

An EC4+1 pool requires exactly 5 OSDs (k=4, m=1). If the cluster has exactly 5 HDD OSDs, reweighting any of them causes CRUSH to place 2147483647 (sentinel = "no OSD available") in the PG up-set. This creates remapped PGs that can never recover — CRUSH has no alternative OSD to move data to.

Symptoms

• ceph status shows ~11% objects misplaced, recovery at 0 MiB/s
• Recovery settings tuned (osd_max_backfills=5, osd_recovery_sleep=0) but recovery doesn't progress
• ceph pg dump pgs_brief shows 100+ PGs in active+clean+remapped
• Remapped PGs show 2147483647 in their up-set:

  8.30 active+clean+remapped [8,10,1,2147483647,6] 8
  8.31 active+clean+remapped [6,1,8,2147483647,2147483647] 6
  

Fix

Reset all HDD OSD weights to 1.0:

ceph osd reweight osd.7 1.0
ceph osd reweight osd.10 1.0

Within 60 seconds, remapped PGs begin recovering.

General Rule

Never reweight OSDs in EC pools unless OSD count > k+m. EC(k=4, m=1) needs 5 OSDs — with exactly 5, all must participate at full weight. To balance an overly-full OSD:

1. Add a 6th HDD OSD (gives CRUSH redistribution flexibility)
2. Move data to a different pool
3. Accept the imbalance — EC with minimal OSDs has no rebalancing slack

Diagnostic Flow for Stagnant Recovery

1. ceph pg dump pgs_brief | awk '{print $2}' | sort | uniq -c | sort -rn — look for active+clean+remapped count
2. Inspect remapped PG up-sets — 2147483647 = CRUSH can't place
3. ceph osd erasure-code-profile get <profile> — check k+m
4. ceph osd tree | grep hdd | wc -l — compare OSD count vs k+m
5. If OSD count == k+m: reset all weights to 1.0

Session Log — 2026-07-06 (cont.)

• Reweighted osd.7 0.80→0.65 to relieve 88% utilization → caused 129 remapped PGs in media_ec (EC4+1, only 5 HDD OSDs)
• Recovery stagnated at 11.5% misplaced for 7+ hours (0 MiB/s)
• Fixed by resetting osd.7 and osd.10 to weight 1.0
• Recovery resumed at 67 MiB/s after reset
• mon_max_pg_per_osd raised to 400 (didn't propagate to MONs without restart — left as-is, PG reduction is the proper fix)
• PG merge (cephfs_data/metadata 128→32) still queued, awaits recovery