# Ceph PG Management & OSD Rebalancing

## PG Count Reduction (too many PGs per OSD)

### Symptom

`ceph health` reports: `too many PGs per OSD (259 > max 250)`

### Procedure

```bash
# 1. Disable autoscale on target pools (CRITICAL — autoscaler overwrites manual values)
ceph osd pool set <pool> pg_autoscale_mode off

# 2. Set pgp_num FIRST (accepts immediately)
ceph osd pool set <pool> pgp_num <new_value>

# 3. Set pg_num (may not take effect immediately)
ceph osd pool set <pool> pg_num <new_value>
```

### Pitfalls

- **Autoscaler overrides manual pg_num**: If `pg_autoscale_mode=on`, setting
  `pg_num` appears to succeed but the value reverts. Must set `off` first.

- **pg_num won't reduce during recovery**: If PGs are not all `active+clean`
  (e.g. OSD reweight recovery in progress), `pg_num` stays at the old value.
  The `pg_num_target` is set (visible in `ceph osd pool ls detail`) and the
  merge executes automatically once recovery completes.

- **`mon_max_pg_per_osd` does NOT hot-reload**: Setting it via
  `ceph config set mon mon_max_pg_per_osd 400` updates the config but the
  MONs continue using the old value. `ceph tell mon.* injectargs` also
  failed to apply it. The health warning persists until either:
  - The PG merge completes (actual PG count drops), or
  - MONs are restarted (picks up new config value)

- **Reducing from 128→32 on near-empty pools is safe**: Pools with 0 B or
  85 KiB of data (e.g. unused cephfs_data, cephfs_metadata) can be reduced
  aggressively. 128→32 saves 576 PG instances across 3 replicas = significant.

### Choosing Target PG Count

| Pool stored data | Recommended PGs |
|-----------------|-----------------|
| 0 B – 1 GiB | 32 |
| 1–100 GiB | 64–128 |
| 100 GiB–1 TiB | 128–256 |
| >1 TiB | 256–512 |

Rule: each PG should hold ~100 GiB of data for optimal balance.

## OSD Reweight (Rebalancing Full OSDs)

### Symptom

One OSD at 85-88% utilization while others are at 30-40%. `VAR` >1.5 in
`ceph osd df`.

### Procedure

```bash
# Gradually reduce weight (default 1.0, lower = less data assigned)
ceph osd reweight osd.<N> 0.65

# Monitor progress
ceph osd df | grep osd.<N>
# Watch %USE decrease over time as data migrates away
```

### Pitfalls

- **Competing I/O defeats reweight**: If new writes target the same OSD's
  pool (e.g. RBD copy writing to `media_ec` HDD pool while reweighting
  an HDD OSD), the OSD fills faster than reweight empties it. Observed:
  osd.7 went from 80% → 85% during a 341 GiB RBD copy despite reweight=0.65.
  The reweight only wins after the concurrent write workload stops.

- **Recovery is slow under I/O contention**: With concurrent fsck + RBD
  copy + reweight recovery all hitting HDD OSDs, expect `BLUESTORE_SLOW_OP_ALERT`
  and recovery rates of 20-90 MiB/s instead of peak. This is temporary —
  speeds recover after the competing workload finishes.

- **Reweight is gradual by design**: CRUST gradually moves PGs away from
  the reweighted OSD. Don't expect instant results — monitor over
  10-60 minutes depending on data volume.

## Concurrent I/O Awareness (Critical)

Multiple Ceph-intensive operations running simultaneously cause cascading
slowness:

| Operation | I/O Profile |
|-----------|------------|
| OSD reweight recovery | Random read + sequential write across OSDs |
| RBD copy (--data-pool) | Sequential read source + EC write target |
| Seafile fsck --repair | Random read across HDD pool |
| PG merge (post-recovery) | Metadata-heavy, blocks on clean state |

**Recommendation**: Run these sequentially when possible. If concurrent,
expect 3-5x slowdown on all operations and temporary HEALTH_WARN from
slow ops. All operations complete eventually — patience is the fix.

## Quick Reference: This Cluster's Pools

| Pool | PGs | Size | Data Class | Notes |
|------|-----|------|------------|-------|
| cephfs_data | 128→32 | 3 | HDD | 0 B stored — massively over-provisioned |
| cephfs_metadata | 128→32 | 3 | HDD | 85 KiB stored — massively over-provisioned |
| vm_disks | 128 | 3 | SSD | VM root disks |
| rbd | 32 | 3 | SSD | Small images |
| hdd_disk | 128 | 3 | HDD | CT root disks |
| tm_disks | 128 | 2 | HDD | Legacy, size=2 (NO fault tolerance) |
| media_ec | 128 | 5 (EC4+1) | HDD | EC data pool |
| media_meta | 32 | 3 | HDD | EC metadata pool |
| .mgr | 1 | 3 | HDD | Manager daemon |

## Session Log — 2026-07-06

Reduced cephfs_data and cephfs_metadata PGs 128→32 (pg_num_target set,
merge pending recovery completion). Reweighted osd.7 from 0.80→0.65 to
rebalance from 88% utilization. Both operations competed with a 341 GiB
RBD copy (tm_disks→media_ec) and Seafile fsck, causing slow ops and
extended timelines. PG warning persisted until recovery completed.