3.9 KiB
Ceph Cluster
Overview
- Cluster ID: 204c8171-e0b1-4f40-9de2-a7cfe4ef68d9
- Health: HEALTH_WARN (rebalancing after osd.7 reweight: 43 PGs backfilling at ~26 MiB/s)
- Nodes: 9 Proxmox hosts, 3 MONs (proxmox5/7/4), MGR on proxmox5
- OSDs: 15 (8 SSD, 6 HDD, 1 new SSD on ms-a2-2), all up/in
- Capacity: 6.0 TiB used / 21 TiB total
OSD Layout
| OSD | Class | Size | Host | Reweight | Notes |
|---|---|---|---|---|---|
| 0 | ssd | 188 GB | proxmox2 | 0.50 | |
| 1 | hdd | 3.7 TiB | n5pro | 1.0 | Large HDD |
| 2 | ssd | 233 GB | proxmox2 | 0.90 | Slow ops, 81% full |
| 3 | ssd | 238 GB | proxmox4 | 0.95 | |
| 4 | ssd | 233 GB | proxmox3 | 0.95 | |
| 5 | ssd | 238 GB | proxmox5 | 0.90 | 80% full |
| 6 | hdd | 2.8 TiB | ubuntu | 1.0 | Large HDD |
| 7 | hdd | 500 GB | proxmox7 | 1.0 | Was 0.80, reweighted 2026-07-24 |
| 8 | hdd | 2.8 TiB | ubuntu | 1.0 | BlueFS spillover |
| 9 | ssd | 1.9 TiB | n5pro | 1.0 | |
| 10 | hdd | 300 GB | proxmox6 | 1.0 | Very small HDD |
| 11 | hdd | 2.8 TiB | n5pro | 1.0 | Large HDD |
| 12 | ssd | 1.9 TiB | n5pro | 1.0 | |
| 13 | ssd | 1.8 TiB | ms-a2-1 | 1.0 | |
| 14 | ssd | 1.8 TiB | ms-a2-2 | 1.0 | New 2026-07-24, nvme0n1 |
Pools
| Pool | Name | Type | Size | Min | CRUSH Rule | PGs | Notes |
|---|---|---|---|---|---|---|---|
| 1 | cephfs_data | replicated | 3 | 2 | 0 (any) | 32 | autoscale off |
| 2 | cephfs_metadata | replicated | 3 | 2 | 2 (ssd) | 32 | autoscale off |
| 3 | vm_disks | replicated | 3 | 2 | 2 (ssd) | 128 | autoscale on |
| 4 | .mgr | replicated | 3 | 2 | 2 (ssd) | 1 | |
| 5 | rbd | replicated | 3 | 2 | 1 (hdd) | 32 | autoscale on |
| 6 | hdd_disk | replicated | 3 | 2 | 1 (hdd) | 112 | nopgchange=true (was 120, equalized to 112) |
| 7 | tm_disks | replicated | 2 | 2 | 1 (hdd) | 128 | target_size 2TiB |
| 8 | media_ec | erasure 4+1 | 5 | 4 | 3 (hdd, osd-level) | 128 | ec_overwrites |
| 9 | media_meta | replicated | 3 | 2 | 0 (any) | 32 | |
| 10 | .rgw.root | replicated | 3 | 2 | 0 (any) | 1 |
CRUSH Rules
- Rule 0 (replicated_rule): default root, host-level placement
- Rule 1 (replicated_hdd): default~hdd, host-level placement
- Rule 2 (replicated_ssd): default~ssd, host-level placement
- Rule 3 (media_ec): default~hdd, OSD-level placement (choose_indep)
Known Issues
Weight Imbalance Causing Placement Failures (2026-07-24)
HDD hosts have extreme weight disparity: n5pro=10.15TB, ubuntu=5.49TB, proxmox7=0.50TB, proxmox6=0.30TB.
CRUSH host-level selection (rule 1) often picks only 2 of 4 HDD hosts → up sets with 2 OSDs instead of 3.
Result: PGs stuck in active+clean+remapped because up set < min_size.
Mitigation (2026-07-24):
- Reweighted osd.7 from 0.80 → 1.0 → fixed EC pool 8.3d (NONE → osd.7)
- Equalized pool 6 pg_num 120 → 112 + nopgchange=true
- Manual pg-upmap for stuck PGs: 5.13 → [1,6,7], 6.6c → [11,8,7], 6.58 → [11,6,7]
- All
clean+remappedeliminated. Triggered rebalancing wave (43 PGs backfilling at 26 MiB/s).
Long-term: Small HDDs (osd.7 0.5TB, osd.10 0.3TB) cause CRUSH placement failures. Replace with larger disks or create separate CRUSH root for large HDDs only.
Pool 6 pg_num/pgp_num Mismatch (Fixed 2026-07-24)
Pool hdd_disk had pg_num=120, pgp_num=112 (autoscaler reducing to 32). Equalized pg_num to 112. Set nopgchange=true to prevent further autoscaler interference.
BlueFS Spillover on osd.8
osd.8 spilled 128KiB metadata from db device (2.1GiB of 30GiB) to slow device. Cosmetic warning, no data risk.
Slow Operations on osd.2 and osd.7
osd.2 (81% full, fragmentation 0.80) and osd.7 (small HDD) experience slow BlueStore ops. osd.2 NVMe has 92% wear — candidate for replacement.
Access
- SSH to Proxmox hosts:
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.92 - Ceph commands:
ceph status,ceph osd tree,ceph pg dump pgs - Mon nodes: proxmox5, proxmox7, proxmox4
- Mgr: proxmox5 (active)