83 lines
3.9 KiB
Markdown
83 lines
3.9 KiB
Markdown
# Ceph Cluster
|
|
|
|
## Overview
|
|
- **Cluster ID**: 204c8171-e0b1-4f40-9de2-a7cfe4ef68d9
|
|
- **Health**: HEALTH_WARN (rebalancing after osd.7 reweight: 43 PGs backfilling at ~26 MiB/s)
|
|
- **Nodes**: 9 Proxmox hosts, 3 MONs (proxmox5/7/4), MGR on proxmox5
|
|
- **OSDs**: 15 (8 SSD, 6 HDD, 1 new SSD on ms-a2-2), all up/in
|
|
- **Capacity**: 6.0 TiB used / 21 TiB total
|
|
|
|
## OSD Layout
|
|
|
|
| OSD | Class | Size | Host | Reweight | Notes |
|
|
|-----|-------|------|------|----------|-------|
|
|
| 0 | ssd | 188 GB | proxmox2 | 0.50 | |
|
|
| 1 | hdd | 3.7 TiB | n5pro | 1.0 | Large HDD |
|
|
| 2 | ssd | 233 GB | proxmox2 | 0.90 | Slow ops, 81% full |
|
|
| 3 | ssd | 238 GB | proxmox4 | 0.95 | |
|
|
| 4 | ssd | 233 GB | proxmox3 | 0.95 | |
|
|
| 5 | ssd | 238 GB | proxmox5 | 0.90 | 80% full |
|
|
| 6 | hdd | 2.8 TiB | ubuntu | 1.0 | Large HDD |
|
|
| 7 | hdd | 500 GB | proxmox7 | 1.0 | Was 0.80, reweighted 2026-07-24 |
|
|
| 8 | hdd | 2.8 TiB | ubuntu | 1.0 | BlueFS spillover |
|
|
| 9 | ssd | 1.9 TiB | n5pro | 1.0 | |
|
|
| 10 | hdd | 300 GB | proxmox6 | 1.0 | Very small HDD |
|
|
| 11 | hdd | 2.8 TiB | n5pro | 1.0 | Large HDD |
|
|
| 12 | ssd | 1.9 TiB | n5pro | 1.0 | |
|
|
| 13 | ssd | 1.8 TiB | ms-a2-1 | 1.0 | |
|
|
| 14 | ssd | 1.8 TiB | ms-a2-2 | 1.0 | New 2026-07-24, nvme0n1 |
|
|
|
|
## Pools
|
|
|
|
| Pool | Name | Type | Size | Min | CRUSH Rule | PGs | Notes |
|
|
|------|------|------|------|-----|------------|-----|-------|
|
|
| 1 | cephfs_data | replicated | 3 | 2 | 0 (any) | 32 | autoscale off |
|
|
| 2 | cephfs_metadata | replicated | 3 | 2 | 2 (ssd) | 32 | autoscale off |
|
|
| 3 | vm_disks | replicated | 3 | 2 | 2 (ssd) | 128 | autoscale on |
|
|
| 4 | .mgr | replicated | 3 | 2 | 2 (ssd) | 1 | |
|
|
| 5 | rbd | replicated | 3 | 2 | 1 (hdd) | 32 | autoscale on |
|
|
| 6 | hdd_disk | replicated | 3 | 2 | 1 (hdd) | 112 | nopgchange=true (was 120, equalized to 112) |
|
|
| 7 | tm_disks | replicated | 2 | 2 | 1 (hdd) | 128 | target_size 2TiB |
|
|
| 8 | media_ec | erasure 4+1 | 5 | 4 | 3 (hdd, osd-level) | 128 | ec_overwrites |
|
|
| 9 | media_meta | replicated | 3 | 2 | 0 (any) | 32 | |
|
|
| 10 | .rgw.root | replicated | 3 | 2 | 0 (any) | 1 | |
|
|
|
|
## CRUSH Rules
|
|
- **Rule 0** (replicated_rule): default root, host-level placement
|
|
- **Rule 1** (replicated_hdd): default~hdd, host-level placement
|
|
- **Rule 2** (replicated_ssd): default~ssd, host-level placement
|
|
- **Rule 3** (media_ec): default~hdd, OSD-level placement (choose_indep)
|
|
|
|
## Known Issues
|
|
|
|
### Weight Imbalance Causing Placement Failures (2026-07-24)
|
|
HDD hosts have extreme weight disparity: n5pro=10.15TB, ubuntu=5.49TB, proxmox7=0.50TB, proxmox6=0.30TB.
|
|
CRUSH host-level selection (rule 1) often picks only 2 of 4 HDD hosts → up sets with 2 OSDs instead of 3.
|
|
Result: PGs stuck in `active+clean+remapped` because up set < min_size.
|
|
|
|
**Mitigation (2026-07-24)**:
|
|
1. Reweighted osd.7 from 0.80 → 1.0 → fixed EC pool 8.3d (NONE → osd.7)
|
|
2. Equalized pool 6 pg_num 120 → 112 + nopgchange=true
|
|
3. Manual pg-upmap for stuck PGs: 5.13 → [1,6,7], 6.6c → [11,8,7], 6.58 → [11,6,7]
|
|
4. All `clean+remapped` eliminated. Triggered rebalancing wave (43 PGs backfilling at 26 MiB/s).
|
|
|
|
**Long-term**: Small HDDs (osd.7 0.5TB, osd.10 0.3TB) cause CRUSH placement failures. Replace with larger disks or create separate CRUSH root for large HDDs only.
|
|
|
|
### Pool 6 pg_num/pgp_num Mismatch (Fixed 2026-07-24)
|
|
Pool hdd_disk had pg_num=120, pgp_num=112 (autoscaler reducing to 32).
|
|
Equalized pg_num to 112. Set nopgchange=true to prevent further autoscaler interference.
|
|
|
|
### BlueFS Spillover on osd.8
|
|
osd.8 spilled 128KiB metadata from db device (2.1GiB of 30GiB) to slow device.
|
|
Cosmetic warning, no data risk.
|
|
|
|
### Slow Operations on osd.2 and osd.7
|
|
osd.2 (81% full, fragmentation 0.80) and osd.7 (small HDD) experience slow BlueStore ops.
|
|
osd.2 NVMe has 92% wear — candidate for replacement.
|
|
|
|
## Access
|
|
- SSH to Proxmox hosts: `ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.92`
|
|
- Ceph commands: `ceph status`, `ceph osd tree`, `ceph pg dump pgs`
|
|
- Mon nodes: proxmox5, proxmox7, proxmox4
|
|
- Mgr: proxmox5 (active)
|