From 9b0c32482ba3d6e1a2037314c92f530f7cacce22 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dominik=20Sch=C3=B6n?= Date: Sat, 25 Jul 2026 05:48:01 +0000 Subject: [PATCH] Fix: merge ceph.md into ceph-cluster.md (eliminate duplicate) --- systems/ceph-cluster.md | 101 +++++++++++++++++++++++++++++++++------- systems/ceph.md | 82 -------------------------------- 2 files changed, 83 insertions(+), 100 deletions(-) delete mode 100644 systems/ceph.md diff --git a/systems/ceph-cluster.md b/systems/ceph-cluster.md index abfdf56..ccfaf80 100644 --- a/systems/ceph-cluster.md +++ b/systems/ceph-cluster.md @@ -3,37 +3,102 @@ title: Ceph Cluster category: systems tags: [ceph, storage, rbd, ec-pool, osd] created: "2026-07-24" -modified: "2026-07-24" +modified: "2026-07-25" --- # Ceph Cluster -## Übersicht -- **14 OSDs** (HDD + NVMe/SSD混合) -- **Health:** HEALTH_WARN (BlueFS spillover osd.8, slow ops, nearfull) -- Läuft auf Proxmox PVE Hosts (integriert via pveceph) +## Overview +- **Cluster ID**: 204c8171-e0b1-4f40-9de2-a7cfe4ef68d9 +- **Health**: HEALTH_WARN (rebalancing after osd.7 reweight: 43 PGs backfilling at ~26 MiB/s) +- **Nodes**: 9 Proxmox hosts, 3 MONs (proxmox5/7/4), MGR on proxmox5 +- **OSDs**: 15 (8 SSD, 6 HDD, 1 new SSD on ms-a2-2), all up/in +- **Capacity**: 6.0 TiB used / 21 TiB total + +## OSD Layout + +| OSD | Class | Size | Host | Reweight | Notes | +|-----|-------|------|------|----------|-------| +| 0 | ssd | 188 GB | proxmox2 | 0.50 | 92% Wear, Critical Warning | +| 1 | hdd | 3.7 TiB | n5pro | 1.0 | Large HDD | +| 2 | ssd | 233 GB | proxmox2 | 0.90 | Slow ops, 81% full | +| 3 | ssd | 238 GB | proxmox4 | 0.95 | | +| 4 | ssd | 233 GB | proxmox3 | 0.95 | | +| 5 | ssd | 238 GB | proxmox5 | 0.90 | 80% full | +| 6 | hdd | 2.8 TiB | ubuntu | 1.0 | Large HDD | +| 7 | hdd | 500 GB | proxmox7 | 1.0 | Was 0.80, reweighted 2026-07-24 | +| 8 | hdd | 2.8 TiB | ubuntu | 1.0 | BlueFS spillover | +| 9 | ssd | 1.9 TiB | n5pro | 1.0 | | +| 10 | hdd | 300 GB | proxmox6 | 1.0 | Very small HDD | +| 11 | hdd | 2.8 TiB | n5pro | 1.0 | Large HDD | +| 12 | ssd | 1.9 TiB | n5pro | 1.0 | | +| 13 | ssd | 1.8 TiB | ms-a2-1 | 1.0 | | +| 14 | ssd | 1.8 TiB | ms-a2-2 | 1.0 | New 2026-07-24, nvme0n1 | ## Pools -| Pool | Type | Usage | -|------|------|-------| -| ceph-flash | replicated | K8s CSI (fast storage) | -| ceph-hdd | replicated | Bulk storage | -| media_ec | EC k=4+m=1 | Media (photos, backups) | -## Bekannte Probleme -- **osd.0 NVMe** — 92% Wear, Critical Warning → Austausch planen -- **osd.2/5** — 93-94% full → entlasten oder neue OSDs -- **EC Pool k=4+m=1** — mit 5 OSDs kein Rebalance Headroom. Siehe Solution Doc: `docs/solutions/architecture/2026-07-12-ceph-ec-pool-no-rebalance-headroom.md` -- **BlueFS Spillover** osd.8 — `ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-8` +| Pool | Name | Type | Size | Min | CRUSH Rule | PGs | Notes | +|------|------|------|------|-----|------------|-----|-------| +| 1 | cephfs_data | replicated | 3 | 2 | 0 (any) | 32 | autoscale off | +| 2 | cephfs_metadata | replicated | 3 | 2 | 2 (ssd) | 32 | autoscale off | +| 3 | vm_disks | replicated | 3 | 2 | 2 (ssd) | 128 | autoscale on | +| 4 | .mgr | replicated | 3 | 2 | 2 (ssd) | 1 | | +| 5 | rbd | replicated | 3 | 2 | 1 (hdd) | 32 | autoscale on | +| 6 | hdd_disk | replicated | 3 | 2 | 1 (hdd) | 112 | nopgchange=true (was 120, equalized to 112) | +| 7 | tm_disks | replicated | 2 | 2 | 1 (hdd) | 128 | target_size 2TiB | +| 8 | media_ec | erasure 4+1 | 5 | 4 | 3 (hdd, osd-level) | 128 | ec_overwrites | +| 9 | media_meta | replicated | 3 | 2 | 0 (any) | 32 | | +| 10 | .rgw.root | replicated | 3 | 2 | 0 (any) | 1 | | -## CRUSH Topology -- Device Classes: hdd, ssd, nvme -- EC Pools benötigen Device-Class-Separation (mixed disk sizes → Anti-Pattern: artificially low weights) +## CRUSH Rules +- **Rule 0** (replicated_rule): default root, host-level placement +- **Rule 1** (replicated_hdd): default~hdd, host-level placement +- **Rule 2** (replicated_ssd): default~ssd, host-level placement +- **Rule 3** (media_ec): default~hdd, OSD-level placement (choose_indep) + +## Known Issues + +### Weight Imbalance Causing Placement Failures (2026-07-24) +HDD hosts have extreme weight disparity: n5pro=10.15TB, ubuntu=5.49TB, proxmox7=0.50TB, proxmox6=0.30TB. +CRUSH host-level selection (rule 1) often picks only 2 of 4 HDD hosts → up sets with 2 OSDs instead of 3. +Result: PGs stuck in `active+clean+remapped` because up set < min_size. + +**Mitigation (2026-07-24)**: +1. Reweighted osd.7 from 0.80 → 1.0 → fixed EC pool 8.3d (NONE → osd.7) +2. Equalized pool 6 pg_num 120 → 112 + nopgchange=true +3. Manual pg-upmap for stuck PGs: 5.13 → [1,6,7], 6.6c → [11,8,7], 6.58 → [11,6,7] +4. All `clean+remapped` eliminated. Triggered rebalancing wave (43 PGs backfilling at 26 MiB/s). + +**Long-term**: Small HDDs (osd.7 0.5TB, osd.10 0.3TB) cause CRUSH placement failures. Replace with larger disks or create separate CRUSH root for large HDDs only. + +### Pool 6 pg_num/pgp_num Mismatch (Fixed 2026-07-24) +Pool hdd_disk had pg_num=120, pgp_num=112 (autoscaler reducing to 32). +Equalized pg_num to 112. Set nopgchange=true to prevent further autoscaler interference. + +### BlueFS Spillover on osd.8 +osd.8 spilled 128KiB metadata from db device (2.1GiB of 30GiB) to slow device. +Cosmetic warning, no data risk. Fix: `ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-8` + +### Slow Operations on osd.2 and osd.7 +osd.2 (81% full, fragmentation 0.80) and osd.7 (small HDD) experience slow BlueStore ops. +osd.2 NVMe has 92% wear — candidate for replacement. + +### osd.0 NVMe Wear +92% Wear, Critical Warning → Austausch planen. + +### EC Pool k=4+m=1 — No Rebalance Headroom +With 5 OSDs kein Rebalance Headroom. Siehe Solution Doc: `docs/solutions/architecture/2026-07-12-ceph-ec-pool-no-rebalance-headroom.md` ## RBD Management - Proxmox RBD Double-Mount Deadlock Pitfall: Niemals `pct mount` und `pct exec` gleichzeitig auf demselben Container - Siehe Solution Doc: `docs/solutions/bug-fixes/2026-07-23-proxmox-rbd-double-mount-deadlock.md` +## Access +- SSH to Proxmox hosts: `ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.92` +- Ceph commands: `ceph status`, `ceph osd tree`, `ceph pg dump pgs` +- Mon nodes: proxmox5, proxmox7, proxmox4 +- Mgr: proxmox5 (active) + ## Related Skills - `ceph-cluster-administration` (devops) diff --git a/systems/ceph.md b/systems/ceph.md deleted file mode 100644 index 8d7261f..0000000 --- a/systems/ceph.md +++ /dev/null @@ -1,82 +0,0 @@ -# Ceph Cluster - -## Overview -- **Cluster ID**: 204c8171-e0b1-4f40-9de2-a7cfe4ef68d9 -- **Health**: HEALTH_WARN (rebalancing after osd.7 reweight: 43 PGs backfilling at ~26 MiB/s) -- **Nodes**: 9 Proxmox hosts, 3 MONs (proxmox5/7/4), MGR on proxmox5 -- **OSDs**: 15 (8 SSD, 6 HDD, 1 new SSD on ms-a2-2), all up/in -- **Capacity**: 6.0 TiB used / 21 TiB total - -## OSD Layout - -| OSD | Class | Size | Host | Reweight | Notes | -|-----|-------|------|------|----------|-------| -| 0 | ssd | 188 GB | proxmox2 | 0.50 | | -| 1 | hdd | 3.7 TiB | n5pro | 1.0 | Large HDD | -| 2 | ssd | 233 GB | proxmox2 | 0.90 | Slow ops, 81% full | -| 3 | ssd | 238 GB | proxmox4 | 0.95 | | -| 4 | ssd | 233 GB | proxmox3 | 0.95 | | -| 5 | ssd | 238 GB | proxmox5 | 0.90 | 80% full | -| 6 | hdd | 2.8 TiB | ubuntu | 1.0 | Large HDD | -| 7 | hdd | 500 GB | proxmox7 | 1.0 | Was 0.80, reweighted 2026-07-24 | -| 8 | hdd | 2.8 TiB | ubuntu | 1.0 | BlueFS spillover | -| 9 | ssd | 1.9 TiB | n5pro | 1.0 | | -| 10 | hdd | 300 GB | proxmox6 | 1.0 | Very small HDD | -| 11 | hdd | 2.8 TiB | n5pro | 1.0 | Large HDD | -| 12 | ssd | 1.9 TiB | n5pro | 1.0 | | -| 13 | ssd | 1.8 TiB | ms-a2-1 | 1.0 | | -| 14 | ssd | 1.8 TiB | ms-a2-2 | 1.0 | New 2026-07-24, nvme0n1 | - -## Pools - -| Pool | Name | Type | Size | Min | CRUSH Rule | PGs | Notes | -|------|------|------|------|-----|------------|-----|-------| -| 1 | cephfs_data | replicated | 3 | 2 | 0 (any) | 32 | autoscale off | -| 2 | cephfs_metadata | replicated | 3 | 2 | 2 (ssd) | 32 | autoscale off | -| 3 | vm_disks | replicated | 3 | 2 | 2 (ssd) | 128 | autoscale on | -| 4 | .mgr | replicated | 3 | 2 | 2 (ssd) | 1 | | -| 5 | rbd | replicated | 3 | 2 | 1 (hdd) | 32 | autoscale on | -| 6 | hdd_disk | replicated | 3 | 2 | 1 (hdd) | 112 | nopgchange=true (was 120, equalized to 112) | -| 7 | tm_disks | replicated | 2 | 2 | 1 (hdd) | 128 | target_size 2TiB | -| 8 | media_ec | erasure 4+1 | 5 | 4 | 3 (hdd, osd-level) | 128 | ec_overwrites | -| 9 | media_meta | replicated | 3 | 2 | 0 (any) | 32 | | -| 10 | .rgw.root | replicated | 3 | 2 | 0 (any) | 1 | | - -## CRUSH Rules -- **Rule 0** (replicated_rule): default root, host-level placement -- **Rule 1** (replicated_hdd): default~hdd, host-level placement -- **Rule 2** (replicated_ssd): default~ssd, host-level placement -- **Rule 3** (media_ec): default~hdd, OSD-level placement (choose_indep) - -## Known Issues - -### Weight Imbalance Causing Placement Failures (2026-07-24) -HDD hosts have extreme weight disparity: n5pro=10.15TB, ubuntu=5.49TB, proxmox7=0.50TB, proxmox6=0.30TB. -CRUSH host-level selection (rule 1) often picks only 2 of 4 HDD hosts → up sets with 2 OSDs instead of 3. -Result: PGs stuck in `active+clean+remapped` because up set < min_size. - -**Mitigation (2026-07-24)**: -1. Reweighted osd.7 from 0.80 → 1.0 → fixed EC pool 8.3d (NONE → osd.7) -2. Equalized pool 6 pg_num 120 → 112 + nopgchange=true -3. Manual pg-upmap for stuck PGs: 5.13 → [1,6,7], 6.6c → [11,8,7], 6.58 → [11,6,7] -4. All `clean+remapped` eliminated. Triggered rebalancing wave (43 PGs backfilling at 26 MiB/s). - -**Long-term**: Small HDDs (osd.7 0.5TB, osd.10 0.3TB) cause CRUSH placement failures. Replace with larger disks or create separate CRUSH root for large HDDs only. - -### Pool 6 pg_num/pgp_num Mismatch (Fixed 2026-07-24) -Pool hdd_disk had pg_num=120, pgp_num=112 (autoscaler reducing to 32). -Equalized pg_num to 112. Set nopgchange=true to prevent further autoscaler interference. - -### BlueFS Spillover on osd.8 -osd.8 spilled 128KiB metadata from db device (2.1GiB of 30GiB) to slow device. -Cosmetic warning, no data risk. - -### Slow Operations on osd.2 and osd.7 -osd.2 (81% full, fragmentation 0.80) and osd.7 (small HDD) experience slow BlueStore ops. -osd.2 NVMe has 92% wear — candidate for replacement. - -## Access -- SSH to Proxmox hosts: `ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.92` -- Ceph commands: `ceph status`, `ceph osd tree`, `ceph pg dump pgs` -- Mon nodes: proxmox5, proxmox7, proxmox4 -- Mgr: proxmox5 (active)