Debian
01bd921ced
- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
249 lines
8.7 KiB
Markdown
249 lines
8.7 KiB
Markdown
# Ceph EC Pool Optimization for Mixed HDD Sizes — 2026-07-13
|
||
|
||
## Problem
|
||
|
||
When a Ceph cluster has HDDs of different sizes (1 TB and 3 TB) and uses
|
||
erasure coding, the 3 TB drives are severely underutilized. This is because
|
||
EC stores equally-sized chunks per PG on each OSD — regardless of OSD
|
||
capacity. A 1 TB OSD and a 3 TB OSD in the same PG each store 1/(k+m) of
|
||
the object.
|
||
|
||
### Why `ceph osd reweight` Doesn't Help for EC
|
||
|
||
With EC k=4, m=1 (5 OSDs per PG) and 6 HDD OSDs total, CRUSH selects 5 of 6.
|
||
`ceph osd reweight` only changes the probability of being selected — but
|
||
with 5/6 selection, even a heavily downweighted OSD is chosen in most PGs.
|
||
The effect is marginal compared to replicated pools where each OSD
|
||
independently participates.
|
||
|
||
### Symptoms
|
||
|
||
| Signal | Cause |
|
||
|--------|-------|
|
||
| 3 TB OSD at 2% while 1 TB OSD at 94% | Equal chunk sizes fill small OSDs first |
|
||
| Same PG count on 3 TB and 1 TB OSDs | CRUSH weight not proportional, or EC masking the effect |
|
||
| `backfill_toofull` on small OSDs | Small OSDs hit full ratio before large ones fill |
|
||
| `ceph osd reweight` has no visible effect | EC selection ratio too tight (5/6) |
|
||
|
||
## Architectural Solution
|
||
|
||
Separate large and small HDDs into different device classes, place them
|
||
under a dedicated CRUSH root, and create an EC pool that only uses the
|
||
large HDDs with a smaller k+m profile.
|
||
|
||
### New CRUSH Topology
|
||
|
||
```
|
||
root default root hdd-root (NEW)
|
||
├── host proxmox2-5 (SSDs) ├── host proxmox1 → osd.1 (hdd-large, 3.6 TB)
|
||
│ ├── host ubuntu → osd.6 (hdd-large, 2.8 TB)
|
||
│ │ └ → osd.8 (hdd-large, 2.8 TB)
|
||
│ ├── host n5pro → osd.11 (hdd-large, 2.8 TB)
|
||
│ ├── host proxmox6 → osd.10 (hdd-small, 1 TB)
|
||
│ └── host proxmox7 → osd.7 (hdd-small, 1 TB)
|
||
```
|
||
|
||
### Key Insight: `take` with vs without Class Filter
|
||
|
||
- `take hdd-root` (no class filter) → selects ALL HDDs (large + small) → for replicated pools
|
||
- `take hdd-root~hdd-large` → selects only large HDDs → for EC pool
|
||
|
||
This allows large HDDs to serve both EC and replicated pools, while small
|
||
HDDs only participate in replicated pools.
|
||
|
||
### New EC Profile
|
||
|
||
| Parameter | Old (ec41-hdd) | New (ec31-hdd-large) |
|
||
|-----------|----------------|---------------------|
|
||
| k | 4 | **3** |
|
||
| m | 1 | **1** |
|
||
| OSDs per PG | 5 | **4** |
|
||
| Overhead | 20% | **33%** |
|
||
| Available OSDs | 6 (2 too small) | **4 (all fully utilized)** |
|
||
| Failure tolerance | 1 OSD | 1 OSD |
|
||
|
||
Trade-off: EC overhead increases from 20% → 33%, but all 4 large OSDs are
|
||
fully utilized. Net usable capacity is HIGHER because the 3 TB drives
|
||
actually fill up instead of sitting at 2%.
|
||
|
||
## Migration Plan (6 Phases)
|
||
|
||
### Phase 0: Wait for Current Rebalance
|
||
|
||
```bash
|
||
ceph -s # until "misplaced 0%" and "recovery: 0 B/s"
|
||
```
|
||
|
||
### Phase 1: CRUSH Structure (Triggers Remapping)
|
||
|
||
```bash
|
||
# 1a. Device classes
|
||
ceph osd crush set-device-class hdd-large osd.1 osd.6 osd.8 osd.11
|
||
ceph osd crush set-device-class hdd-small osd.7 osd.10
|
||
|
||
# 1b. New root
|
||
ceph osd crush add-bucket hdd-root root
|
||
|
||
# 1c. Move HDD hosts to hdd-root (triggers remapping!)
|
||
ceph osd crush move proxmox1 root=hdd-root
|
||
ceph osd crush move ubuntu root=hdd-root
|
||
ceph osd crush move n5pro root=hdd-root
|
||
ceph osd crush move proxmox6 root=hdd-root
|
||
ceph osd crush move proxmox7 root=hdd-root
|
||
```
|
||
|
||
⚠️ Step 1c triggers IMMEDIATE remapping of all HDD PGs. For gentler
|
||
migration: move hosts one at a time with 30-min intervals between each,
|
||
waiting for rebalance to stabilize.
|
||
|
||
### Phase 2: New Rules + EC Profile
|
||
|
||
```bash
|
||
# 2a. New EC profile (k=3, m=1, large HDDs only)
|
||
ceph osd erasure-code-profile set ec31-hdd-large \
|
||
k=3 m=1 \
|
||
crush-root=hdd-root \
|
||
crush-device-class=hdd-large \
|
||
crush-failure-domain=osd \
|
||
plugin=jerasure technique=reed_sol_van
|
||
|
||
# 2b. New EC rule
|
||
ceph osd crush rule create-erasure media_ec_large ec31-hdd-large
|
||
|
||
# 2c. New replicated rule (ALL HDDs, no class filter)
|
||
ceph osd crush rule create-replicated replicated_hdd_all hdd-root host
|
||
```
|
||
|
||
### Phase 3: Switch Replicated Pools
|
||
|
||
```bash
|
||
ceph osd pool set hdd_disk crush_rule replicated_hdd_all
|
||
ceph osd pool set rbd crush_rule replicated_hdd_all
|
||
ceph osd pool set tm_disks crush_rule replicated_hdd_all
|
||
```
|
||
|
||
### Phase 4: EC Pool Migration (Critical)
|
||
|
||
EC pools cannot change their k/m profile. Must create a new pool and
|
||
migrate data.
|
||
|
||
#### RBD Online Migration (preferred — VMs keep running)
|
||
|
||
```bash
|
||
# 4a. Create new EC pool
|
||
ceph osd pool create media_ec_new 128 128 erasure ec31-hdd-large
|
||
ceph osd pool set media_ec_new crush_rule media_ec_large
|
||
ceph osd pool set media_ec_new ec_overwrites
|
||
ceph osd pool application enable media_ec_new rbd
|
||
|
||
# 4b. Create new metadata pool
|
||
ceph osd pool create media_meta_new 32
|
||
ceph osd pool set media_meta_new crush_rule replicated_ssd
|
||
|
||
# 4c. Migrate each RBD image (online, VM keeps running)
|
||
for img in vm-114-disk-0 vm-136-disk-0 vm-137-disk-0 vm-138-disk-1 vm-143-disk-0; do
|
||
rbd migration prepare media_meta/${img} media_meta_new/${img} \
|
||
--data-pool media_ec_new
|
||
rbd migration execute media_meta_new/${img}
|
||
rbd migration commit media_meta_new/${img}
|
||
done
|
||
```
|
||
|
||
Per-image timeline: `prepare` (instant, starts copy) → `execute` (copies
|
||
data, 30-60 min per 500 GiB image) → `commit` (atomic switch, old image
|
||
deleted). VMs continue running throughout — the migration is transparent.
|
||
|
||
#### Offline Alternative (faster, VM stopped)
|
||
|
||
Stop VM, `rbd copy --data-pool media_ec_new old_pool/img new_meta_pool/img`,
|
||
update VM disk config, start VM. See `rbd-pool-migration-2026-07.md`.
|
||
|
||
### Phase 5: Cleanup
|
||
|
||
```bash
|
||
ceph osd pool delete media_ec --yes-i-really-really-mean-it
|
||
ceph osd crush rule rm media_ec
|
||
ceph osd crush rule rm replicated_hdd
|
||
ceph osd crush rm proxmox # empty host bucket
|
||
ceph osd crush rm px-tmp20 # empty host bucket
|
||
ceph osd erasure-code-profile rm ec41-hdd
|
||
ceph osd pool rename media_ec_new media_ec
|
||
ceph osd pool rename media_meta_new media_meta
|
||
```
|
||
|
||
### Phase 6: Correct CRUSH Weights
|
||
|
||
```bash
|
||
# Now safe — small HDDs no longer in EC pool
|
||
ceph osd crush reweight osd.7 0.96 # was 0.50, should be ~0.96 for 1 TB
|
||
ceph osd crush reweight osd.10 0.96 # was 0.30, should be ~0.96 for 1 TB
|
||
```
|
||
|
||
## Capacity Analysis
|
||
|
||
| Pool | Raw Capacity | Usable | Currently Used | Free |
|
||
|------|-------------|--------|----------------|------|
|
||
| media_ec (EC 3+1) | 4× ~2.9 TB = 11.6 TB | 8.7 TB | 764 GiB | ~7.9 TB |
|
||
| hdd_disk (Repl 3) | 15 TB | 5.0 TB | 1.0 TiB | ~4.0 TB |
|
||
|
||
## Safety-Net
|
||
|
||
- Backup CRUSH map: `ceph osd getcrushmap -o /tmp/crushmap.backup`
|
||
- RBD snapshots of critical images before migration
|
||
- Rollback: restore CRUSH map, revert pool rules
|
||
|
||
## Pitfalls
|
||
|
||
### EC Pools Cannot Change k/m In Place
|
||
|
||
An EC pool's erasure-code profile is fixed at creation. Changing k or m
|
||
requires creating a new pool and migrating data. Do NOT attempt
|
||
`ceph osd pool set media_ec erasure-code-profile ec31-hdd-large` — it
|
||
will fail or corrupt data.
|
||
|
||
### `rbd migration prepare` Needs Both Pools
|
||
|
||
The `prepare` command takes `source_image dest_image` where dest is in
|
||
the new metadata pool. The `--data-pool` flag specifies where the EC data
|
||
chunks go. The metadata pool stores RBD headers (small, replicated).
|
||
|
||
### Moving Hosts Between Roots Triggers Full Remap
|
||
|
||
Every `ceph osd crush move` changes the CRUSH hash inputs for all PGs in
|
||
that subtree. Moving 5 hosts at once causes massive simultaneous remapping.
|
||
Move hosts individually with rebalance gaps for production clusters.
|
||
|
||
### Device Class Must Match Reality
|
||
|
||
`ceph osd crush set-device-class hdd-large osd.X` overrides the
|
||
auto-detected class. If the OSD is later recreated, it reverts to
|
||
auto-detected `hdd`. Re-apply the custom class after any OSD rebuild.
|
||
|
||
## Gitea Issue Creation Pattern
|
||
|
||
This plan was saved as Gitea issue #14 in `dominik/iac-homelab`. Pattern
|
||
for large issue bodies:
|
||
|
||
```bash
|
||
# 1. Write JSON to temp file locally
|
||
cat > /tmp/issue.json << 'EOF'
|
||
{"title":"...", "body":"...markdown...", "labels":[]}
|
||
EOF
|
||
|
||
# 2. SCP to Gitea host (CT108 on proxmox6)
|
||
scp -i ~/.ssh/id_ed25519_proxmox /tmp/issue.json root@10.0.30.105:/tmp/
|
||
|
||
# 3. POST via curl on the Gitea host (avoids remote curl auth issues)
|
||
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.105 \
|
||
"curl -s -X POST -H 'Authorization: token TOKEN' \
|
||
-H 'Content-Type: application/json' \
|
||
-d @/tmp/issue.json \
|
||
http://localhost:3000/api/v1/repos/dominik/iac-homelab/issues"
|
||
```
|
||
|
||
Using a file (`-d @file`) avoids shell escaping issues with large markdown
|
||
bodies containing backticks, quotes, and special characters.
|
||
|
||
See also: `references/gitea-api-and-skills-versioning-2026-07.md` for
|
||
token generation and full API patterns.
|