Debian
01bd921ced
- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
8.7 KiB
Ceph EC Pool Optimization for Mixed HDD Sizes — 2026-07-13
Problem
When a Ceph cluster has HDDs of different sizes (1 TB and 3 TB) and uses erasure coding, the 3 TB drives are severely underutilized. This is because EC stores equally-sized chunks per PG on each OSD — regardless of OSD capacity. A 1 TB OSD and a 3 TB OSD in the same PG each store 1/(k+m) of the object.
Why ceph osd reweight Doesn't Help for EC
With EC k=4, m=1 (5 OSDs per PG) and 6 HDD OSDs total, CRUSH selects 5 of 6.
ceph osd reweight only changes the probability of being selected — but
with 5/6 selection, even a heavily downweighted OSD is chosen in most PGs.
The effect is marginal compared to replicated pools where each OSD
independently participates.
Symptoms
| Signal | Cause |
|---|---|
| 3 TB OSD at 2% while 1 TB OSD at 94% | Equal chunk sizes fill small OSDs first |
| Same PG count on 3 TB and 1 TB OSDs | CRUSH weight not proportional, or EC masking the effect |
backfill_toofull on small OSDs |
Small OSDs hit full ratio before large ones fill |
ceph osd reweight has no visible effect |
EC selection ratio too tight (5/6) |
Architectural Solution
Separate large and small HDDs into different device classes, place them under a dedicated CRUSH root, and create an EC pool that only uses the large HDDs with a smaller k+m profile.
New CRUSH Topology
root default root hdd-root (NEW)
├── host proxmox2-5 (SSDs) ├── host proxmox1 → osd.1 (hdd-large, 3.6 TB)
│ ├── host ubuntu → osd.6 (hdd-large, 2.8 TB)
│ │ └ → osd.8 (hdd-large, 2.8 TB)
│ ├── host n5pro → osd.11 (hdd-large, 2.8 TB)
│ ├── host proxmox6 → osd.10 (hdd-small, 1 TB)
│ └── host proxmox7 → osd.7 (hdd-small, 1 TB)
Key Insight: take with vs without Class Filter
take hdd-root(no class filter) → selects ALL HDDs (large + small) → for replicated poolstake hdd-root~hdd-large→ selects only large HDDs → for EC pool
This allows large HDDs to serve both EC and replicated pools, while small HDDs only participate in replicated pools.
New EC Profile
| Parameter | Old (ec41-hdd) | New (ec31-hdd-large) |
|---|---|---|
| k | 4 | 3 |
| m | 1 | 1 |
| OSDs per PG | 5 | 4 |
| Overhead | 20% | 33% |
| Available OSDs | 6 (2 too small) | 4 (all fully utilized) |
| Failure tolerance | 1 OSD | 1 OSD |
Trade-off: EC overhead increases from 20% → 33%, but all 4 large OSDs are fully utilized. Net usable capacity is HIGHER because the 3 TB drives actually fill up instead of sitting at 2%.
Migration Plan (6 Phases)
Phase 0: Wait for Current Rebalance
ceph -s # until "misplaced 0%" and "recovery: 0 B/s"
Phase 1: CRUSH Structure (Triggers Remapping)
# 1a. Device classes
ceph osd crush set-device-class hdd-large osd.1 osd.6 osd.8 osd.11
ceph osd crush set-device-class hdd-small osd.7 osd.10
# 1b. New root
ceph osd crush add-bucket hdd-root root
# 1c. Move HDD hosts to hdd-root (triggers remapping!)
ceph osd crush move proxmox1 root=hdd-root
ceph osd crush move ubuntu root=hdd-root
ceph osd crush move n5pro root=hdd-root
ceph osd crush move proxmox6 root=hdd-root
ceph osd crush move proxmox7 root=hdd-root
⚠️ Step 1c triggers IMMEDIATE remapping of all HDD PGs. For gentler migration: move hosts one at a time with 30-min intervals between each, waiting for rebalance to stabilize.
Phase 2: New Rules + EC Profile
# 2a. New EC profile (k=3, m=1, large HDDs only)
ceph osd erasure-code-profile set ec31-hdd-large \
k=3 m=1 \
crush-root=hdd-root \
crush-device-class=hdd-large \
crush-failure-domain=osd \
plugin=jerasure technique=reed_sol_van
# 2b. New EC rule
ceph osd crush rule create-erasure media_ec_large ec31-hdd-large
# 2c. New replicated rule (ALL HDDs, no class filter)
ceph osd crush rule create-replicated replicated_hdd_all hdd-root host
Phase 3: Switch Replicated Pools
ceph osd pool set hdd_disk crush_rule replicated_hdd_all
ceph osd pool set rbd crush_rule replicated_hdd_all
ceph osd pool set tm_disks crush_rule replicated_hdd_all
Phase 4: EC Pool Migration (Critical)
EC pools cannot change their k/m profile. Must create a new pool and migrate data.
RBD Online Migration (preferred — VMs keep running)
# 4a. Create new EC pool
ceph osd pool create media_ec_new 128 128 erasure ec31-hdd-large
ceph osd pool set media_ec_new crush_rule media_ec_large
ceph osd pool set media_ec_new ec_overwrites
ceph osd pool application enable media_ec_new rbd
# 4b. Create new metadata pool
ceph osd pool create media_meta_new 32
ceph osd pool set media_meta_new crush_rule replicated_ssd
# 4c. Migrate each RBD image (online, VM keeps running)
for img in vm-114-disk-0 vm-136-disk-0 vm-137-disk-0 vm-138-disk-1 vm-143-disk-0; do
rbd migration prepare media_meta/${img} media_meta_new/${img} \
--data-pool media_ec_new
rbd migration execute media_meta_new/${img}
rbd migration commit media_meta_new/${img}
done
Per-image timeline: prepare (instant, starts copy) → execute (copies
data, 30-60 min per 500 GiB image) → commit (atomic switch, old image
deleted). VMs continue running throughout — the migration is transparent.
Offline Alternative (faster, VM stopped)
Stop VM, rbd copy --data-pool media_ec_new old_pool/img new_meta_pool/img,
update VM disk config, start VM. See rbd-pool-migration-2026-07.md.
Phase 5: Cleanup
ceph osd pool delete media_ec --yes-i-really-really-mean-it
ceph osd crush rule rm media_ec
ceph osd crush rule rm replicated_hdd
ceph osd crush rm proxmox # empty host bucket
ceph osd crush rm px-tmp20 # empty host bucket
ceph osd erasure-code-profile rm ec41-hdd
ceph osd pool rename media_ec_new media_ec
ceph osd pool rename media_meta_new media_meta
Phase 6: Correct CRUSH Weights
# Now safe — small HDDs no longer in EC pool
ceph osd crush reweight osd.7 0.96 # was 0.50, should be ~0.96 for 1 TB
ceph osd crush reweight osd.10 0.96 # was 0.30, should be ~0.96 for 1 TB
Capacity Analysis
| Pool | Raw Capacity | Usable | Currently Used | Free |
|---|---|---|---|---|
| media_ec (EC 3+1) | 4× ~2.9 TB = 11.6 TB | 8.7 TB | 764 GiB | ~7.9 TB |
| hdd_disk (Repl 3) | 15 TB | 5.0 TB | 1.0 TiB | ~4.0 TB |
Safety-Net
- Backup CRUSH map:
ceph osd getcrushmap -o /tmp/crushmap.backup - RBD snapshots of critical images before migration
- Rollback: restore CRUSH map, revert pool rules
Pitfalls
EC Pools Cannot Change k/m In Place
An EC pool's erasure-code profile is fixed at creation. Changing k or m
requires creating a new pool and migrating data. Do NOT attempt
ceph osd pool set media_ec erasure-code-profile ec31-hdd-large — it
will fail or corrupt data.
rbd migration prepare Needs Both Pools
The prepare command takes source_image dest_image where dest is in
the new metadata pool. The --data-pool flag specifies where the EC data
chunks go. The metadata pool stores RBD headers (small, replicated).
Moving Hosts Between Roots Triggers Full Remap
Every ceph osd crush move changes the CRUSH hash inputs for all PGs in
that subtree. Moving 5 hosts at once causes massive simultaneous remapping.
Move hosts individually with rebalance gaps for production clusters.
Device Class Must Match Reality
ceph osd crush set-device-class hdd-large osd.X overrides the
auto-detected class. If the OSD is later recreated, it reverts to
auto-detected hdd. Re-apply the custom class after any OSD rebuild.
Gitea Issue Creation Pattern
This plan was saved as Gitea issue #14 in dominik/iac-homelab. Pattern
for large issue bodies:
# 1. Write JSON to temp file locally
cat > /tmp/issue.json << 'EOF'
{"title":"...", "body":"...markdown...", "labels":[]}
EOF
# 2. SCP to Gitea host (CT108 on proxmox6)
scp -i ~/.ssh/id_ed25519_proxmox /tmp/issue.json root@10.0.30.105:/tmp/
# 3. POST via curl on the Gitea host (avoids remote curl auth issues)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.105 \
"curl -s -X POST -H 'Authorization: token TOKEN' \
-H 'Content-Type: application/json' \
-d @/tmp/issue.json \
http://localhost:3000/api/v1/repos/dominik/iac-homelab/issues"
Using a file (-d @file) avoids shell escaping issues with large markdown
bodies containing backticks, quotes, and special characters.
See also: references/gitea-api-and-skills-versioning-2026-07.md for
token generation and full API patterns.