# Ceph EC Pool Optimization for Mixed HDD Sizes — 2026-07-13

## Problem

When a Ceph cluster has HDDs of different sizes (1 TB and 3 TB) and uses
erasure coding, the 3 TB drives are severely underutilized. This is because
EC stores equally-sized chunks per PG on each OSD — regardless of OSD
capacity. A 1 TB OSD and a 3 TB OSD in the same PG each store 1/(k+m) of
the object.

### Why `ceph osd reweight` Doesn't Help for EC

With EC k=4, m=1 (5 OSDs per PG) and 6 HDD OSDs total, CRUSH selects 5 of 6.
`ceph osd reweight` only changes the probability of being selected — but
with 5/6 selection, even a heavily downweighted OSD is chosen in most PGs.
The effect is marginal compared to replicated pools where each OSD
independently participates.

### Symptoms

| Signal | Cause |
|--------|-------|
| 3 TB OSD at 2% while 1 TB OSD at 94% | Equal chunk sizes fill small OSDs first |
| Same PG count on 3 TB and 1 TB OSDs | CRUSH weight not proportional, or EC masking the effect |
| `backfill_toofull` on small OSDs | Small OSDs hit full ratio before large ones fill |
| `ceph osd reweight` has no visible effect | EC selection ratio too tight (5/6) |

## Architectural Solution

Separate large and small HDDs into different device classes, place them
under a dedicated CRUSH root, and create an EC pool that only uses the
large HDDs with a smaller k+m profile.

### New CRUSH Topology

```
root default                    root hdd-root (NEW)
├── host proxmox2-5 (SSDs)       ├── host proxmox1    → osd.1  (hdd-large, 3.6 TB)
│                               ├── host ubuntu       → osd.6  (hdd-large, 2.8 TB)
│                               │                   └ → osd.8  (hdd-large, 2.8 TB)
│                               ├── host n5pro        → osd.11 (hdd-large, 2.8 TB)
│                               ├── host proxmox6     → osd.10 (hdd-small, 1 TB)
│                               └── host proxmox7     → osd.7  (hdd-small, 1 TB)
```

### Key Insight: `take` with vs without Class Filter

- `take hdd-root` (no class filter) → selects ALL HDDs (large + small) → for replicated pools
- `take hdd-root~hdd-large` → selects only large HDDs → for EC pool

This allows large HDDs to serve both EC and replicated pools, while small
HDDs only participate in replicated pools.

### New EC Profile

| Parameter | Old (ec41-hdd) | New (ec31-hdd-large) |
|-----------|----------------|---------------------|
| k | 4 | **3** |
| m | 1 | **1** |
| OSDs per PG | 5 | **4** |
| Overhead | 20% | **33%** |
| Available OSDs | 6 (2 too small) | **4 (all fully utilized)** |
| Failure tolerance | 1 OSD | 1 OSD |

Trade-off: EC overhead increases from 20% → 33%, but all 4 large OSDs are
fully utilized. Net usable capacity is HIGHER because the 3 TB drives
actually fill up instead of sitting at 2%.

## Migration Plan (6 Phases)

### Phase 0: Wait for Current Rebalance

```bash
ceph -s  # until "misplaced 0%" and "recovery: 0 B/s"
```

### Phase 1: CRUSH Structure (Triggers Remapping)

```bash
# 1a. Device classes
ceph osd crush set-device-class hdd-large osd.1 osd.6 osd.8 osd.11
ceph osd crush set-device-class hdd-small osd.7 osd.10

# 1b. New root
ceph osd crush add-bucket hdd-root root

# 1c. Move HDD hosts to hdd-root (triggers remapping!)
ceph osd crush move proxmox1 root=hdd-root
ceph osd crush move ubuntu root=hdd-root
ceph osd crush move n5pro root=hdd-root
ceph osd crush move proxmox6 root=hdd-root
ceph osd crush move proxmox7 root=hdd-root
```

⚠️ Step 1c triggers IMMEDIATE remapping of all HDD PGs. For gentler
migration: move hosts one at a time with 30-min intervals between each,
waiting for rebalance to stabilize.

### Phase 2: New Rules + EC Profile

```bash
# 2a. New EC profile (k=3, m=1, large HDDs only)
ceph osd erasure-code-profile set ec31-hdd-large \
  k=3 m=1 \
  crush-root=hdd-root \
  crush-device-class=hdd-large \
  crush-failure-domain=osd \
  plugin=jerasure technique=reed_sol_van

# 2b. New EC rule
ceph osd crush rule create-erasure media_ec_large ec31-hdd-large

# 2c. New replicated rule (ALL HDDs, no class filter)
ceph osd crush rule create-replicated replicated_hdd_all hdd-root host
```

### Phase 3: Switch Replicated Pools

```bash
ceph osd pool set hdd_disk crush_rule replicated_hdd_all
ceph osd pool set rbd crush_rule replicated_hdd_all
ceph osd pool set tm_disks crush_rule replicated_hdd_all
```

### Phase 4: EC Pool Migration (Critical)

EC pools cannot change their k/m profile. Must create a new pool and
migrate data.

#### RBD Online Migration (preferred — VMs keep running)

```bash
# 4a. Create new EC pool
ceph osd pool create media_ec_new 128 128 erasure ec31-hdd-large
ceph osd pool set media_ec_new crush_rule media_ec_large
ceph osd pool set media_ec_new ec_overwrites
ceph osd pool application enable media_ec_new rbd

# 4b. Create new metadata pool
ceph osd pool create media_meta_new 32
ceph osd pool set media_meta_new crush_rule replicated_ssd

# 4c. Migrate each RBD image (online, VM keeps running)
for img in vm-114-disk-0 vm-136-disk-0 vm-137-disk-0 vm-138-disk-1 vm-143-disk-0; do
  rbd migration prepare media_meta/${img} media_meta_new/${img} \
    --data-pool media_ec_new
  rbd migration execute media_meta_new/${img}
  rbd migration commit media_meta_new/${img}
done
```

Per-image timeline: `prepare` (instant, starts copy) → `execute` (copies
data, 30-60 min per 500 GiB image) → `commit` (atomic switch, old image
deleted). VMs continue running throughout — the migration is transparent.

#### Offline Alternative (faster, VM stopped)

Stop VM, `rbd copy --data-pool media_ec_new old_pool/img new_meta_pool/img`,
update VM disk config, start VM. See `rbd-pool-migration-2026-07.md`.

### Phase 5: Cleanup

```bash
ceph osd pool delete media_ec --yes-i-really-really-mean-it
ceph osd crush rule rm media_ec
ceph osd crush rule rm replicated_hdd
ceph osd crush rm proxmox       # empty host bucket
ceph osd crush rm px-tmp20      # empty host bucket
ceph osd erasure-code-profile rm ec41-hdd
ceph osd pool rename media_ec_new media_ec
ceph osd pool rename media_meta_new media_meta
```

### Phase 6: Correct CRUSH Weights

```bash
# Now safe — small HDDs no longer in EC pool
ceph osd crush reweight osd.7 0.96   # was 0.50, should be ~0.96 for 1 TB
ceph osd crush reweight osd.10 0.96  # was 0.30, should be ~0.96 for 1 TB
```

## Capacity Analysis

| Pool | Raw Capacity | Usable | Currently Used | Free |
|------|-------------|--------|----------------|------|
| media_ec (EC 3+1) | 4× ~2.9 TB = 11.6 TB | 8.7 TB | 764 GiB | ~7.9 TB |
| hdd_disk (Repl 3) | 15 TB | 5.0 TB | 1.0 TiB | ~4.0 TB |

## Safety-Net

- Backup CRUSH map: `ceph osd getcrushmap -o /tmp/crushmap.backup`
- RBD snapshots of critical images before migration
- Rollback: restore CRUSH map, revert pool rules

## Pitfalls

### EC Pools Cannot Change k/m In Place

An EC pool's erasure-code profile is fixed at creation. Changing k or m
requires creating a new pool and migrating data. Do NOT attempt
`ceph osd pool set media_ec erasure-code-profile ec31-hdd-large` — it
will fail or corrupt data.

### `rbd migration prepare` Needs Both Pools

The `prepare` command takes `source_image dest_image` where dest is in
the new metadata pool. The `--data-pool` flag specifies where the EC data
chunks go. The metadata pool stores RBD headers (small, replicated).

### Moving Hosts Between Roots Triggers Full Remap

Every `ceph osd crush move` changes the CRUSH hash inputs for all PGs in
that subtree. Moving 5 hosts at once causes massive simultaneous remapping.
Move hosts individually with rebalance gaps for production clusters.

### Device Class Must Match Reality

`ceph osd crush set-device-class hdd-large osd.X` overrides the
auto-detected class. If the OSD is later recreated, it reverts to
auto-detected `hdd`. Re-apply the custom class after any OSD rebuild.

## Gitea Issue Creation Pattern

This plan was saved as Gitea issue #14 in `dominik/iac-homelab`. Pattern
for large issue bodies:

```bash
# 1. Write JSON to temp file locally
cat > /tmp/issue.json << 'EOF'
{"title":"...", "body":"...markdown...", "labels":[]}
EOF

# 2. SCP to Gitea host (CT108 on proxmox6)
scp -i ~/.ssh/id_ed25519_proxmox /tmp/issue.json root@10.0.30.105:/tmp/

# 3. POST via curl on the Gitea host (avoids remote curl auth issues)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.105 \
  "curl -s -X POST -H 'Authorization: token TOKEN' \
   -H 'Content-Type: application/json' \
   -d @/tmp/issue.json \
   http://localhost:3000/api/v1/repos/dominik/iac-homelab/issues"
```

Using a file (`-d @file`) avoids shell escaping issues with large markdown
bodies containing backticks, quotes, and special characters.

See also: `references/gitea-api-and-skills-versioning-2026-07.md` for
token generation and full API patterns.