hermes-skills/devops/ceph-cluster-administration/SKILL.md

name, description, version, author, tags, metadata

name	description	version	author	tags	metadata
ceph-cluster-administration	Class-level skill for Ceph cluster administration on Proxmox VE. Covers OSD lifecycle (create/destroy/WAL-DB), CRUSH topology (device classes, roots, rules, weights), EC pools (profiles, mixed disk sizes, migration), recovery & rebalancing (tuning, ratios, monitoring), RBD management (orphan cleanup, migration, inspection), pool management (PG count, full recovery, autoscale), and health diagnosis.	1.0.0	Hermes Agent	ceph proxmox bluestore crush erasure-coding rbd recovery osd storage	hermes tags related_skills ceph proxmox bluestore crush erasure-coding rbd recovery osd storage proxmox-ve-administration rke2-cluster-administration

Ceph Cluster Administration

Scope

This skill covers the Homelab (self-hosted) Ceph cluster. Not noris infrastructure. At noris only remote PBS + S3 Storage exist — neither involves Ceph. Do not confuse the two environments.

Environment

Component	Detail
Environment	Homelab (self-hosted) — NOT noris
PVE Master	10.0.20.10 (SSH: root@10.0.20.10, key ~/.ssh/id_ed25519_proxmox)
PVE Nodes (SSH)	All PVE hosts reachable via 10.0.20.x (storage VLAN), NOT 10.0.30.x (management VLAN — SSH key rejected). px1=.10, px2=.20, px3=.30, px4=.40, px5=.50, px6=.60, px7(n5pro)=.70 or .91
OSDs	11 total (6 HDD + 5 SSD)
HDD mix	4× ~3 TB (osd.1, 6, 8, 11) + 2× ~1 TB (osd.7, 10)
EC Profile	ec41-hdd (k=4, m=1) → migrating to ec31-hdd-large (k=3, m=1)
Pools	media_ec (EC), hdd_disk (Repl 3), rbd (Repl 3), tm_disks (Repl 2), media_meta (Repl 3), vm_disks (Repl 3)
Ratios	nearfull=0.94, backfillfull=0.96, full=0.97 (failsafe hardcoded 0.97)
Monitoring	Grafana CT141 (DS UID=PBFA97CFB590B2093), Prometheus, Alertmanager→Telegram
Ceph Mon Network	10.0.20.x (storage VLAN) — NOT 10.0.30.x (management)

All Ceph commands run from PVE master (10.0.20.10 — storage network) unless noted. Non-Ceph nodes (e.g. n5pro) need ceph.conf + admin keyring copied and rbd/ceph commands run from a monitor node.

SSH Access: 10.0.20.x Works, 10.0.30.x Does NOT

PVE hosts have two networks: storage (10.0.20.x) and management (10.0.30.x). SSH with ~/.ssh/id_ed25519_proxmox works on 10.0.20.x but is rejected on 10.0.30.x (key not in authorized_keys). Always use 10.0.20.x IPs for PVE host SSH:

# CORRECT — storage network
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 "ceph -s"

# WRONG — management network (key rejected)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.10 "ceph -s"  # Permission denied

PVE host IP mapping (storage network): px1=.10, px2=.20, px3=.30, px4=.40, px5=.50, px6=.60, px7=.70, n5pro=.91.

Fallback: Ceph CLI via K8s ceph-csi Pods

When PVE SSH is unavailable, ceph commands can be run from inside K8s ceph-csi-rbd-nodeplugin pods (the csi-rbdplugin container has /usr/bin/ceph). Requires a ceph.conf with monitor addresses and the csi-rbd-secret credentials:

# On mgmt-runner (10.0.30.124):
POD=$(kubectl get pods -n kube-system --no-headers | grep ceph-csi-rbd-nodeplugin | head -1 | awk '{print $1}')

# Get cephx key from K8s secret
USER_KEY=$(kubectl get secret csi-rbd-secret -n kube-system -o jsonpath='{.data.userKey}' | base64 -d)

# Write ceph.conf into the pod and query
kubectl exec -n kube-system $POD -c csi-rbdplugin -- bash -c "
cat > /tmp/ceph.conf << 'EOF'
[global]
  auth_cluster_required = cephx
  auth_service_required = cephx
  auth_client_required = cephx
  mon host = 10.0.20.50:6789,10.0.20.70:6789,10.0.20.40:6789
[client.kubernetes]
  key = $USER_KEY
EOF
ceph -c /tmp/ceph.conf --name client.kubernetes status
"

Monitor addresses are in the ceph-csi-config ConfigMap: kubectl get cm ceph-csi-config -n kube-system -o jsonpath='{.data.config\.json}'.

Pitfall: Not all monitors are always reachable. Check connectivity first (bash -c "echo > /dev/tcp/IP/6789" from a Galera VM or K8s node). Only use reachable monitors in the mon host list.

Dual-network pitfall: PVE hosts have SSH on 10.0.20.x (storage network, port 22 open) but NOT on 10.0.30.x (management network, SSH key rejected). Always use 10.0.20.10 for ssh root@10.0.20.10. The 10.0.30.x IPs are for VM bridges, not PVE host management.

Fallback when PVE SSH is unavailable: Run ceph commands from inside a ceph-csi-rbd-nodeplugin pod in K8s (see Monitoring section below for the full procedure).

⚠️ Pitfall: PVE hosts are on 10.0.20.x (storage net), NOT 10.0.30.x (management net). The 10.0.30.x addresses are for VM bridges, not PVE host SSH. Always use root@10.0.20.10 for Ceph commands. If SSH to 10.0.30.x PVE hosts fails with "Permission denied", you're on the wrong network — check this skill first, not the SSH key.

Fallback: Ceph CLI via K8s ceph-csi Pods

When PVE host SSH is unavailable (key rotation, host down, network change), ceph CLI is available inside the csi-rbdplugin container of ceph-csi-rbd-nodeplugin pods in K8s. Steps:

1. Find monitor addresses from the ceph-csi-config ConfigMap:

   kubectl get cm ceph-csi-config -n kube-system -o jsonpath='{.data.config\.json}'
   # Parse JSON for "monitors" array (e.g. ["10.0.20.10:6789","10.0.20.20:6789",...])
   

2. Get cephx credentials from the csi-rbd-secret:

   kubectl get secret csi-rbd-secret -n kube-system -o jsonpath='{.data.userKey}' | base64 -d
   

3. Write a temporary ceph.conf into the pod and query:

   POD=$(kubectl get pods -n kube-system --no-headers | grep ceph-csi-rbd-nodeplugin | head -1 | awk '{print $1}')
   # Create /tmp/ceph.conf with [global] mon host = <reachable monitors> and [client.kubernetes] key = <userKey>
   # Then: kubectl exec -n kube-system $POD -c csi-rbdplugin -- ceph -c /tmp/ceph.conf --name client.kubernetes status
   

⚠️ Not all monitors may be reachable from K8s nodes — test connectivity first. Only use monitors that respond on port 6789.

Decision Trees

OSD Creation

Is target a standard PVE node?
├─ YES → pveceph osd create /dev/sdX [--wal_dev /dev/nvme0n1pN]
│         ⚠️ pveceph CANNOT use LVM device-mapper paths (/dev/dm-*) as WAL/DB
│         If WAL/DB is on an LV: use ceph-volume directly (see below)
└─ NO (non-PVE node like n5pro)
   ├─ Has /etc/ceph/ceph.conf symlink to /etc/pve/ceph.conf?
   │  ├─ NO → rm /etc/ceph/ceph.conf && ln -s /etc/pve/ceph.conf /etc/ceph/ceph.conf
   │  └─ YES → proceed
   ├─ WAL/DB on raw partition? → pveceph osd create /dev/sdX --wal_dev /dev/nvme0n1pN
   └─ WAL/DB on LVM LV? → ceph-volume lvm prepare --bluestore --data /dev/sdX --block.db /dev/pve/wal-db-osd-X
                           (WAL auto-collocates on DB — do NOT specify --block.wal separately)

Recovery / Rebalance

ceph -s shows degraded/misplaced/backfill?
├─ Any OSD > full_ratio (0.97)?
│  └─ EMERGENCY: ceph osd set-full-ratio 0.99 → reweight overfull OSD → fix root cause
├─ backfill_toofull PGs > 0?
│  └─ Target OSDs above backfillfull_ratio → raise ratio (keep ordering!) or reweight
├─ Recovery speed too slow?
│  └─ Tune: osd_max_backfills, osd_recovery_op_priority, osd_recovery_sleep
│     ⚠️ ceph config set osd X does NOT propagate to running OSDs
│     Use: ceph tell osd.* injectargs "--osd_max_backfills 3" + verify with ceph daemon osd.N config get
├─ Recovery stuck at 0 MiB/s with remapped PGs?
│  └─ Check for sentinel 2147483647 in up-sets → EC with OSD count == k+m → reset all weights to 1.0
└─ upmap balancer not helping?
   └─ Balancer skips when >5% misplaced → wait for recovery, then balancer resumes

EC Pool with Mixed Disk Sizes

EC pool has OSDs of different capacities?
├─ Small OSDs filling disproportionately?
│  └─ ROOT CAUSE: EC stores equal-sized chunks regardless of OSD capacity
│     ceph osd reweight has marginal effect when k+m ≈ total OSD count
│  SOLUTION:
│  1. Define device classes: hdd-large, hdd-small
│  2. Create separate CRUSH root (hdd-root) for all HDDs
│  3. EC rule: take hdd-root~hdd-large (only large OSDs)
│  4. Replicated rule: take hdd-root (all HDDs, no class filter)
│  5. New EC profile with k adjusted to large-OSD count
│  6. Migrate RBD images to new pool (online via rbd migration)
└─ Artificially low CRUSH weights on small OSDs?
   └─ ANTI-PATTERN: causes vicious cycle → reset to true device size after separating classes

Core Operations

1. OSD Lifecycle

Create OSD with WAL/DB on NVMe (LVM-based)

# 1. Create WAL/DB LV on NVMe
lvcreate -L 30G -n wal-db-osd-X pve

# 2. Prepare OSD — ONLY --block.db, WAL auto-collocates
ceph-volume lvm prepare --bluestore \
  --data /dev/sdX \
  --block.db /dev/pve/wal-db-osd-X

# 3. Activate (use osd_id + fsid from prepare output)
ceph-volume lvm activate <osd_id> <osd_uuid>

# 4. Start service
systemctl enable --now ceph-osd@<osd_id>

# 5. Set CRUSH host + device class
ceph osd crush set osd.<id> <weight> root=default host=<hostname>
ceph osd crush set-device-class hdd osd.<id>

⚠️ NEVER specify --block.wal AND --block.db on the same LV — BlueStore opens the device exclusively → "Device or resource busy".

Destroy and Rebuild OSD

systemctl stop ceph-osd@<id>
ceph osd down <id>
ceph osd destroy <id> --yes-i-really-mean-it
ceph-volume lvm zap /dev/sdX --destroy
# Then recreate as above

Hot-Plug SATA Drive Detection

# JMB58x controller doesn't auto-scan
for h in /sys/class/scsi_host/host{0,1,2,3,4}; do
  echo "- - -" > ${h}/scan
done
sleep 2
lsblk -d -o NAME,MODEL,SERIAL,SIZE,TRAN,ROTA,STATE | grep -v rbd | grep -v loop

Expand BlueFS DB (Online LV Extend + Offline BlueStore Resize)

When BlueFS spillover occurs (RocksDB grows beyond DB LV onto HDD), expand the DB LV:

# 1. Extend the LV (online, no OSD stop needed for LVM)
lvextend -L +20G pve/ceph-db

# 2. Stop the OSD — bluefs-bdev-expand CANNOT run while OSD is active
#    Error if attempted online: "Device or resource busy" + assert crash
systemctl stop ceph-osd@<id>

# 3. Expand BlueFS awareness of the new DB size
ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-<id>
# Output: "1 : Expanding to 0x... (... GiB)" + "size updated"

# 4. Restart OSD
systemctl start ceph-osd@<id>

# 5. Verify BlueFS sees the new size
ceph daemon osd.<id> bluefs stats | head -3
# Should show: "1 : device size 0x...(XX GiB)"

# 6. Verify OSD is back up in cluster
ceph osd tree | grep "osd.<id>"

⚠️ bluefs-bdev-expand MUST be run with OSD stopped. Running it online crashes with FAILED ceph_assert(r == 0) — the block device is locked by the running OSD. The ceph daemon osd.N admin socket does NOT have an expand command (unlike config get/set). The only path is: stop OSD → bluestore-tool → start OSD.

⚠️ Spillover warning clears asynchronously. After expansion, BLUEFS_SPILLOVER health warning persists until BlueFS compacts RocksDB back onto the expanded DB device. This happens in the background over minutes to hours.

When NOT to Expand (Cosmetic Spillover)

Spillover < 1 MiB with >50% DB free → cosmetic, do NOT expand. Tiny spillovers (e.g. 128 KiB) are RocksDB metadata overflow that self-resolves on next compaction. Stopping an OSD for bluefs-bdev-expand triggers PG degradation and recovery I/O — worse than the warning itself.

Expand only when: spillover > 1 GiB OR DB device >80% full. This indicates sustained RocksDB growth that won't self-resolve.

2. CRUSH Topology

Device Class Separation

# Assign device classes
ceph osd crush set-device-class hdd-large osd.1 osd.6 osd.8 osd.11
ceph osd crush set-device-class hdd-small osd.7 osd.10

# Create separate root
ceph osd crush add-bucket hdd-root root

# Move HDD hosts to new root (⚠️ TRIGGERS REMAPPING!)
ceph osd crush move <hostname> root=hdd-root

CRUSH Rules

# EC rule — only large HDDs
ceph osd erasure-code-profile set ec31-hdd-large \
  k=3 m=1 crush-root=hdd-root crush-device-class=hdd-large \
  crush-failure-domain=osd plugin=jerasure technique=reed_sol_van
ceph osd crush rule create-erasure media_ec_large ec31-hdd-large

# Replicated rule — ALL HDDs (no class filter)
ceph osd crush rule create-replicated replicated_hdd_all hdd-root host

Weight Management

Command	Scope	Persistence	Use Case
ceph osd reweight N X	Runtime reweight	Lost on restart	Emergency drain of overfull OSD
ceph osd crush reweight osd.N X	CRUSH weight	Permanent	Match weight to device size

Correct CRUSH weights by device size:

Device	Weight
1 TB	~0.96
2 TB	~1.82
3 TB	~2.73
3.6 TiB	~3.64

⚠️ Anti-pattern: Lowering CRUSH weight to "protect" a full OSD. This prevents new PGs but doesn't move existing data → vicious cycle.

3. EC Pool Migration

EC pools cannot change k/m in-place. Must create new pool and migrate.

# 1. Create new EC pool
ceph osd pool create media_ec_new 128 128 erasure ec31-hdd-large
ceph osd pool set media_ec_new ec_overwrites
ceph osd pool application enable media_ec_new rbd

# 2. Create new metadata pool
ceph osd pool create media_meta_new 32

# 3. Online RBD migration (VMs keep running!)
for img in $(rbd ls media_meta); do
  rbd migration prepare media_meta/${img} media_meta_new/${img} --data-pool media_ec_new
  rbd migration execute media_meta_new/${img}
  rbd migration commit media_meta_new/${img}
done

# 4. Cleanup old pool
ceph osd pool delete media_ec --yes-i-really-really-mean-it
ceph osd pool rename media_ec_new media_ec

Timeline: prepare (instant) → execute (30-60 min per 500 GiB) → commit (atomic switch).

4. Recovery Tuning

Full Recovery Recipe

# 1. Unblock backfill (keep ratio ordering: nearfull < backfillfull < full ≤ 0.97)
ceph osd set-nearfull-ratio 0.94
ceph osd set-backfillfull-ratio 0.96
ceph osd set-full-ratio 0.97

# 2. Reweight overfull OSD (emergency, temporary)
ceph osd reweight <osd_id> 0.8

# 3. Set recovery params (persistent for future OSDs)
ceph config set osd osd_max_backfills 3
ceph config set osd osd_recovery_op_priority 10
ceph config set osd osd_recovery_sleep 0

# 4. Push to running OSDs (config set alone does NOT propagate!)
ceph tell osd.* injectargs "--osd_max_backfills 3"

# 5. Verify EACH OSD picked up the config
ceph daemon osd.<N> config get osd_max_backfills
# Must return "3" — if "1", config not applied

Post-Recovery Cleanup

# Restore defaults
ceph config set osd osd_max_backfills 1
ceph tell osd.* injectargs "--osd_max_backfills 1"
ceph osd reweight <osd_id> 1.0
# Ratios: keep at 0.94/0.96/0.97 (safe for mixed-size cluster)

5. RBD Orphan Cleanup

# 1. Get cluster-wide VM inventory (NOT per-node qm config!)
pvesh get /cluster/resources --type vm --output-format json | python3 -c "
import sys, json
for g in json.load(sys.stdin): print(g['vmid'])
" | sort -n | tr '\n' ' '

# 2. Fast orphan check for specific image (O(1))
grep -rl "vm-NNN" /etc/pve/qemu-server/ /etc/pve/lxc/ 2>/dev/null
# Exit 0 = referenced. Exit 1 = orphan.

# 3. Delete with protected snapshot
rbd snap unprotect "hdd_disk/base-NNN-disk-0"@__base__
rbd snap purge "hdd_disk/base-NNN-disk-0"
rbd rm "hdd_disk/base-NNN-disk-0"

⚠️ Always use pvesh get /cluster/resources --type vm for VM existence — per-node qm config only sees local VMs → false positives.

⚠️ Non-Ceph nodes cannot run rbd commands — run from a monitor node (e.g. proxmox7 = 10.0.20.70).

6. PG Management

# Reduce PG count (must disable autoscale first!)
ceph osd pool set <pool> pg_autoscale_mode off
ceph osd pool set <pool> pgp_num 32
ceph osd pool set <pool> pg_num 32
# Merge happens AFTER all PGs reach active+clean

# Raise mon_max_pg_per_osd (temporary relief)
ceph config set mon mon_max_pg_per_osd 400
ceph tell mon.* injectargs --mon_max_pg_per_osd=400

Pitfalls

Ratio Ordering (HEALTH_ERR)

nearfull < backfillfull < full ≤ osd_failsafe_full_ratio (0.97 hardcoded). Violation → HEALTH_ERR: OSD_OUT_OF_ORDER_FULL. Always set all three ratios together.

injectargs Reliability

ceph tell osd.* injectargs returns empty {} which looks like failure but may have succeeded. Always verify with ceph daemon osd.N config get <param>. Most reliable: ceph config set (persistent) + injectargs (runtime) + verify.

EC Reweight with Minimal OSDs

Never reweight OSDs in EC pools when OSD count == k+m. CRUSH places sentinel 2147483647 ("no OSD available") in up-sets → remapped PGs can never recover. Fix: reset all weights to 1.0. General rule: only reweight when OSD count > k+m.

pveceph LVM WAL/DB Limitation

pveceph osd create --wal_dev /dev/pve/wal-db-osd-X fails with "unable to get device info for '/dev/dm-2'". pveceph cannot resolve LVM device-mapper symlinks. Use ceph-volume lvm prepare directly with --block.db.

BlueStore WAL/DB Same Device

Specifying --block.wal AND --block.db on the same LV → "Device or resource busy". BlueStore opens WAL exclusively. Solution: specify ONLY --block.db — WAL collocates automatically.

ceph config set Doesn't Propagate to Running OSDs

ceph config set osd osd_max_backfills 3 updates MON config store but running OSDs keep old value. Must use ceph tell osd.* injectargs for runtime change, or restart OSDs. Always verify with ceph daemon osd.N config get.

backfillfull Is Derived, Not Settable

Pool-level backfillfull flag is derived from OSD conditions, not a settable pool flag. ceph osd pool unset <pool> backfillfull → EINVAL. Fix: raise backfillfull_ratio or reweight overfull OSDs.

Ghost RBD Images

Images appearing in rbd ls but rbd info returns empty → header gone, data objects remain. Need rados rm purge, not rbd rm.

pvesm list Fails When Pool Backfillfull

pvesm list hdd_disk returns empty when Ceph pool is in backfillfull state. Fall back to rbd ls <pool> for image listing.

Monitoring

# Quick health
ceph -s

# Per-OSD utilization
ceph osd df

# PG state summary
ceph pg dump pgs_brief | awk '{print $2}' | sort | uniq -c | sort -rn

# Recovery progress
ceph -s | grep -E "recovery:|misplaced"

# Balancer status
ceph balancer status

# Slow ops
ceph health detail | grep -A5 "slow"

Ceph Status via ceph-csi Pod (Fallback When PVE SSH Unavailable)

When PVE hosts are unreachable via SSH (key rotation, network issue, host down), but K8s nodes can reach the Ceph monitors, run ceph commands from inside a ceph-csi-rbd-nodeplugin pod:

# 1. Find the ceph-csi pod (on mgmt-runner or any K8s node with kubectl)
POD=$(kubectl get pods -n kube-system --no-headers | \
  grep ceph-csi-rbd-nodeplugin | head -1 | awk '{print $1}')

# 2. Get monitor addresses from the ceph-csi-config ConfigMap
MONS=$(kubectl get cm ceph-csi-config -n kube-system \
  -o jsonpath='{.data.config\.json}' | \
  python3 -c "import sys,json; print(','.join(json.load(sys.stdin)[0]['monitors']))")

# 3. Get the cephx key from the csi-rbd-secret
KEY=$(kubectl get secret csi-rbd-secret -n kube-system \
  -o jsonpath='{.data.userKey}' | base64 -d)

# 4. Write ceph.conf + keyring into the pod, then run ceph commands
kubectl exec -n kube-system $POD -c csi-rbdplugin -- bash -c "
cat > /tmp/ceph.conf << EOF
[global]
  auth_cluster_required = cephx
  auth_service_required = cephx
  auth_client_required = cephx
  mon host = $MONS
[client.kubernetes]
  key = $KEY
"
ceph -c /tmp/ceph.conf --name client.kubernetes status
ceph -c /tmp/ceph.conf --name client.kubernetes osd df
"

Key details:

• The csi-rbd-plugin container has /usr/bin/ceph installed
• Monitor addresses are in the ceph-csi-config ConfigMap (may be on a different subnet like 10.0.20.x — the storage VLAN)
• The cephx key is in the csi-rbd-secret Secret (field userKey)
• Not all monitors need to be up — 3 of 7 was sufficient in practice
• client.kubernetes has limited permissions (reads status, OSD df, pool df) but NOT admin operations (no ceph osd set-full-ratio etc.)

Pitfall: Monitor addresses in ceph-csi-config may be on a different subnet (e.g. 10.0.20.x storage VLAN) than the management network (10.0.30.x). K8s pods can usually reach both, but Galera VMs may only see 10.0.30.x.

Pitfall: The ceph -c /tmp/ceph.conf approach requires writing the config file into the pod. If kubectl cp is blocked, use kubectl exec -- bash -c "cat > /tmp/ceph.conf << EOF ..." via a heredoc through SSH to the mgmt-runner.

Recovery Monitoring Cron Pattern

Schedule: every 30 minutes
Report: PGs remapped/backfilling, % misplaced, OSD utilization, recovery speed
Alert: any OSD down or health ERR
Announce "COMPLETE" when 0 remapped PGs

Cross-References

Detailed session logs and edge cases in proxmox-ve-administration skill references:

• references/ceph-osd-create-hotplug-2026-07.md — Full OSD hot-plug walkthrough
• references/ceph-ec-mixed-size-optimization-2026-07.md — Complete EC migration plan
• references/ceph-recovery-acceleration-2026-07.md — Advanced recovery tuning
• references/ceph-crush-weight-optimization-2026-07.md — Weight alignment for mixed sizes
• references/ceph-pg-management-2026-07.md — PG count reduction + EC reweight pitfall
• references/ceph-pool-full-recovery-2026-07.md — Pool-full diagnosis + OSD preparation
• references/ceph-orphaned-rbd-cleanup-2026-07.md — Full orphan audit methodology
• references/ceph-recovery-ec-reweight-2026-07.md — EC reweight stagnation fix
• references/ceph-pg-osd-management-2026-07.md — PG/OSD rebalancing
• references/ceph-ec-pool-and-service-migration-2026-07.md — EC pool + service migration
• references/ceph-pg-reduction-and-osd-rebalance-2026-07.md — PG reduction + rebalance
• references/rbd-pool-migration-2026-07.md — Offline RBD pool migration
• references/hdd-selection-for-osd-2026-07.md — Drive selection criteria
• references/bluefs-db-expansion-2026-07.md — BlueFS DB LV expansion + cosmetic spillover assessment

Compound-learning solution docs in ~/docs/solutions/:

• bug-fixes/2026-07-13-pveceph-wal-lvm-device-mapper-limitation.md
• bug-fixes/2026-07-13-ceph-ratio-ordering-constraint.md
• architecture/2026-07-13-ec-mixed-disk-sizes-optimization.md
• tooling/2026-07-13-injectargs-vs-config-set-reliability.md