---
pattern_id: PAT-007
title: "Ceph EC pool unusable + SSD wear-level failing — monitor pg states"
category: storage
severity: high
status: active
first_observed: 2026-07
last_updated: 2026-08-30
related_systems: [ceph-cluster]
related_solution_docs:
  - docs/solutions/architecture/2026-07-12-ceph-ec-pool-no-rebalance-headroom.md
  - docs/solutions/bug-fixes/2026-07-13-ceph-ratio-ordering-constraint.md
  - docs/solutions/bug-fixes/2026-07-13-pveceph-wal-lvm-device-mapper-limitation.md
related_skills: [ceph-cluster-administration, ceph-ec-incomplete-pg-recovery]
---

# Ceph EC pool unusable + SSD wear-level failing — monitor pg states

## Symptom

Multiple Ceph failure modes manifest simultaneously:
1. EC pool (pool 8/media_ec) becomes unusable — PGs stuck in `incomplete` state
2. SSD OSD (osd.3 on px4) reports `Wear Leveling Count` SMART attribute indicating
   imminent failure
3. `ceph health` shows `HEALTH_ERR`

## Root Cause

These are compounded issues:
1. **EC pool incomplete PGs**: After OSD loss, EC pools cannot recover without enough
   surviving shards. The minimum copies requirement for EC k+m encoding is stricter than
   replicated pools.
2. **SSD wear-level failure**: Consumer-grade SSDs in Ceph OSD duty cycle reach wear limits.
   Once `Wear Leveling Count` exceeds threshold, the SSD becomes read-only or unreliable.
3. **Weight imbalance**: Unequal OSD weights cause CRUSH placement failures, leaving PGs
   perpetually remapped.

## Mitigation

1. **EC pool recovery**: Use the `ceph-ec-incomplete-pg-recovery` skill — specialized
   procedure for forcing EC PG recovery after OSD loss
2. **Replace failing SSD**: Mark OSD `out` → `destroy` → `zap` → physically replace → recreate
3. **Fix weight imbalance**: Equalize `pg_num` → `pgp_num`, set `nopgchange=true`,
   reweight OSDs proportionally to disk capacity
4. **Remove empty CRUSH buckets**: Clean up stale host entries from decommissioned nodes

## Prevention

- Monitor SMART attributes monthly — alert on wear-level > 80%
- Never use consumer SSDs for Ceph OSDs in production (use enterprise grade with PLP)
- Keep EC pool k+m ratio proportional to available OSDs (don't over-provision redundancy)
- Regular `ceph pg dump` audits for stuck/unactive PGs
- Separate PBS onto its own tier (CephFS), away from RBD pools

## Evidence

- Multiple incidents Jul 2026 (OSD 0+2 destruction, OSD 3 wear-level, EC pool unusable)
- Solution docs: `2026-07-12-ceph-ec-pool-no-rebalance-headroom.md`,
  `2026-07-13-ceph-ratio-ordering-constraint.md`
- MEMORY.md entry: "Ceph:HEALTH_ERR. Pool8/media_ec unusable. px4 SSD(osd.3) WearLevel FAILING."