# Ceph Orphaned RBD Image Audit & Cleanup — 2026-07-04

## When to Use

Ceph pool is nearfull/backfill blocked, and you need to quickly free space without adding OSDs. Orphaned RBD images (disks belonging to deleted VMs/CTs) consume real pool capacity and can be safely removed.

⚠️ **User preference: Use PVE-native tools (`pvesh`, `pvesm`) for storage inventory, NOT raw `rbd` commands.** `rbd ls` is acceptable for listing images, but VM/CT existence must be checked cluster-wide via `pvesh`, not per-node `qm config`/`pct config` (which only see local VMs).

## Identification Workflow

### Step 1: Get cluster-wide VM/CT inventory (PRIMARY METHOD)

```bash
# From PVE coordinator — returns ALL VMs/CTs cluster-wide
pvesh get /cluster/resources --type vm --output-format json | python3 -c "
import sys, json
for g in json.load(sys.stdin):
    print(g['vmid'])
" | sort -n | tr '\n' ' '
```

This gives you the authoritative list of existing VMIDs across all nodes. Store this in a variable for cross-referencing.

### Step 2: List all RBD images on the pool

```bash
# List images (rbd ls is fine for listing — it's the VM existence check that must be PVE-native)
rbd ls hdd_disk
# With sizes:
rbd ls hdd_disk -l
```

### Step 3: Cross-reference — find orphans

For each RBD image, extract the VMID (`vm-NNN-disk-X` → NNN) and check against the cluster-wide VMID list from Step 1. If the VMID doesn't exist in the `pvesh` output, the image is orphaned.

```bash
# Get existing VMIDs
EXISTING=$(pvesh get /cluster/resources --type vm --output-format json 2>/dev/null | \
  python3 -c "import sys,json; [print(g['vmid']) for g in json.load(sys.stdin)]" | \
  sort -n | tr '\n' ' ')

# Find orphans
for img in $(rbd ls hdd_disk 2>/dev/null); do
  vmid=$(echo "$img" | sed -n 's/^vm-\([0-9]*\)-.*/\1/p; s/^base-\([0-9]*\)-.*/\1/p')
  if [ -n "$vmid" ]; then
    if ! echo " $EXISTING " | grep -q " $vmid "; then
      size=$(rbd info "hdd_disk/$img" 2>/dev/null | grep "size" | awk '{print $2, $3}')
      echo "ORPHAN: hdd_disk:$img ($size) — VMID $vmid does not exist"
    fi
  else
    # Non-VM image (csi-vol, test-dummy, etc.)
    size=$(rbd info "hdd_disk/$img" 2>/dev/null | grep "size" | awk '{print $2, $3}')
    echo "NON-VM: hdd_disk:$img ($size)"
  fi
done
```

### ⚠️ Why NOT to use per-node `qm config`/`pct config`

`qm config NNN` and `pct config NNN` only check the LOCAL node. A VM running on proxmox5 won't be found by `qm config` executed on proxmox1. This leads to **false positives** — images incorrectly classified as orphaned because the VM exists on a different node.

**Always use `pvesh get /cluster/resources --type vm` for cluster-wide inventory.**

### Step 4: Verify no dependencies before deleting

For each orphaned image, check:

```bash
# Active watchers (attached clients)
rbd status -p hdd_disk <image>
# Watchers: none  ← safe to delete

# Clone children (if base/parent image)
rbd children -p hdd_disk <base-image>
# (empty output) ← no clones, safe to delete

# For CSI volumes (csi-vol-*) — check watchers only
rbd status -p hdd_disk csi-vol-<uuid>
```

### Step 5: Delete orphaned images

```bash
rbd rm -p hdd_disk <image>
# Or batch:
for img in vm-104-disk-0 vm-104-disk-1 vm-108-disk-0 ...; do
  rbd rm -p hdd_disk $img
done
```

## Classification Rules

| Image Pattern | Check | Safe to Delete? |
|---------------|-------|-----------------|
| `vm-NNN-disk-X` | `qm config NNN` / `pct config NNN` | ✅ if VMID doesn't exist |
| `vm-NNN-cloudinit` | Same VMID check | ✅ if VMID doesn't exist |
| `base-NNN-disk-X` | `rbd children` | ✅ if no clones |
| `csi-vol-<uuid>` | `rbd status` (watchers) | ✅ if no watchers |
| `test-dummy` | Name pattern | ✅ always (test artifact) |

## Case Study: 2026-07-04 Audit

### Pool: hdd_disk (92% full, backfill blocked)

Found **40 orphaned RBD images** totaling ~2.1 TB thin-provisioned across 24 non-existent VMIDs. Largest orphans:

| Image | Size | VMID Exists? |
|-------|------|-------------|
| vm-202-disk-2 | 300 GB | ❌ |
| vm-203-disk-1 | 300 GB | ❌ |
| vm-116-disk-1 | 192 GB | ❌ |
| vm-204-disk-0 | 150 GB | ❌ |
| csi-vol-b78002cf | 100 GB | ❌ (no watchers) |
| vm-128/131/132-disk-0 | 80 GB each | ❌ |
| vm-202-disk-1, vm-203-disk-0, vm-126-disk-0 | 64 GB each | ❌ |
| ... + 27 more | | |

### Images NOT deleted (VM/CT still exists)

| Image | Size | Owner | Status |
|-------|------|-------|--------|
| vm-107-disk-0 | 82 GB | VM 107 (openclaw) | stopped |
| vm-111-disk-0/3 | 350 GB each | CT 111 (docker) | stopped, backup lock |
| vm-123-disk-0 | 40 GB | CT 123 (ollama) | stopped |
| vm-136-disk-0 | 8 GB | CT 136 (smb-media) | running |

### Key Insight

Even though images are thin-provisioned (actual usage < provisioned size), deleting orphans frees real pool capacity because Ceph tracks allocated objects per image. On a pool at 92% with `backfill_toofull` blocking recovery, removing 40 orphaned images can unblock the backfill without adding new OSDs.

## VM Destruction on Non-Ceph Nodes (stale locks)

When `qm destroy NNN` fails with `got timeout` on CFS storage locks (common when Ceph is under load), the VM config can be removed manually:

### Step 1: Delete VM config and locks on the node hosting the VM

```bash
# On the node where the VM lives (e.g., n5pro = 10.0.20.91)
rm -f /etc/pve/qemu-server/NNN.conf
rm -f /var/lock/qemu-server/lock-NNN.conf
```

⚠️ **Non-Ceph nodes (no `ceph.conf`) cannot run `rbd` commands.** n5pro has no Ceph monitor — `rbd remove` fails with "can't open ceph.conf" / "couldn't connect to the cluster". RBD image removal must be run from a Ceph monitor node (e.g., proxmox7 = 10.0.20.70).

### Step 2: Remove RBD images from a Ceph monitor node

```bash
# On proxmox7 (10.0.20.70) — the Ceph monitor node
for img in vm-NNN-disk-0 vm-NNN-disk-1 vm-NNN-cloudinit; do
  rbd remove -p hdd_disk $img 2>&1
done
```

Some images may already be gone ("No such file or directory") if a partial `qm destroy` succeeded before timing out.

### Step 3: Handle "image still has watchers" errors

If `rbd remove` fails with "image still has watchers", a previous `rbd map` or crashed client holds the image open:

```bash
# Find which node has the stale mapping
rbd showmapped                    # on local node
# Check ALL proxmox nodes:
for node in proxmox4 proxmox5 proxmox6 proxmox7; do
  ssh root@$node "rbd showmapped | grep NNN"
done

# Force unmap on the offending node
ssh root@<node> "rbd unmap -o force /dev/rbdN"

# Then retry removal from monitor node
rbd remove -p <pool> vm-NNN-disk-X
```

⚠️ `rbd unmap` may fail with "Device or resource busy" if system processes hold it. Use `-o force` flag. After force unmap, the device disappears from `/dev/rbdN` and the watcher expires within 30s.

## Identifying Orphaned VM Contents (RBD Inspection)

When a VM config is already deleted and you need to know what the VM was (for risk assessment before deleting disks):

```bash
# Map the RBD image to a block device
rbd map -p hdd_disk vm-NNN-disk-X
# → /dev/rbd10

# Check if it has a partition table
fdisk -l /dev/rbd10
blkid /dev/rbd10

# If ext4 directly (no partition table):
mount /dev/rbd10 /mnt/inspect
ls /mnt/inspect/

# If GPT partition table:
mount /dev/rbd10p1 /mnt/inspect

# Identify the OS / purpose
cat /mnt/inspect/etc/hostname
cat /mnt/inspect/etc/os-release | head -3
ls /mnt/inspect/  # look for model files, docker volumes, etc.

# ALWAYS clean up
umount /mnt/inspect
rbd unmap /dev/rbd10
```

⚠️ Mounting RBD images on a cluster under heavy backfill load can timeout (30s+). If mount hangs, kill it and try later — or infer contents from disk size and structure (e.g., 300 GB ext4 with a single `.gguf` file = AI model storage).

⚠️ **Always `rbd unmap` after inspection.** Leftover mappings create "image still has watchers" errors when you later try to delete the image.

### Case Study: VM 202/203/116 Destruction (2026-07-04)

VMs 201, 202, 203 existed on n5pro (10.0.20.91, non-Ceph node) and VM 116 on proxmox6. All were stopped, `qm destroy` failed with CFS storage lock timeouts because Ceph was under heavy backfill load.

**Resolution:**
1. Deleted VM configs manually on n5pro: `rm -f /etc/pve/qemu-server/{201,202,203}.conf` + stale locks
2. Removed RBD images from proxmox7 (Ceph monitor node): `rbd remove -p hdd_disk vm-{201,202,203}-disk-*`
3. VM 116 had a stale RBD mapping on proxmox6 (`/dev/rbd4` → `vm-116-disk-0` on `vm_disks` pool). Force unmapped: `rbd unmap -o force /dev/rbd4`, then removed both disk-0 (vm_disks) and disk-1 (hdd_disk, already gone).

**Contents identified via RBD inspection:**
- VM 202: AI inference worker (Qwen3.6-35B-A3B-UD-IQ4_NL.gguf on 300 GB disk)
- VM 203: Same structure (64 GB OS + 300 GB data), likely AI worker
- VM 116: 192 GB ext4 data disk, no partition table (likely MariaDB-01)
- Total freed: ~920 GB thin-provisioned

## Pitfalls

1. **`rbd du` timeouts** — On pools with many images, `rbd du` can hang. Use per-image `rbd info` instead, but batch to avoid SSH timeouts.
2. **CSI volumes look orphaned but may be Kubernetes/external provisions** — Always check `rbd status` for watchers before deleting `csi-vol-*` images.
3. **Base images with clones** — `rbd children` must return empty before deleting. Deleting a parent with active clones corrupts the clone.
4. **Stopped VMs still own their disks** — A stopped VM (like VM 107 openclaw) still has a valid config referencing the disk. Only delete if the VMID has NO config at all (`qm config` AND `pct config` both fail).
5. **Backup-locked CTs** — CT 111 had a `backup` lock. Don't delete disks of locked CTs even if stopped.
6. **Timeout management** — Checking 50+ VMIDs via SSH can exceed 15s. Increase timeout to 30-60s or batch in groups of 10-15.
7. **Non-Ceph nodes can't run `rbd` commands** — Nodes without `ceph.conf` (like n5pro) cannot connect to the Ceph cluster. Run all `rbd` operations from a monitor node.
8. **`qm destroy` CFS lock timeout under Ceph load** — When Ceph is slow (backfilling, nearfull), Proxmox's CFS storage lock acquisition times out. Delete configs manually with `rm -f` and remove RBD images separately from a monitor node.
9. **Stale RBD mappings block image deletion** — A previous `rbd map` that wasn't cleaned up (or a crashed QEMU) keeps watchers on the image. Check `rbd showmapped` on ALL nodes, force unmap, then retry removal.
10. **Always unmap RBD images after inspection** — Mounted RBD images create watchers that prevent later deletion. `umount` + `rbd unmap` after every inspection session.
11. **Mount can hang under heavy backfill** — When Ceph recovery I/O is high, `mount /dev/rbdN` can hang for 30s+. Abort and infer from disk structure if needed.