hermes-skills/devops/proxmox-ve-administration/references/ceph-orphaned-rbd-cleanup-2026-07.md

Ceph Orphaned RBD Image Audit & Cleanup — 2026-07-04

When to Use

Ceph pool is nearfull/backfill blocked, and you need to quickly free space without adding OSDs. Orphaned RBD images (disks belonging to deleted VMs/CTs) consume real pool capacity and can be safely removed.

⚠️ User preference: Use PVE-native tools (pvesh, pvesm) for storage inventory, NOT raw rbd commands. rbd ls is acceptable for listing images, but VM/CT existence must be checked cluster-wide via pvesh, not per-node qm config/pct config (which only see local VMs).

Identification Workflow

Step 1: Get cluster-wide VM/CT inventory (PRIMARY METHOD)

# From PVE coordinator — returns ALL VMs/CTs cluster-wide
pvesh get /cluster/resources --type vm --output-format json | python3 -c "
import sys, json
for g in json.load(sys.stdin):
    print(g['vmid'])
" | sort -n | tr '\n' ' '

This gives you the authoritative list of existing VMIDs across all nodes. Store this in a variable for cross-referencing.

Step 2: List all RBD images on the pool

# List images (rbd ls is fine for listing — it's the VM existence check that must be PVE-native)
rbd ls hdd_disk
# With sizes:
rbd ls hdd_disk -l

Step 3: Cross-reference — find orphans

For each RBD image, extract the VMID (vm-NNN-disk-X → NNN) and check against the cluster-wide VMID list from Step 1. If the VMID doesn't exist in the pvesh output, the image is orphaned.

# Get existing VMIDs
EXISTING=$(pvesh get /cluster/resources --type vm --output-format json 2>/dev/null | \
  python3 -c "import sys,json; [print(g['vmid']) for g in json.load(sys.stdin)]" | \
  sort -n | tr '\n' ' ')

# Find orphans
for img in $(rbd ls hdd_disk 2>/dev/null); do
  vmid=$(echo "$img" | sed -n 's/^vm-\([0-9]*\)-.*/\1/p; s/^base-\([0-9]*\)-.*/\1/p')
  if [ -n "$vmid" ]; then
    if ! echo " $EXISTING " | grep -q " $vmid "; then
      size=$(rbd info "hdd_disk/$img" 2>/dev/null | grep "size" | awk '{print $2, $3}')
      echo "ORPHAN: hdd_disk:$img ($size) — VMID $vmid does not exist"
    fi
  else
    # Non-VM image (csi-vol, test-dummy, etc.)
    size=$(rbd info "hdd_disk/$img" 2>/dev/null | grep "size" | awk '{print $2, $3}')
    echo "NON-VM: hdd_disk:$img ($size)"
  fi
done

⚠️ Why NOT to use per-node qm config/pct config

qm config NNN and pct config NNN only check the LOCAL node. A VM running on proxmox5 won't be found by qm config executed on proxmox1. This leads to false positives — images incorrectly classified as orphaned because the VM exists on a different node.

Always use pvesh get /cluster/resources --type vm for cluster-wide inventory.

Step 4: Verify no dependencies before deleting

For each orphaned image, check:

# Active watchers (attached clients)
rbd status -p hdd_disk <image>
# Watchers: none  ← safe to delete

# Clone children (if base/parent image)
rbd children -p hdd_disk <base-image>
# (empty output) ← no clones, safe to delete

# For CSI volumes (csi-vol-*) — check watchers only
rbd status -p hdd_disk csi-vol-<uuid>

Step 5: Delete orphaned images

rbd rm -p hdd_disk <image>
# Or batch:
for img in vm-104-disk-0 vm-104-disk-1 vm-108-disk-0 ...; do
  rbd rm -p hdd_disk $img
done

Classification Rules

Image Pattern	Check	Safe to Delete?
vm-NNN-disk-X	qm config NNN / pct config NNN	✅ if VMID doesn't exist
vm-NNN-cloudinit	Same VMID check	✅ if VMID doesn't exist
base-NNN-disk-X	rbd children	✅ if no clones
csi-vol-<uuid>	rbd status (watchers)	✅ if no watchers
test-dummy	Name pattern	✅ always (test artifact)

Case Study: 2026-07-04 Audit

Pool: hdd_disk (92% full, backfill blocked)

Found 40 orphaned RBD images totaling ~2.1 TB thin-provisioned across 24 non-existent VMIDs. Largest orphans:

Image	Size	VMID Exists?
vm-202-disk-2	300 GB	❌
vm-203-disk-1	300 GB	❌
vm-116-disk-1	192 GB	❌
vm-204-disk-0	150 GB	❌
csi-vol-b78002cf	100 GB	❌ (no watchers)
vm-128/131/132-disk-0	80 GB each	❌
vm-202-disk-1, vm-203-disk-0, vm-126-disk-0	64 GB each	❌
... + 27 more

Images NOT deleted (VM/CT still exists)

Image	Size	Owner	Status
vm-107-disk-0	82 GB	VM 107 (openclaw)	stopped
vm-111-disk-0/3	350 GB each	CT 111 (docker)	stopped, backup lock
vm-123-disk-0	40 GB	CT 123 (ollama)	stopped
vm-136-disk-0	8 GB	CT 136 (smb-media)	running

Key Insight

Even though images are thin-provisioned (actual usage < provisioned size), deleting orphans frees real pool capacity because Ceph tracks allocated objects per image. On a pool at 92% with backfill_toofull blocking recovery, removing 40 orphaned images can unblock the backfill without adding new OSDs.

VM Destruction on Non-Ceph Nodes (stale locks)

When qm destroy NNN fails with got timeout on CFS storage locks (common when Ceph is under load), the VM config can be removed manually:

Step 1: Delete VM config and locks on the node hosting the VM

# On the node where the VM lives (e.g., n5pro = 10.0.20.91)
rm -f /etc/pve/qemu-server/NNN.conf
rm -f /var/lock/qemu-server/lock-NNN.conf

⚠️ Non-Ceph nodes (no ceph.conf) cannot run rbd commands. n5pro has no Ceph monitor — rbd remove fails with "can't open ceph.conf" / "couldn't connect to the cluster". RBD image removal must be run from a Ceph monitor node (e.g., proxmox7 = 10.0.20.70).

Step 2: Remove RBD images from a Ceph monitor node

# On proxmox7 (10.0.20.70) — the Ceph monitor node
for img in vm-NNN-disk-0 vm-NNN-disk-1 vm-NNN-cloudinit; do
  rbd remove -p hdd_disk $img 2>&1
done

Some images may already be gone ("No such file or directory") if a partial qm destroy succeeded before timing out.

Step 3: Handle "image still has watchers" errors

If rbd remove fails with "image still has watchers", a previous rbd map or crashed client holds the image open:

# Find which node has the stale mapping
rbd showmapped                    # on local node
# Check ALL proxmox nodes:
for node in proxmox4 proxmox5 proxmox6 proxmox7; do
  ssh root@$node "rbd showmapped | grep NNN"
done

# Force unmap on the offending node
ssh root@<node> "rbd unmap -o force /dev/rbdN"

# Then retry removal from monitor node
rbd remove -p <pool> vm-NNN-disk-X

⚠️ rbd unmap may fail with "Device or resource busy" if system processes hold it. Use -o force flag. After force unmap, the device disappears from /dev/rbdN and the watcher expires within 30s.

Identifying Orphaned VM Contents (RBD Inspection)

When a VM config is already deleted and you need to know what the VM was (for risk assessment before deleting disks):

# Map the RBD image to a block device
rbd map -p hdd_disk vm-NNN-disk-X
# → /dev/rbd10

# Check if it has a partition table
fdisk -l /dev/rbd10
blkid /dev/rbd10

# If ext4 directly (no partition table):
mount /dev/rbd10 /mnt/inspect
ls /mnt/inspect/

# If GPT partition table:
mount /dev/rbd10p1 /mnt/inspect

# Identify the OS / purpose
cat /mnt/inspect/etc/hostname
cat /mnt/inspect/etc/os-release | head -3
ls /mnt/inspect/  # look for model files, docker volumes, etc.

# ALWAYS clean up
umount /mnt/inspect
rbd unmap /dev/rbd10

⚠️ Mounting RBD images on a cluster under heavy backfill load can timeout (30s+). If mount hangs, kill it and try later — or infer contents from disk size and structure (e.g., 300 GB ext4 with a single .gguf file = AI model storage).

⚠️ Always rbd unmap after inspection. Leftover mappings create "image still has watchers" errors when you later try to delete the image.

Case Study: VM 202/203/116 Destruction (2026-07-04)

VMs 201, 202, 203 existed on n5pro (10.0.20.91, non-Ceph node) and VM 116 on proxmox6. All were stopped, qm destroy failed with CFS storage lock timeouts because Ceph was under heavy backfill load.

Resolution:

1. Deleted VM configs manually on n5pro: rm -f /etc/pve/qemu-server/{201,202,203}.conf + stale locks
2. Removed RBD images from proxmox7 (Ceph monitor node): rbd remove -p hdd_disk vm-{201,202,203}-disk-*
3. VM 116 had a stale RBD mapping on proxmox6 (/dev/rbd4 → vm-116-disk-0 on vm_disks pool). Force unmapped: rbd unmap -o force /dev/rbd4, then removed both disk-0 (vm_disks) and disk-1 (hdd_disk, already gone).

Contents identified via RBD inspection:

• VM 202: AI inference worker (Qwen3.6-35B-A3B-UD-IQ4_NL.gguf on 300 GB disk)
• VM 203: Same structure (64 GB OS + 300 GB data), likely AI worker
• VM 116: 192 GB ext4 data disk, no partition table (likely MariaDB-01)
• Total freed: ~920 GB thin-provisioned

Pitfalls

1. rbd du timeouts — On pools with many images, rbd du can hang. Use per-image rbd info instead, but batch to avoid SSH timeouts.
2. CSI volumes look orphaned but may be Kubernetes/external provisions — Always check rbd status for watchers before deleting csi-vol-* images.
3. Base images with clones — rbd children must return empty before deleting. Deleting a parent with active clones corrupts the clone.
4. Stopped VMs still own their disks — A stopped VM (like VM 107 openclaw) still has a valid config referencing the disk. Only delete if the VMID has NO config at all (qm config AND pct config both fail).
5. Backup-locked CTs — CT 111 had a backup lock. Don't delete disks of locked CTs even if stopped.
6. Timeout management — Checking 50+ VMIDs via SSH can exceed 15s. Increase timeout to 30-60s or batch in groups of 10-15.
7. Non-Ceph nodes can't run rbd commands — Nodes without ceph.conf (like n5pro) cannot connect to the Ceph cluster. Run all rbd operations from a monitor node.
8. qm destroy CFS lock timeout under Ceph load — When Ceph is slow (backfilling, nearfull), Proxmox's CFS storage lock acquisition times out. Delete configs manually with rm -f and remove RBD images separately from a monitor node.
9. Stale RBD mappings block image deletion — A previous rbd map that wasn't cleaned up (or a crashed QEMU) keeps watchers on the image. Check rbd showmapped on ALL nodes, force unmap, then retry removal.
10. Always unmap RBD images after inspection — Mounted RBD images create watchers that prevent later deletion. umount + rbd unmap after every inspection session.
11. Mount can hang under heavy backfill — When Ceph recovery I/O is high, mount /dev/rbdN can hang for 30s+. Abort and infer from disk structure if needed.