Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
@@ -0,0 +1,239 @@
|
||||
# Ceph Orphaned RBD Image Audit & Cleanup — 2026-07-04
|
||||
|
||||
## When to Use
|
||||
|
||||
Ceph pool is nearfull/backfill blocked, and you need to quickly free space without adding OSDs. Orphaned RBD images (disks belonging to deleted VMs/CTs) consume real pool capacity and can be safely removed.
|
||||
|
||||
⚠️ **User preference: Use PVE-native tools (`pvesh`, `pvesm`) for storage inventory, NOT raw `rbd` commands.** `rbd ls` is acceptable for listing images, but VM/CT existence must be checked cluster-wide via `pvesh`, not per-node `qm config`/`pct config` (which only see local VMs).
|
||||
|
||||
## Identification Workflow
|
||||
|
||||
### Step 1: Get cluster-wide VM/CT inventory (PRIMARY METHOD)
|
||||
|
||||
```bash
|
||||
# From PVE coordinator — returns ALL VMs/CTs cluster-wide
|
||||
pvesh get /cluster/resources --type vm --output-format json | python3 -c "
|
||||
import sys, json
|
||||
for g in json.load(sys.stdin):
|
||||
print(g['vmid'])
|
||||
" | sort -n | tr '\n' ' '
|
||||
```
|
||||
|
||||
This gives you the authoritative list of existing VMIDs across all nodes. Store this in a variable for cross-referencing.
|
||||
|
||||
### Step 2: List all RBD images on the pool
|
||||
|
||||
```bash
|
||||
# List images (rbd ls is fine for listing — it's the VM existence check that must be PVE-native)
|
||||
rbd ls hdd_disk
|
||||
# With sizes:
|
||||
rbd ls hdd_disk -l
|
||||
```
|
||||
|
||||
### Step 3: Cross-reference — find orphans
|
||||
|
||||
For each RBD image, extract the VMID (`vm-NNN-disk-X` → NNN) and check against the cluster-wide VMID list from Step 1. If the VMID doesn't exist in the `pvesh` output, the image is orphaned.
|
||||
|
||||
```bash
|
||||
# Get existing VMIDs
|
||||
EXISTING=$(pvesh get /cluster/resources --type vm --output-format json 2>/dev/null | \
|
||||
python3 -c "import sys,json; [print(g['vmid']) for g in json.load(sys.stdin)]" | \
|
||||
sort -n | tr '\n' ' ')
|
||||
|
||||
# Find orphans
|
||||
for img in $(rbd ls hdd_disk 2>/dev/null); do
|
||||
vmid=$(echo "$img" | sed -n 's/^vm-\([0-9]*\)-.*/\1/p; s/^base-\([0-9]*\)-.*/\1/p')
|
||||
if [ -n "$vmid" ]; then
|
||||
if ! echo " $EXISTING " | grep -q " $vmid "; then
|
||||
size=$(rbd info "hdd_disk/$img" 2>/dev/null | grep "size" | awk '{print $2, $3}')
|
||||
echo "ORPHAN: hdd_disk:$img ($size) — VMID $vmid does not exist"
|
||||
fi
|
||||
else
|
||||
# Non-VM image (csi-vol, test-dummy, etc.)
|
||||
size=$(rbd info "hdd_disk/$img" 2>/dev/null | grep "size" | awk '{print $2, $3}')
|
||||
echo "NON-VM: hdd_disk:$img ($size)"
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
### ⚠️ Why NOT to use per-node `qm config`/`pct config`
|
||||
|
||||
`qm config NNN` and `pct config NNN` only check the LOCAL node. A VM running on proxmox5 won't be found by `qm config` executed on proxmox1. This leads to **false positives** — images incorrectly classified as orphaned because the VM exists on a different node.
|
||||
|
||||
**Always use `pvesh get /cluster/resources --type vm` for cluster-wide inventory.**
|
||||
|
||||
### Step 4: Verify no dependencies before deleting
|
||||
|
||||
For each orphaned image, check:
|
||||
|
||||
```bash
|
||||
# Active watchers (attached clients)
|
||||
rbd status -p hdd_disk <image>
|
||||
# Watchers: none ← safe to delete
|
||||
|
||||
# Clone children (if base/parent image)
|
||||
rbd children -p hdd_disk <base-image>
|
||||
# (empty output) ← no clones, safe to delete
|
||||
|
||||
# For CSI volumes (csi-vol-*) — check watchers only
|
||||
rbd status -p hdd_disk csi-vol-<uuid>
|
||||
```
|
||||
|
||||
### Step 5: Delete orphaned images
|
||||
|
||||
```bash
|
||||
rbd rm -p hdd_disk <image>
|
||||
# Or batch:
|
||||
for img in vm-104-disk-0 vm-104-disk-1 vm-108-disk-0 ...; do
|
||||
rbd rm -p hdd_disk $img
|
||||
done
|
||||
```
|
||||
|
||||
## Classification Rules
|
||||
|
||||
| Image Pattern | Check | Safe to Delete? |
|
||||
|---------------|-------|-----------------|
|
||||
| `vm-NNN-disk-X` | `qm config NNN` / `pct config NNN` | ✅ if VMID doesn't exist |
|
||||
| `vm-NNN-cloudinit` | Same VMID check | ✅ if VMID doesn't exist |
|
||||
| `base-NNN-disk-X` | `rbd children` | ✅ if no clones |
|
||||
| `csi-vol-<uuid>` | `rbd status` (watchers) | ✅ if no watchers |
|
||||
| `test-dummy` | Name pattern | ✅ always (test artifact) |
|
||||
|
||||
## Case Study: 2026-07-04 Audit
|
||||
|
||||
### Pool: hdd_disk (92% full, backfill blocked)
|
||||
|
||||
Found **40 orphaned RBD images** totaling ~2.1 TB thin-provisioned across 24 non-existent VMIDs. Largest orphans:
|
||||
|
||||
| Image | Size | VMID Exists? |
|
||||
|-------|------|-------------|
|
||||
| vm-202-disk-2 | 300 GB | ❌ |
|
||||
| vm-203-disk-1 | 300 GB | ❌ |
|
||||
| vm-116-disk-1 | 192 GB | ❌ |
|
||||
| vm-204-disk-0 | 150 GB | ❌ |
|
||||
| csi-vol-b78002cf | 100 GB | ❌ (no watchers) |
|
||||
| vm-128/131/132-disk-0 | 80 GB each | ❌ |
|
||||
| vm-202-disk-1, vm-203-disk-0, vm-126-disk-0 | 64 GB each | ❌ |
|
||||
| ... + 27 more | | |
|
||||
|
||||
### Images NOT deleted (VM/CT still exists)
|
||||
|
||||
| Image | Size | Owner | Status |
|
||||
|-------|------|-------|--------|
|
||||
| vm-107-disk-0 | 82 GB | VM 107 (openclaw) | stopped |
|
||||
| vm-111-disk-0/3 | 350 GB each | CT 111 (docker) | stopped, backup lock |
|
||||
| vm-123-disk-0 | 40 GB | CT 123 (ollama) | stopped |
|
||||
| vm-136-disk-0 | 8 GB | CT 136 (smb-media) | running |
|
||||
|
||||
### Key Insight
|
||||
|
||||
Even though images are thin-provisioned (actual usage < provisioned size), deleting orphans frees real pool capacity because Ceph tracks allocated objects per image. On a pool at 92% with `backfill_toofull` blocking recovery, removing 40 orphaned images can unblock the backfill without adding new OSDs.
|
||||
|
||||
## VM Destruction on Non-Ceph Nodes (stale locks)
|
||||
|
||||
When `qm destroy NNN` fails with `got timeout` on CFS storage locks (common when Ceph is under load), the VM config can be removed manually:
|
||||
|
||||
### Step 1: Delete VM config and locks on the node hosting the VM
|
||||
|
||||
```bash
|
||||
# On the node where the VM lives (e.g., n5pro = 10.0.20.91)
|
||||
rm -f /etc/pve/qemu-server/NNN.conf
|
||||
rm -f /var/lock/qemu-server/lock-NNN.conf
|
||||
```
|
||||
|
||||
⚠️ **Non-Ceph nodes (no `ceph.conf`) cannot run `rbd` commands.** n5pro has no Ceph monitor — `rbd remove` fails with "can't open ceph.conf" / "couldn't connect to the cluster". RBD image removal must be run from a Ceph monitor node (e.g., proxmox7 = 10.0.20.70).
|
||||
|
||||
### Step 2: Remove RBD images from a Ceph monitor node
|
||||
|
||||
```bash
|
||||
# On proxmox7 (10.0.20.70) — the Ceph monitor node
|
||||
for img in vm-NNN-disk-0 vm-NNN-disk-1 vm-NNN-cloudinit; do
|
||||
rbd remove -p hdd_disk $img 2>&1
|
||||
done
|
||||
```
|
||||
|
||||
Some images may already be gone ("No such file or directory") if a partial `qm destroy` succeeded before timing out.
|
||||
|
||||
### Step 3: Handle "image still has watchers" errors
|
||||
|
||||
If `rbd remove` fails with "image still has watchers", a previous `rbd map` or crashed client holds the image open:
|
||||
|
||||
```bash
|
||||
# Find which node has the stale mapping
|
||||
rbd showmapped # on local node
|
||||
# Check ALL proxmox nodes:
|
||||
for node in proxmox4 proxmox5 proxmox6 proxmox7; do
|
||||
ssh root@$node "rbd showmapped | grep NNN"
|
||||
done
|
||||
|
||||
# Force unmap on the offending node
|
||||
ssh root@<node> "rbd unmap -o force /dev/rbdN"
|
||||
|
||||
# Then retry removal from monitor node
|
||||
rbd remove -p <pool> vm-NNN-disk-X
|
||||
```
|
||||
|
||||
⚠️ `rbd unmap` may fail with "Device or resource busy" if system processes hold it. Use `-o force` flag. After force unmap, the device disappears from `/dev/rbdN` and the watcher expires within 30s.
|
||||
|
||||
## Identifying Orphaned VM Contents (RBD Inspection)
|
||||
|
||||
When a VM config is already deleted and you need to know what the VM was (for risk assessment before deleting disks):
|
||||
|
||||
```bash
|
||||
# Map the RBD image to a block device
|
||||
rbd map -p hdd_disk vm-NNN-disk-X
|
||||
# → /dev/rbd10
|
||||
|
||||
# Check if it has a partition table
|
||||
fdisk -l /dev/rbd10
|
||||
blkid /dev/rbd10
|
||||
|
||||
# If ext4 directly (no partition table):
|
||||
mount /dev/rbd10 /mnt/inspect
|
||||
ls /mnt/inspect/
|
||||
|
||||
# If GPT partition table:
|
||||
mount /dev/rbd10p1 /mnt/inspect
|
||||
|
||||
# Identify the OS / purpose
|
||||
cat /mnt/inspect/etc/hostname
|
||||
cat /mnt/inspect/etc/os-release | head -3
|
||||
ls /mnt/inspect/ # look for model files, docker volumes, etc.
|
||||
|
||||
# ALWAYS clean up
|
||||
umount /mnt/inspect
|
||||
rbd unmap /dev/rbd10
|
||||
```
|
||||
|
||||
⚠️ Mounting RBD images on a cluster under heavy backfill load can timeout (30s+). If mount hangs, kill it and try later — or infer contents from disk size and structure (e.g., 300 GB ext4 with a single `.gguf` file = AI model storage).
|
||||
|
||||
⚠️ **Always `rbd unmap` after inspection.** Leftover mappings create "image still has watchers" errors when you later try to delete the image.
|
||||
|
||||
### Case Study: VM 202/203/116 Destruction (2026-07-04)
|
||||
|
||||
VMs 201, 202, 203 existed on n5pro (10.0.20.91, non-Ceph node) and VM 116 on proxmox6. All were stopped, `qm destroy` failed with CFS storage lock timeouts because Ceph was under heavy backfill load.
|
||||
|
||||
**Resolution:**
|
||||
1. Deleted VM configs manually on n5pro: `rm -f /etc/pve/qemu-server/{201,202,203}.conf` + stale locks
|
||||
2. Removed RBD images from proxmox7 (Ceph monitor node): `rbd remove -p hdd_disk vm-{201,202,203}-disk-*`
|
||||
3. VM 116 had a stale RBD mapping on proxmox6 (`/dev/rbd4` → `vm-116-disk-0` on `vm_disks` pool). Force unmapped: `rbd unmap -o force /dev/rbd4`, then removed both disk-0 (vm_disks) and disk-1 (hdd_disk, already gone).
|
||||
|
||||
**Contents identified via RBD inspection:**
|
||||
- VM 202: AI inference worker (Qwen3.6-35B-A3B-UD-IQ4_NL.gguf on 300 GB disk)
|
||||
- VM 203: Same structure (64 GB OS + 300 GB data), likely AI worker
|
||||
- VM 116: 192 GB ext4 data disk, no partition table (likely MariaDB-01)
|
||||
- Total freed: ~920 GB thin-provisioned
|
||||
|
||||
## Pitfalls
|
||||
|
||||
1. **`rbd du` timeouts** — On pools with many images, `rbd du` can hang. Use per-image `rbd info` instead, but batch to avoid SSH timeouts.
|
||||
2. **CSI volumes look orphaned but may be Kubernetes/external provisions** — Always check `rbd status` for watchers before deleting `csi-vol-*` images.
|
||||
3. **Base images with clones** — `rbd children` must return empty before deleting. Deleting a parent with active clones corrupts the clone.
|
||||
4. **Stopped VMs still own their disks** — A stopped VM (like VM 107 openclaw) still has a valid config referencing the disk. Only delete if the VMID has NO config at all (`qm config` AND `pct config` both fail).
|
||||
5. **Backup-locked CTs** — CT 111 had a `backup` lock. Don't delete disks of locked CTs even if stopped.
|
||||
6. **Timeout management** — Checking 50+ VMIDs via SSH can exceed 15s. Increase timeout to 30-60s or batch in groups of 10-15.
|
||||
7. **Non-Ceph nodes can't run `rbd` commands** — Nodes without `ceph.conf` (like n5pro) cannot connect to the Ceph cluster. Run all `rbd` operations from a monitor node.
|
||||
8. **`qm destroy` CFS lock timeout under Ceph load** — When Ceph is slow (backfilling, nearfull), Proxmox's CFS storage lock acquisition times out. Delete configs manually with `rm -f` and remove RBD images separately from a monitor node.
|
||||
9. **Stale RBD mappings block image deletion** — A previous `rbd map` that wasn't cleaned up (or a crashed QEMU) keeps watchers on the image. Check `rbd showmapped` on ALL nodes, force unmap, then retry removal.
|
||||
10. **Always unmap RBD images after inspection** — Mounted RBD images create watchers that prevent later deletion. `umount` + `rbd unmap` after every inspection session.
|
||||
11. **Mount can hang under heavy backfill** — When Ceph recovery I/O is high, `mount /dev/rbdN` can hang for 30s+. Abort and infer from disk structure if needed.
|
||||
Reference in New Issue
Block a user