CFS Lock Stuck — `storage-rbd` Timeout Workaround

When to Use

When pct create or pct destroy fails with cfs-lock 'storage-rbd' error: got lock request timeout, the cluster-wide CFS lock for a storage pool is stuck. Symptoms: repeated "trying to acquire cfs lock 'storage-rbd'..." for ~30s then timeout. Other storage pools are unaffected.

Root Cause

A previous operation (often a killed/interrupted pct create or pct destroy) left a stale lock in the pmxcfs (Proxmox cluster filesystem) sqlite database. The lock is cluster-wide — restarting pve-cluster on one node does NOT clear it.

Fix: Use a Different Storage Pool

The fastest workaround: create the CT on a different RBD pool that isn't locked. All RBD pools share the same Ceph cluster, so the CT runs identically.

# This fails (rbd pool locked):
pct create 142 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --rootfs rbd:32 ...

# This works (vm_disks pool not locked):
pct create 142 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --rootfs vm_disks:32 ...

Fix: Force-Remove Stale CT + Lock

If the stuck lock is on a CT that was partially created:

# 1. Remove the CT config file directly (bypasses pct destroy)
rm -f /etc/pve/lxc/142.conf

# 2. Remove the RBD image in the background
rbd remove rbd/vm-142-disk-0 --no-progress &

# 3. Remove stale lock files on the node
rm -f /run/lock/lxc/pve-config-142.lock

# 4. Verify CT is gone
pct list | grep 142  # should return nothing

Creating CTs with Correct Network Parameters

Bridge: Most nodes use vmbr0 (not vmbr1) as the primary bridge. VLAN tagging is done via the tag=N parameter, not a separate bridge.

Template storage: rbd pools do NOT support vztmpl content type. Use hdd_templates (dir storage, mounted at /mnt/hdd_templates) or nfs_ubuntu for template storage. Download templates with:

pveam download hdd_templates debian-12-standard_12.12-1_amd64.tar.zst

DNS: Use --nameserver 10.0.30.1 (NOT --net0 ...,dns=... which is invalid — the dns parameter is not in the pct schema).

Correct CT creation command (matching existing CT 141 pattern):

pct create 142 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --arch amd64 --ostype debian --cores 2 --memory 4096 --swap 2048 \
  --rootfs vm_disks:32 \
  --net0 name=eth0,bridge=vmbr0,tag=30,ip=10.0.30.142/24,gw=10.0.30.1,type=veth \
  --hostname omada-controller --onboot 1 --start 1 --nameserver 10.0.30.1

Finding Which Node Hosts a CT

Use pvesh to query cluster resources:

pvesh get /cluster/resources --type vm --output-format json | python3 -c "
import json, sys
for i in json.load(sys.stdin):
    if i.get('type') == 'lxc' and i.get('status') == 'running':
        print(f\"CT {i['vmid']:>5} on {i['node']:<12}: {i['name']}\")
"

Docker Host Inventory (2026-07-05)

Only 2 CTs run Docker in the cluster:

CT 141 (monitoring, proxmox7): 6 containers — telegram-bridge, grafana, pve-exporter, prometheus, alertmanager, blackbox
CT 121 (docker, proxmox3): 1 container — portainer

All other CTs (100, 104, 105, 108, 109, 112, 113, 116, 117, 124, 127, 133, 134, 135, 136, 137, 138, 140, 142, 231, 99999) run services natively without Docker. VM 230 (hermes-agent-01) also has no Docker.

CT 142 Creation (2026-07-05)

Created CT 142 (omada-controller) on proxmox7 with vm_disks storage after rbd CFS lock stuck. 2 cores, 4 GB RAM, 32 GB disk, VLAN 30, IP 10.0.30.142. Will host local Omada Software Controller (Docker) to replace cloud controller for SNMP/API monitoring.

3.6 KiB Raw Blame History

CFS Lock Stuck — storage-rbd Timeout Workaround