# CFS Lock Stuck — `storage-rbd` Timeout Workaround

## When to Use

When `pct create` or `pct destroy` fails with `cfs-lock 'storage-rbd' error:
got lock request timeout`, the cluster-wide CFS lock for a storage pool
is stuck. Symptoms: repeated "trying to acquire cfs lock 'storage-rbd'..."
for ~30s then timeout. Other storage pools are unaffected.

## Root Cause

A previous operation (often a killed/interrupted `pct create` or `pct destroy`)
left a stale lock in the pmxcfs (Proxmox cluster filesystem) sqlite database.
The lock is cluster-wide — restarting `pve-cluster` on one node does NOT clear it.

## Fix: Use a Different Storage Pool

The fastest workaround: create the CT on a different RBD pool that isn't
locked. All RBD pools share the same Ceph cluster, so the CT runs identically.

```bash
# This fails (rbd pool locked):
pct create 142 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --rootfs rbd:32 ...

# This works (vm_disks pool not locked):
pct create 142 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --rootfs vm_disks:32 ...
```

## Fix: Force-Remove Stale CT + Lock

If the stuck lock is on a CT that was partially created:

```bash
# 1. Remove the CT config file directly (bypasses pct destroy)
rm -f /etc/pve/lxc/142.conf

# 2. Remove the RBD image in the background
rbd remove rbd/vm-142-disk-0 --no-progress &

# 3. Remove stale lock files on the node
rm -f /run/lock/lxc/pve-config-142.lock

# 4. Verify CT is gone
pct list | grep 142  # should return nothing
```

## Creating CTs with Correct Network Parameters

**Bridge**: Most nodes use `vmbr0` (not `vmbr1`) as the primary bridge.
VLAN tagging is done via the `tag=N` parameter, not a separate bridge.

**Template storage**: `rbd` pools do NOT support `vztmpl` content type.
Use `hdd_templates` (dir storage, mounted at `/mnt/hdd_templates`) or
`nfs_ubuntu` for template storage. Download templates with:
```bash
pveam download hdd_templates debian-12-standard_12.12-1_amd64.tar.zst
```

**DNS**: Use `--nameserver 10.0.30.1` (NOT `--net0 ...,dns=...` which is
invalid — the `dns` parameter is not in the pct schema).

**Correct CT creation command** (matching existing CT 141 pattern):
```bash
pct create 142 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --arch amd64 --ostype debian --cores 2 --memory 4096 --swap 2048 \
  --rootfs vm_disks:32 \
  --net0 name=eth0,bridge=vmbr0,tag=30,ip=10.0.30.142/24,gw=10.0.30.1,type=veth \
  --hostname omada-controller --onboot 1 --start 1 --nameserver 10.0.30.1
```

## Finding Which Node Hosts a CT

Use `pvesh` to query cluster resources:
```bash
pvesh get /cluster/resources --type vm --output-format json | python3 -c "
import json, sys
for i in json.load(sys.stdin):
    if i.get('type') == 'lxc' and i.get('status') == 'running':
        print(f\"CT {i['vmid']:>5} on {i['node']:<12}: {i['name']}\")
"
```

## Docker Host Inventory (2026-07-05)

Only 2 CTs run Docker in the cluster:
- **CT 141** (monitoring, proxmox7): 6 containers — telegram-bridge, grafana, pve-exporter, prometheus, alertmanager, blackbox
- **CT 121** (docker, proxmox3): 1 container — portainer

All other CTs (100, 104, 105, 108, 109, 112, 113, 116, 117, 124, 127, 133, 134, 135, 136, 137, 138, 140, 142, 231, 99999) run services natively without Docker. VM 230 (hermes-agent-01) also has no Docker.

## CT 142 Creation (2026-07-05)

Created CT 142 (omada-controller) on proxmox7 with vm_disks storage
after rbd CFS lock stuck. 2 cores, 4 GB RAM, 32 GB disk, VLAN 30,
IP 10.0.30.142. Will host local Omada Software Controller (Docker)
to replace cloud controller for SNMP/API monitoring.