99 lines
3.6 KiB
Markdown
99 lines
3.6 KiB
Markdown
# CFS Lock Stuck — `storage-rbd` Timeout Workaround
|
|
|
|
## When to Use
|
|
|
|
When `pct create` or `pct destroy` fails with `cfs-lock 'storage-rbd' error:
|
|
got lock request timeout`, the cluster-wide CFS lock for a storage pool
|
|
is stuck. Symptoms: repeated "trying to acquire cfs lock 'storage-rbd'..."
|
|
for ~30s then timeout. Other storage pools are unaffected.
|
|
|
|
## Root Cause
|
|
|
|
A previous operation (often a killed/interrupted `pct create` or `pct destroy`)
|
|
left a stale lock in the pmxcfs (Proxmox cluster filesystem) sqlite database.
|
|
The lock is cluster-wide — restarting `pve-cluster` on one node does NOT clear it.
|
|
|
|
## Fix: Use a Different Storage Pool
|
|
|
|
The fastest workaround: create the CT on a different RBD pool that isn't
|
|
locked. All RBD pools share the same Ceph cluster, so the CT runs identically.
|
|
|
|
```bash
|
|
# This fails (rbd pool locked):
|
|
pct create 142 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
|
|
--rootfs rbd:32 ...
|
|
|
|
# This works (vm_disks pool not locked):
|
|
pct create 142 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
|
|
--rootfs vm_disks:32 ...
|
|
```
|
|
|
|
## Fix: Force-Remove Stale CT + Lock
|
|
|
|
If the stuck lock is on a CT that was partially created:
|
|
|
|
```bash
|
|
# 1. Remove the CT config file directly (bypasses pct destroy)
|
|
rm -f /etc/pve/lxc/142.conf
|
|
|
|
# 2. Remove the RBD image in the background
|
|
rbd remove rbd/vm-142-disk-0 --no-progress &
|
|
|
|
# 3. Remove stale lock files on the node
|
|
rm -f /run/lock/lxc/pve-config-142.lock
|
|
|
|
# 4. Verify CT is gone
|
|
pct list | grep 142 # should return nothing
|
|
```
|
|
|
|
## Creating CTs with Correct Network Parameters
|
|
|
|
**Bridge**: Most nodes use `vmbr0` (not `vmbr1`) as the primary bridge.
|
|
VLAN tagging is done via the `tag=N` parameter, not a separate bridge.
|
|
|
|
**Template storage**: `rbd` pools do NOT support `vztmpl` content type.
|
|
Use `hdd_templates` (dir storage, mounted at `/mnt/hdd_templates`) or
|
|
`nfs_ubuntu` for template storage. Download templates with:
|
|
```bash
|
|
pveam download hdd_templates debian-12-standard_12.12-1_amd64.tar.zst
|
|
```
|
|
|
|
**DNS**: Use `--nameserver 10.0.30.1` (NOT `--net0 ...,dns=...` which is
|
|
invalid — the `dns` parameter is not in the pct schema).
|
|
|
|
**Correct CT creation command** (matching existing CT 141 pattern):
|
|
```bash
|
|
pct create 142 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
|
|
--arch amd64 --ostype debian --cores 2 --memory 4096 --swap 2048 \
|
|
--rootfs vm_disks:32 \
|
|
--net0 name=eth0,bridge=vmbr0,tag=30,ip=10.0.30.142/24,gw=10.0.30.1,type=veth \
|
|
--hostname omada-controller --onboot 1 --start 1 --nameserver 10.0.30.1
|
|
```
|
|
|
|
## Finding Which Node Hosts a CT
|
|
|
|
Use `pvesh` to query cluster resources:
|
|
```bash
|
|
pvesh get /cluster/resources --type vm --output-format json | python3 -c "
|
|
import json, sys
|
|
for i in json.load(sys.stdin):
|
|
if i.get('type') == 'lxc' and i.get('status') == 'running':
|
|
print(f\"CT {i['vmid']:>5} on {i['node']:<12}: {i['name']}\")
|
|
"
|
|
```
|
|
|
|
## Docker Host Inventory (2026-07-05)
|
|
|
|
Only 2 CTs run Docker in the cluster:
|
|
- **CT 141** (monitoring, proxmox7): 6 containers — telegram-bridge, grafana, pve-exporter, prometheus, alertmanager, blackbox
|
|
- **CT 121** (docker, proxmox3): 1 container — portainer
|
|
|
|
All other CTs (100, 104, 105, 108, 109, 112, 113, 116, 117, 124, 127, 133, 134, 135, 136, 137, 138, 140, 142, 231, 99999) run services natively without Docker. VM 230 (hermes-agent-01) also has no Docker.
|
|
|
|
## CT 142 Creation (2026-07-05)
|
|
|
|
Created CT 142 (omada-controller) on proxmox7 with vm_disks storage
|
|
after rbd CFS lock stuck. 2 cores, 4 GB RAM, 32 GB disk, VLAN 30,
|
|
IP 10.0.30.142. Will host local Omada Software Controller (Docker)
|
|
to replace cloud controller for SNMP/API monitoring.
|