hermes-skills/devops/proxmox-ve-administration/SKILL.md at e9cc10662516e4b1d2ba8d8803ed80060e4bda62

dominik/hermes-skills

Fork 0

Files

T

Debian e9cc106625 Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00

98 KiB

Raw Blame History

name, description, version, tags

name

description

version

Overview

This umbrella skill consolidates all Proxmox VE administration tasks:

API/Auth — SSH key ~/.ssh/id_ed25519_proxmox, no BatchMode. RBD LXC: see references/lxc-creation-rbd-clone-2026-07.md
Avahi Reflector — CT 127 dual-VLAN, unidirectional. See references/avahi-reflector-setup.md
LXC GPU — privileged CT, device pass-through, ROCm/CUDA/Intel Arc drivers
PVE-Native Volume Ops — pct move-volume, pct create --storage, pvesm alloc + mke2fs, unprivileged CT permission mapping, Samba TimeMachine CT setup. See references/pve-native-volume-ops-2026-07.md
Avahi mDNS Reflector — cross-VLAN Bonjour/TimeMachine discovery via dual-interface CT. See references/avahi-reflector-setup.md
Storage — iSCSI/ZFS/RBD, Ceph pool redundancy, EC clone caveats, PG management, pool-full recovery, EC reweight trap, backfillfull flag, recovery tuning. See references/lxc-creation-rbd-clone-2026-07.md, references/ceph-pg-management-2026-07.md, and references/ceph-recovery-ec-reweight-2026-07.md
VLAN Bridge Setup — correct interface creation order (VLAN first, bridge second), CT networking
Infrastructure Monitoring — Prometheus + Grafana + Alertmanager + Hermes webhook (auto RCA to Telegram). PVE exporter tuning, dashboard datasource fixes, alerting rules, Ceph/Galera/Blackbox coverage. See references/alertmanager-webhook-setup.md and templates/alerting_rules.yml.

Load this skill for any Proxmox cluster operation.

Section 1: API Authentication & Multi-Node SSH

1.1 API Token Format (CRITICAL)

PVEAPIToken=USERNAME!TOKENID=SECRET

USERNAME — PVE user (e.g. root@pam)
TOKENID — token identifier (e.g. hermes)
SECRET — UUID secret key

Common mistake: repeating the token ID: root@pam!hermes!hermes=... is WRONG. Correct is root@pam!hermes=SECRET.

1.2 Multi-Node SSH Key Deployment

# 1. Discover nodes
ssh root@10.0.20.91 "pvesh get /cluster/status --output-format json"

# 2. Generate key
ssh-keygen -t ed25519 -C "hermes-agent@proxmox" -f ~/.ssh/id_ed25519_proxmox -N ""

# 3. Deploy via sshpass
for ip in 10.0.20.{10,20,30,40,50,60,70,91}; do
  sshpass -p 'PASSWORD' ssh -o StrictHostKeyChecking=no \
    root@$ip "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys" \
    < ~/.ssh/id_ed25519_proxmox.pub
done

# 4. Verify
for ip in 10.0.20.{10,20,30,40,50,60,70,91}; do
  ssh -o BatchMode=yes -i ~/.ssh/id_ed25519_proxmox root@$ip "echo OK; hostname"
done

1.3 Store SSH Key in 1Password (SECURE_NOTE workaround)

1Password's built-in "SSH Key" category silently drops private keys. Use SECURE_NOTE + CONCEALED field instead.

python3 -c "
import json, subprocess
pk=open('/home/debian/.ssh/id_ed25519_proxmox').read().strip()
puk=open('/home/debian/.ssh/id_ed25519_proxmox.pub').read().strip()
fpr=subprocess.run(['ssh-keygen','-lf','/home/debian/.ssh/id_ed25519_proxmox.pub'], capture_output=True, text=True).stdout.strip()
template={'title':'hermes-agent Proxmox SSH Key','category':'SECURE_NOTE','fields':[
  {'id':'notesPlain','type':'STRING','purpose':'NOTES','label':'notesPlain','value':f'Public key:\n{puk}\n\nFingerprint:\n{fpr}'},
  {'id':'private_key','type':'CONCEALED','label':'private key','value':pk}
]}
json.dump(template, open('/tmp/op_template.json','w'))
"
cat /tmp/op_template.json | op item create - --vault Hermes

1.4 API Response Units (CRITICAL)

Memory (/status): BYTES → divide by 1024**3 for GB
Storage (/storage): kB → divide by 1024**2 for GB
No top-level status key in /nodes/{node}/status — the response IS the raw dict
Use (data or {}).get(key, 0) or 0 safe access since PVE returns None for missing values

1.5 Common Endpoints

Endpoint	Description
`/api2/json/nodes`	List nodes
`/api2/json/nodes/{node}/status`	Node health
`/api2/json/nodes/{node}/storage`	Storage list
`/api2/json/nodes/{node}/storage/{storage}/status`	Specific storage status
`/api2/json/nodes/{node}/qemu`	VMs on node
`/api2/json/nodes/{node}/lxc`	CTs on node

1.6 Node IP Mapping (Schön Homelab)

Cluster "Homelab" — 8 nodes. IPs from corosync.conf:

Node	Management IP	Role
proxmox1	10.0.20.10	HA (VM 106)
proxmox2	10.0.20.20	MariaDB-02 (VM 301)
proxmox3	10.0.20.30	—
proxmox4	10.0.20.40	MariaDB-03 (VM 302)
proxmox5	10.0.20.50	—
proxmox6	10.0.20.60	—
proxmox7	10.0.20.70	—
n5pro	10.0.20.91	Ceph OSDs, GPU compute

SSH: ssh -i ~/.ssh/id_ed25519_proxmox root@<IP>

1.7 Checking Updates Across All Nodes

# Running version
ssh root@NODE "pveversion"

# Pending updates count
ssh root@NODE "apt list --upgradable 2>/dev/null | grep -c '[upgradable]'"

# Key packages (PVE, QEMU, Ceph, Security, Kernel)
ssh root@NODE "apt list --upgradable 2>/dev/null | grep -E 'proxmox|pve-|qemu-server|ceph/stable|kernel|openssl|libssl'"

1.8 Node Update Procedure (ITIL Change Management)

Check all nodes for pending updates:

# Node IP mapping from corosync.conf
for ip_info in "10.0.20.10:proxmox1" "10.0.20.20:proxmox2" "10.0.20.30:proxmox3" \
  "10.0.20.40:proxmox4" "10.0.20.50:proxmox5" "10.0.20.60:proxmox6" \
  "10.0.20.70:proxmox7" "10.0.20.91:n5pro"; do
  ip="${ip_info%%:*}"; node="${ip_info##*:}"
  echo "=== $node ($ip) ==="
  ssh -o ConnectTimeout=5 -i ~/.ssh/id_ed25519_proxmox root@$ip \
    "pveversion | head -1; apt list --upgradable 2>/dev/null | grep -c '[upgradable]'"
done

⚠️ PVE 9.1.x maintenance mode: The API endpoint /nodes/{node}/control/maintenance does NOT exist in PVE 9.1.x (added in 9.2). On 9.1.x nodes, you cannot set maintenance mode. Instead:

Live-migrate VMs (qm migrate) — works for KVM guests
LXC live migration is NOT implemented — CTs cannot be hot-migrated. They must be stopped/moved or the node is updated in-place with CTs running
Safe approach for CTs: apt full-upgrade (host packages update alongside running CTs), then reboot (CTs restart on the same node)

Update one node at a time (recommended order):

Compute-only nodes first (proxmox6, proxmox7) — lowest risk
DB-hosting nodes (proxmox4, proxmox5) — MariaDB VMs should be live-migrated off first
Management nodes (proxmox1-3) — HA VM on proxmox1 needs care
n5pro last — Ceph OSDs, last-resort storage

Per-node procedure:

# 1. (For 9.2.x nodes only) Set maintenance mode
# pvesh create /nodes/{node}/control/maintenance --startup=ignore
# NOTE: Fails on PVE 9.1.x — endpoint does not exist!

# 2. (Optional) Live-migrate VMs off the node for safety
# qm migrate <vmid> <target-node> --online

# 3. Update packages (CTs keep running during this step)
apt update && apt full-upgrade -y

# 4. Reboot (required for kernel changes, CTs restart automatically)
reboot

# 5. After reboot, verify
pveversion
pvecm status
pvesh get /nodes/{node}/lxc | grep running  # CTs came back up

# 6. (For 9.2.x nodes) Clear maintenance mode
# pvesh create /nodes/{node}/control/maintenance --startup=started

Major version gaps (9.1.x → 9.2.x): Expect ~170 packages including QEMU 10→11, kernel 7.0.0→7.0.12, and openssl security patches. apt full-upgrade is required (not upgrade).

1.9 Rolling Cluster Update: Guest Migration Strategies

When updating nodes that host running guests, guests must be evacuated first. The strategy depends on guest type and HA management status.

Migration Decision Matrix

Guest Type	HA-managed?	Method	Downtime
QEMU VM (Ceph RBD)	Yes	`qm migrate <vmid> <target> --online` (manual, NOT via HA-manager)	Zero (live)
QEMU VM (Ceph RBD)	No	`qm migrate <vmid> <target> --online`	Zero (live)
LXC CT (any storage)	Any	Cold migration: `pct stop` → `pct migrate` → `pct start`	Full stop duration
LXC CT (local storage)	Any	Backup/restore or clone to target	Full stop + transfer

⚠️ Prefer manual qm migrate over HA-manager migration. The HA-manager can abort migrations unpredictably, leaving the VM in a locked state. Manual qm migrate gives you full control, especially critical for Galera cluster members where quorum must be preserved.

Cold Migration Workflow for LXC CTs (Confirmed Working)

# 1. Stop the CT
pct stop <ctid>

# 2. Migrate to target node (storage must be shared or migrated)
pct migrate <ctid> <target-node>

# 3. Start on target
pct start <ctid>

# 4. Verify
pct status <ctid>  # Should show "running"

Live Migration Workflow for QEMU VMs

# Manual live migration (works with Ceph RBD shared storage)
qm migrate <vmid> <target-node> --online

# Check migration success
qm status <vmid>  # Should show "running" on target

1.10 HA Migration Lock Recovery (CRITICAL)

When HA-managed VM migration fails or is interrupted, the VM can get stuck with a lock: migrate state. The VM may have actually migrated successfully, but the lock prevents further operations.

Symptoms:

qm status <vmid> shows lock: migrate
ha-manager status shows the service in migrate state
Further qm commands fail with "VM is locked"

Recovery procedure:

# 1. Check if VM actually migrated (it may have!)
ssh root@<original-node> "qm status <vmid>"  # May show "stopped" or not-found
ssh root@<target-node> "qm status <vmid>"   # May show "running"

# 2. Clear QEMU lock
qm set <vmid> --lock none

# 3. Clear HA-manager state (if HA-managed)
ha-manager set --disable migrate qemu<vmid>  # or ct<vmid>

# 4. Verify HA state normalizes
ha-manager status | grep <vmid>

Key insight: Always verify the VM's ACTUAL location before attempting re-migration. The lock may be stale — the VM may already be running on the target node.

1.11 Galera Cluster Quorum Awareness During Migration

When migrating MariaDB Galera cluster members (VMs), quorum must be preserved:

Never migrate more than one Galera VM at a time
Check wsrep_cluster_status after each migration step — should show Primary
Migration order: Migrate one node, verify quorum, then proceed to next
Avoid parallel migrations of Galera members even if technically possible

In the Schön homelab: VM 300 (mariadb-01), VM 301 (mariadb-02), VM 302 (mariadb-03) form the Galera cluster. Always sequential.

1.12 Shell Quoting Through SSH (PITFALL)

Complex shell commands with nested quotes fail through SSH. Multiple quoting layers (local shell → SSH → remote shell) cause unexpected EOF and parse errors.

❌ Fails — nested single quotes:

ssh root@node "apt-get full-upgrade -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold'"

✅ Works — single-quote outer, double-quote inner:

ssh root@node 'DEBIAN_FRONTEND=noninteractive apt-get update -qq && apt-get full-upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" 2>&1 | tail -20'

Rule: Use single quotes for the outer SSH command argument, double quotes for any inner quoting needs. Avoid backticks and $() inside SSH command strings — use pipes instead.

1.13 Cluster-Wide Guest Inventory & HA Status (CRITICAL)

⚠️ Do NOT rely on pvesh get /cluster/resources --type vm for HA status. The ha field in that API response is unreliable — it reported all 55 guests as non-HA when 12 were actually HA-managed.

Correct method — use ha-manager status for HA membership, then cross-reference with /cluster/resources for guest metadata:

# Step 1: Get HA-managed service IDs from ha-manager
HA_SERVICES=$(ssh root@<any-node> 'ha-manager status' | grep '^service ' | \
  awk '{gsub(/[:;]/,"",$2); gsub(/[(:]/,"",$3); print $2":"$3}')

# Step 2: Get full guest inventory
ssh root@<any-node> 'pvesh get /cluster/resources --type vm --output-format json' | \
  python3 -c "
import json, sys, subprocess
data = json.load(sys.stdin)
# Parse ha-manager status to get HA-managed IDs
hm = subprocess.run(['ssh', '-i', '~/.ssh/id_ed25519_proxmox', 'root@10.0.20.10',
    'ha-manager status'], capture_output=True, text=True).stdout
ha_ids = set()
for line in hm.splitlines():
    if line.startswith('service '):
        parts = line.split()
        # service ct:104 (n5pro, started) → extract 'ct:104' → normalize to '104'
        sid = parts[1].rstrip(':')
        ha_ids.add(sid.replace('ct:','').replace('vm:',''))
for g in sorted(data, key=lambda x: (x['node'], x['vmid'])):
    ha = 'YES' if str(g['vmid']) in ha_ids else 'no'
    print(f\"{g['node']:12} {g['vmid']:6} {g.get('name','?'):25} {g['status']:10} HA:{ha}  type={g.get('type','?')}\")
"

Simple quick-check (just HA services):

ssh root@<any-node> 'ha-manager status' | grep '^service '

Output format: service <type>:<id> (<node>, <state>) — e.g. service vm:300 (proxmox2, started).

LRM health check (also from ha-manager status):

ssh root@<any-node> 'ha-manager status' | grep '^lrm '

Watch for old timestamp - dead? — indicates the Local Resource Manager on that node is not reporting. See Section 1.15 for recovery.

1.14 LRM Disabled After Rolling Update (PITFALL)

During the 2026-07-02 rolling cluster update, pve-ha-lrm on proxmox1 was left disabled and inactive (dead). The ha-manager status showed lrm proxmox1 (old timestamp - dead?) with a stale timestamp from March 2026. This went unnoticed because the CRM master was on proxmox4 and HA services on other nodes continued normally.

Root cause: The pve-ha-lrm service was manually disabled at some point (or a reboot cycle didn't re-enable it). Unlike corosync and pvedaemon, the LRM is not auto-enabled by Proxmox's post-boot hooks if it was previously masked/disabled.

Detection:

# Check all nodes for LRM health
ssh root@<any-node> 'ha-manager status | grep "^lrm "'
# Watch for: "old timestamp - dead?" or missing entries

# Direct check on suspected node
ssh root@<node-ip> 'systemctl is-enabled pve-ha-lrm; systemctl is-active pve-ha-lrm'
# Healthy: enabled / active (running) OR idle (no HA guests on that node)
# Broken: disabled / inactive (dead)

Recovery:

ssh root@<affected-node-ip> 'systemctl enable pve-ha-lrm && systemctl start pve-ha-lrm'
# Wait ~10s for lock acquisition, then verify
sleep 10
ssh root@<any-node> 'ha-manager status | grep "^lrm "'
# Affected node should now show: idle/watchdog standby (no HA guests) or active/watchdog active (has HA guests)

Post-update checklist — add LRM verification to the rolling update procedure (Section 1.8):

# After ALL nodes are updated and rebooted, verify LRM on every node
for ip_info in "10.0.20.10:proxmox1" "10.0.20.20:proxmox2" "10.0.20.30:proxmox3" \
  "10.0.20.40:proxmox4" "10.0.20.50:proxmox5" "10.0.20.60:proxmox6" \
  "10.0.20.70:proxmox7" "10.0.20.91:n5pro"; do
  ip="${ip_info%%:*}"; node="${ip_info##*:}"
  echo "=== $node ==="
  ssh -o ConnectTimeout=5 -i ~/.ssh/id_ed25519_proxmox root@$ip \
    'systemctl is-enabled pve-ha-lrm; systemctl is-active pve-ha-lrm'
done

LRM states explained:

`ha-manager status` output	Meaning
`active, watchdog active`	Healthy, hosting HA-managed guests
`idle, watchdog standby`	Healthy, no HA-managed guests on this node
`old timestamp - dead?`	BROKEN — LRM not running, needs enable+start

Note: pve-ha-crm (CRM) only runs on the elected master node — it is normal for CRM to be inactive on non-master nodes. Only pve-ha-lrm must run on ALL nodes.

1.15 HA Management Operations (Add, Remove, Configure, Audit)

Adding a Guest to HA Management

ha-manager add <type>:<id>
# e.g. ha-manager add vm:106
# e.g. ha-manager add ct:108

The guest must exist and be on a node with a healthy LRM (Section 1.14). After adding, the HA-manager transitions through request_start_balance → starting → started (~10-25s).

Removing a Guest from HA Management

ha-manager remove <type>:<id>

Does NOT delete the guest config — only removes the HA management overlay.

Setting HA Parameters

ha-manager set <type>:<id> --max_relocate <N> --max_restart <N>

max_restart — max restart attempts on the SAME node before relocating
max_relocate — max relocation attempts to a DIFFERENT node before giving up

Recommended values:

Guest Class	max_relocate	max_restart	Rationale
Production (DB, infra)	3	2	Balanced — local restart twice, then relocate 3×
Non-critical	1	1	Default — minimal retry pressure
Overly aggressive (avoid)	5	5	Causes restart storms on flapping nodes

Inspecting HA Resource Config (CRITICAL — CLI gotcha)

⚠️ ha-manager config <sid> does NOT work — returns 400 too many arguments. No CLI subcommand exists to view individual HA resource config.

Use pvesh instead:

# All HA resources with full config
pvesh get /cluster/ha/resources --output-format json

# Quick summary
pvesh get /cluster/ha/resources --output-format json | \
  python3 -c "
import json,sys
data=json.load(sys.stdin)
for g in sorted(data, key=lambda x: x['sid']):
    print(f\"{g['sid']:12} state={g.get('state','?'):10} reloc={g.get('max_relocate','?')} restart={g.get('max_restart','?')}\")"

Fields: sid, state, type, max_relocate, max_restart, digest.

Cleaning Up Orphaned Guest Configs from Removed Nodes

When a node is removed from the cluster (pvecm delnode), its guest config files remain under /etc/pve/nodes/<removed-node>/. These appear in /cluster/resources with status=unknown — "ghost guests".

Detection:

pvesh get /cluster/resources --type vm --output-format json | python3 -c "
import json,sys
data=json.load(sys.stdin)
valid_nodes={'proxmox1','proxmox2','proxmox3','proxmox4','proxmox5','proxmox6','proxmox7','n5pro'}
for g in data:
    if g['status']=='unknown' or g['node'] not in valid_nodes:
        t='ct' if g['type']=='lxc' else 'vm'
        print(f\"GHOST: {t}:{g['vmid']} node={g['node']}\")"

Cleanup:

find /etc/pve/nodes/<removed-node>/qemu-server -name "*.conf"
find /etc/pve/nodes/<removed-node>/lxc -name "*.conf"
rm /etc/pve/nodes/<removed-node>/qemu-server/<id>.conf
rm /etc/pve/nodes/<removed-node>/lxc/<id>.conf

Do NOT delete directories for active cluster nodes.

1.14 EFI Partition Cleanup Before Major Upgrades

Before upgrading across major PVE versions (especially 9.1.x → 9.2.x with kernel changes), check and clean the EFI partition:

# Check EFI usage
df -h /boot/efi

# List ESP contents
ls /boot/efi/

# Remove old kernel EFI entries if full (keep current + previous)
# Use efibootmgr to identify stale entries
efibootmgr -v

EFI partitions can fill up with stale kernel entries across multiple upgrades, blocking new kernel installation.

1.15 Root Disk Space Recovery: Remove Unused pve-data Thin Pool

PVE's default installer creates a pve-data LVM thin pool (local-lvm storage) that is often unused when all VM disks are on Ceph/RBD. This wastes 10-140 GB that could extend pve-root.

Detection — check all nodes:

for ip in 10.0.20.{10,20,30,40,50,60,70} 10.0.20.91; do
  ssh -o ConnectTimeout=5 -i ~/.ssh/id_ed25519_proxmox root@$ip \
    'lvs -o lv_name,data_percent --units g 2>/dev/null | grep -E "data|root|swap"; echo "VG Free: $(vgs pve -o vg_free --units g --noheadings 2>/dev/null)"; df -h / | tail -1'
done

Recovery (online, no downtime):

# 1. Verify pve-data is empty (0% data_percent)
lvs -o lv_name,data_percent pve/data 2>/dev/null

# 2. Remove the thin pool
lvremove -y pve/data

# 3. Extend root LV to use freed space
lvextend -l +100%FREE pve/root

# 4. Online resize ext4 filesystem
resize2fs /dev/pve/root

# 5. Verify
df -h /
lvs -o lv_name,lv_size --units g

⚠️ Only remove if local-lvm storage is not used. Check pvesm status — if local-lvm shows 0 used, it's safe. If VMs/CTs use local-lvm, their disks will be destroyed.

1.16 Comprehensive Cluster Audit Checklist

For a full cluster health audit, collect these data points:

Corosync: pvecm status, corosync-cfgtool -s (ring connectivity), corosync-quorumtool (quorum)
HA-Manager: ha-manager status (services + LRM health + CRM master)
Node versions: pveversion on each node (compare PVE + kernel versions)
Node resources: pvesh get /cluster/resources --type node (CPU/MEM/disk/uptime)
Per-node details: swap usage, CPU model, Ceph mon/OSD presence, disk listing
Ceph health: ceph -s, ceph health detail, ceph osd tree, ceph osd df, ceph df
Ceph performance: ceph osd perf (commit/apply latency per OSD)
Ceph config: ceph config dump, pool configs via ceph osd pool get <pool> all
Ceph CRUSH: ceph osd crush rule dump (verify class-based rules)
Ceph Mon placement: ceph mon dump (recommend 3 or 5, not 7+)
Disk SMART: smartctl -a /dev/sdX on OSD disks (health, temp, power-on hours)
Network: ip link show (MTU, bonds), /proc/net/bonding/bond0 (bond mode, slave count)
Datacenter config: /etc/pve/datacenter.cfg (CRS, migration, replication settings)

See references/cluster-audit-2026-07.md for a complete worked example with findings and recommendations.

2.1 Privileged Container Requirement

Privileged LXC is usually required because /dev/dri and /dev/kfd are root-owned. Unprivileged works for some Intel Arc setups with proper lxc.idmap + subuid/gid mapping.

2.2 Container Config (`/etc/pve/lxc/<ctid>.conf`)

AMD ROCm:

features: nesting=1,keyctl=1
unprivileged: 0
lxc.cgroup2.devices.allow: c 226:* rwm    # /dev/dri
lxc.cgroup2.devices.allow: c 511:0 rwm    # /dev/kfd
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file

NVIDIA CUDA:

lxc.cgroup2.devices.allow: c 195:* rwm    # /dev/nvidia*
lxc.cgroup2.devices.allow: c 243:* rwm    # /dev/nvidia-uvm*
lxc.cgroup2.devices.allow: c 235:* rwm    # /dev/nvidiactl
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file

2.3 ROCm Userland-Only Install (`--no-dkms`)

Do NOT install kernel modules inside the container — host already loads amdgpu.

wget https://repo.radeon.com/amdgpu-install/6.3.3/ubuntu/noble/amdgpu-install_6.3.60303-1_all.deb
dpkg -i amdgpu-install_6.3.60303-1_all.deb && apt-get update
amdgpu-install --usecase=rocm --no-dkms -y

Takes 30–90 min. Run in background:

nohup bash -c "amdgpu-install --usecase=rocm --no-dkms -y > /tmp/rocm_install.log 2>&1; echo DONE > /tmp/rocm_done" &
watch 'tail -5 /tmp/rocm_install.log'

2.4 Disk Space Management

ROCm + PyTorch + ComfyUI = 40–60 GB. Rootfs needs 80 GB minimum.

Emergency resize:

pct exec <ctid> -- bash -c 'pip cache purge; apt-get clean; rm -rf /root/.cache/pip /tmp/pip-*'
pct resize <ctid> rootfs 80G
pct exec <ctid> -- bash -c 'resize2fs $(df / | tail -1 | awk "{print \$1}")'

2.5 RDNA3.5 Workaround (`HSA_OVERRIDE_GFX_VERSION`)

Ryzen AI 9 HX PRO 370 (gfx1150) reports HIP error: invalid device function. Fix:

export HSA_OVERRIDE_GFX_VERSION=11.0.0
# In systemd service:
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"

2.6 PyTorch ROCm Wheel Pitfalls

torchaudio ROCm wheel links against CUDA libraries → OSError: libcudart.so. Fix: install CPU torchaudio or stub it out.
python3-rich Debian package blocks pip upgrades → dpkg --remove --force-depends python3-rich
LD_LIBRARY_PATH must include /usr/local/lib/python3.12/dist-packages/torch/lib for libtorchaudio.so
After ROCm upgrade, verify PyTorch compatibility. If hipErrorNoBinaryForGpu, downgrade PyTorch ROCm wheel to match host's ROCm version.

2.7 Verification Steps

export PATH=/opt/rocm/bin:$PATH
rocminfo | grep "Name:" | head -5
rocm-smi
python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"
python3 -c "import torch; print(torch.cuda.get_device_capability())"

2.8 When Agent Is Not on the Proxmox Host

pct/pveam are unavailable locally. Use SSH/paramiko:

import paramiko
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(hostname="10.0.20.91", username="root", password="...")
stdin, stdout, stderr = client.exec_command("pct list && pct exec 204 -- uname -a")

2.9 Template Download Gotcha

pveam download requires a storage name, not a node name:

# WRONG: pveam download n5pro ubuntu-24.04-standard_24.04-2_amd64.tar.zst
# CORRECT: pveam download hdd_templates ubuntu-24.04-standard_24.04-2_amd64.tar.zst

2.10 Safe HuggingFace Model Download Inside LXC

Use resolve URL, not browser blob URL:
- https://huggingface.co/org/repo/resolve/main/path/file.safetensors
Always use -O filename with wget
Resume with wget -c
Single pct exec command for large downloads:

pct exec <ctid> -- bash -c 'cd /opt/ComfyUI/models/diffusion_models && wget -c --tries=0 --read-timeout=30 "https://huggingface.co/Comfy-Org/Ideogram-4/resolve/main/diffusion_models/ideogram4_fp8_scaled.safetensors" -O ideogram4_fp8_scaled.safetensors'

Section 3: Storage Diagnostics

3.1 Identify the Outlier Node

Shared storage works on most nodes but fails on one — the most common pattern.

for node in nodes:
    r = subprocess.run(["curl","-sk",f"https://{PVE}:8006/api2/json/nodes/{node}/storage/{storage_name}/status","-H",f"Authorization: {auth}"], capture_output=True, text=True, timeout=15)
    data = json.loads(r.stdout).get("data", {})
    if data:
        print(f"{node:<12} active={data.get('active')} total={data.get('total',0)/(1024**3):.0f}GB")
    else:
        print(f"{node:<12} ❌ NO DATA")

3.2 Check iSCSI Target Config

r = subprocess.run(["curl","-sk",f"https://{PVE}:8006/api2/json/storage/{storage_name}","-H",f"Authorization: {auth}"], capture_output=True, text=True)
config = json.loads(r.stdout).get("data", {})
# Key fields: portal, target, pool, iscsiprovider

3.3 Common Failure Patterns

Symptom	Cause
`active=0, total=0` on one node	iSCSI session not connected
Empty storage list	ZFS pool gone or renamed
"Permission denied" but config exists	Underlying resource missing
Memory values look impossibly large	Values are BYTES, not kB

3.4 Ext4 MMP Deadlock on RBD-Backed LXC Containers (CRITICAL)

When an LXC container with an RBD (Ceph) root disk is stopped uncleanly (hard stop, host crash, or the mount process was killed mid-mount), the Ext4 filesystem's MMP (Multi-Mount Protection) block can be left in an active state. On the next pct start, the mount process enters D-state (uninterruptible sleep) waiting for MMP to stabilize — but MMP never stabilizes because the mount process itself holds the MMP lock. This creates a kernel-level deadlock that cannot be resolved by killing the process (D-state is immune to SIGKILL).

Symptoms:

pct start <ctid> hangs indefinitely (timeout)
pct status <ctid> also hangs
mount -o discard /dev/rbd0 /var/lib/lxc/.pve-staged-mounts/rootfs process in D state
dmesg shows: EXT4-fs warning (device rbd0): ext4_multi_mount_protect:324: MMP interval 42 higher than expected, please wait.
rbd unmap fails with Device or resource busy
tune2fs -E clear_mmp fails with MMP: device currently active
e2fsck fails with MMP: e2fsck being run while checking MMP block

Recovery Procedure (confirmed working on PVE 9.2.3 / Ceph 19.2.3):

# PHASE 1: Break the D-state deadlock via Ceph client blacklist

# 1. Identify the stuck mount process and the RBD client
ps aux | grep "mount.*rbd" | grep -v grep
# Note the PID — it will be in D state

# 2. Find the RBD exclusive lock holder
rbd --id admin lock list <pool>/<image> --format json
# Note: locker ID, address, and cookie

# 3. Blacklist the Ceph client — this forces the kernel RBD driver
#    to error out all outstanding I/O, which releases the D-state process
ceph --conf /etc/pve/ceph.conf --keyring /etc/pve/priv/ceph.client.admin.keyring \
  osd blacklist add <client_address>
# e.g. ceph osd blacklist add 10.0.20.50:0

# 4. Wait for the D-state process to exit (usually 3-5 seconds)
sleep 5
ps aux | grep "mount.*rbd" | grep -v grep || echo "mount process freed!"

# PHASE 2: Clean up RBD state

# 5. Unmap the RBD device
rbd --id admin unmap /dev/rbd0
# If unmap fails with "Device or resource busy", blacklist again (new client ID)
# and retry. The blacklist creates a new RBD client session each time.

# 6. Remove ALL blacklists (they persist for 3600s by default)
ceph osd blacklist ls  # List all
ceph osd blacklist rm <address>  # Remove each one

# 7. Verify RBD lock is cleared
rbd --id admin lock list <pool>/<image>
# Should show no locks

# PHASE 3: Clear MMP and restart container

# 8. Map RBD fresh
rbd --id admin map <pool>/<image>

# 9. Clear MMP with FORCE flag (CRITICAL: -f required!)
tune2fs -f -E clear_mmp /dev/rbd/<pool>/<image>
# Without -f: "MMP: device currently active while trying to open"
# With -f: "Recovering journal." + success

# 10. Verify MMP is cleared
dumpe2fs /dev/rbd/<pool>/<image> | grep -i "multi-mount" || echo "MMP cleared!"

# 11. Unmap RBD (let PVE handle remapping during pct start)
rbd --id admin unmap /dev/rbd/<pool>/<image>

# 12. Start the container
pct start <ctid>

Key Pitfalls:

tune2fs -E clear_mmp WITHOUT -f fails — it sees the MMP block as "active" even when no process holds the device. The -f (force) flag is mandatory.
Blacklisting creates new RBD client sessions — each osd blacklist add generates a new client ID. You may need to blacklist 2-3 times as each new mapping attempt creates a new client that also gets stuck. List and remove ALL blacklists before the final clean map.
rbd lock remove syntax — the command is rbd lock remove <image> <locker> <cookie>, NOT rbd lock break. The locker and cookie must be parsed from rbd lock list --format json. However, blacklisting the client is usually more effective than trying to remove the lock manually.
smbd may not auto-start after recovery — after an unclean CT restart, smbd may remain inactive (dead) even though the container is running. Manually start it: pct exec <ctid> -- systemctl start smbd. Check with pct exec <ctid> -- systemctl is-active smbd nmbd avahi-daemon.
Do NOT reload the rbd kernel module — this would disconnect ALL RBD devices on the node, affecting all VMs/CTs using Ceph storage. The blacklist approach is surgical — it only affects the specific stuck client.

When to use this procedure:

LXC container with RBD root disk fails to start (hangs on pct start)
dmesg shows MMP-related Ext4 warnings on an RBD device
A mount process is stuck in D-state on an RBD device
pct status for the CT hangs or times out

Alternative (if blacklist doesn't work): Rebooting the PVE node hosting the CT will clear all D-state processes and RBD mappings. This is heavier (affects all guests on the node) but guaranteed to work. Use only if the blacklist approach fails after 3 attempts.

Section 4: VLAN Bridge Setup

4.1 Critical Order: VLAN FIRST, Bridge SECOND

The bridge's bridge_ports must reference an interface that already exists.

# 1. Create VLAN interface
curl -k -X POST https://10.0.20.10:8006/api2/json/nodes/proxmox1/network \
  -H "Cookie: PVEAuthCookie=TICKET" -H "CSRFPreventionToken: TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"iface":"bond0.30","type":"vlan","vlan-id":"30","vlan-raw-device":"bond0","autostart":1}'

# 2. Create bridge referencing the VLAN
curl -k -X POST https://10.0.20.10:8006/api2/json/nodes/proxmox1/network \
  -H "Cookie: PVEAuthCookie=TICKET" -H "CSRFPreventionToken: TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"iface":"vmbr30","type":" bridge","cidr":"10.0.30.1/24","bridge_ports":"bond0.30","autostart":1}'

4.2 Adding a NIC to a VM with VLAN Tagging

On a VLAN-aware bridge (bridge-vlan-aware yes), add a NIC tagged to a specific VLAN:

qm set <vmid> -net1 virtio=<MAC>,bridge=vmbr0,tag=50
# tag=50 = VLAN 50. Use a specific MAC if recreating a previously-removed NIC.
# Requires VM reboot for the guest OS to detect the new interface.

Use case: giving a HAOS VM access to an IoT VLAN for mDNS/Zeroconf discovery (mDNS multicast doesn't cross VLAN boundaries). After adding the NIC, configure it inside the guest OS (for HAOS: ha network update <iface> --ipv4-method auto).

Multiple NICs for multiple VLANs: The same pattern applies for additional VLANs. This homelab's HA VM 106 now has 3 NICs:

net0 → VLAN 30 (10.0.30.10) — management/services
net1 → VLAN 50 (10.0.50.127) — Shellys/IoT (added 2026-06-27)
net2 → VLAN 40 (10.0.40.61) — Midea AC on 10.0.40.54 (added 2026-06-27)

Each additional NIC follows the same qm set <vmid> -netN virtio=<MAC>,bridge=vmbr0,tag=<VLAN> + reboot + ha network update enp0s<N> --ipv4-method auto pattern. Interface names increment: enp0s18, enp0s19, enp0s20.

4.3 qm guest exec Shell Escaping (CRITICAL)

qm guest exec <vmid> -- sh -c "..." passes through multiple shell layers (local SSH → remote qm → guest agent → sh -c), causing severe escaping issues with special characters.

Characters that break:

! — bash history expansion at every layer; sed 's/!/X/' produces literal x21 or gets eaten
$ — variable expansion
` — backtick command substitution
\ — inconsistent escaping across layers

HAOS guests have minimal tools: sed, awk, sh, curl, grep, ha CLI. NO python3, perl, busybox, or tcpdump.

Working approaches:

awk with sprintf — the most reliable for special char substitution:

# Produces literal ! via sprintf("%c", 33) without shell interpretation
qm guest exec 106 -- sh -c "awk '{gsub(/x21/, sprintf(\"%c\",33)); print}' /path/file > /tmp/fixed && mv /tmp/fixed /path/file"

Base64 round-trip — for whole-file replacement: read file via cat, modify locally, write back via echo BASE64 | base64 -d. But ! in decoded content can still get eaten if passed through echo.
SCP — if the guest runs sshd (HAOS doesn't by default), bypass qm guest exec entirely.

HAOS redacts secrets in output: grep/cat on files containing credentials returns *** instead of actual values. Cannot verify password patches via grep — must test functionality instead.

4.4 CT Start

curl -k -X POST https://10.0.20.10:8006/api2/json/nodes/proxmox1/lxc/CTID/status/start \
  -H "Cookie: PVEAuthCookie=TICKET" -H "CSRFPreventionToken: TOKEN"

Verify: status should be "running", not "stopped".

Section 5: NUT/UPS Power Management for Proxmox Clusters

Configure controlled shutdown on power loss: all Proxmox nodes stop their guests gracefully, then shut down. On power return, guests auto-start via onboot=1.

5.1 Architecture

APC UPS (USB) → NUT Master (10.0.30.100) → upsd :3493
                        ↓ upssched (FSD after 30s on battery)
                   ┌────┴────────────────┐
            NUT Slaves (all Proxmox nodes)
            upsmon receives FSD → runs shutdown script
            → qm shutdown all VMs → pct shutdown all CTs → shutdown -h now

NUT Master: Host with USB-connected UPS. Runs upsd + upsmon (master) + upssched.
NUT Slaves: All Proxmox nodes. Run upsmon (slave). On FSD, execute guest-shutdown script.
Shutdown order: Slaves shut down first (guests + node), master shuts down last.
Restart: Proxmox boots → onboot=1 guests start automatically.

5.2 Setting onboot=1 for Auto-Restart (PITFALL)

⚠️ qm set and pct set operate on LOCAL config files. Running them from one node for guests on other nodes fails with "Configuration file 'nodes/X/qemu-server/Y.conf' does not exist".

Correct approach — SSH to EACH node separately:

for ip in 10.0.20.{10,20,30,40,50,60,70} 10.0.20.91; do
  ssh -i ~/.ssh/id_ed25519_proxmox root@$ip "
    # Set onboot=1 for all running VMs
    for vmid in \$(qm list --running 2>/dev/null | awk 'NR>1{print \$1}'); do
      qm set \$vmid -onboot 1
    done
    # Set onboot=1 for all running CTs
    for ctid in \$(pct list --running 2>/dev/null | awk 'NR>1{print \$1}'); do
      pct set \$ctid -onboot 1
    done
  "
done

Verify across all nodes:

for ip in 10.0.20.{10,20,30,40,50,60,70} 10.0.20.91; do
  ssh -i ~/.ssh/id_ed25519_proxmox root@$ip "
    for ct in \$(pct list 2>/dev/null | awk 'NR>1{print \$1}'); do
      echo \"CT \$ct: \$(grep '^onboot' /etc/pve/lxc/\${ct}.conf 2>/dev/null || echo 'not-set')\"
    done
    for vm in \$(qm list 2>/dev/null | awk 'NR>1{print \$1}'); do
      echo \"VM \$vm: \$(grep '^onboot' /etc/pve/qemu-server/\${vm}.conf 2>/dev/null || echo 'not-set')\"
    done
  "
done

5.3 Deploying NUT Clients to Proxmox Nodes

Install nut-client and configure upsmon as slave on each Proxmox node:

# Prepare config files locally, then SCP + install on each node
# See templates/nut-slave-upsmon.conf and templates/ups-shutdown.sh

for ip in 10.0.20.{10,20,30,40,50,60,70} 10.0.20.91; do
  scp -i ~/.ssh/id_ed25519_proxmox \
    templates/nut-slave-upsmon.conf templates/ups-shutdown.sh \
    root@$ip:/tmp/

  ssh -i ~/.ssh/id_ed25519_proxmox root@$ip '
    export DEBIAN_FRONTEND=noninteractive
    apt-get install -y -qq nut-client
    cp /tmp/nut-slave-upsmon.conf /etc/nut/upsmon.conf
    cp /tmp/ups-shutdown.sh /usr/local/bin/ups-shutdown.sh
    chmod 755 /usr/local/bin/ups-shutdown.sh
    echo "MODE=netclient" > /etc/nut/nut.conf
    systemctl enable nut-monitor
    systemctl restart nut-monitor
    # Verify
    systemctl is-active nut-monitor
    upsc apc@NUT_MASTER_IP 2>&1 | head -2
  '
done

5.4 NUT Master Configuration

On the UPS-connected host (e.g. 10.0.30.100):

/etc/nut/nut.conf: MODE=netserver
/etc/nut/ups.conf: Driver section matching USB device (see templates/nut-master-ups.conf)
/etc/nut/upsd.conf: LISTEN 0.0.0.0 3493
/etc/nut/upsd.users: Both master and slave users (see templates/nut-master-upsd-users.conf)
/etc/nut/upsmon.conf: Master mode with NOTIFYCMD /usr/sbin/upssched (see templates/nut-master-upsmon.conf)
/etc/nut/upssched.conf: Timed FSD trigger (see templates/nut-upssched.conf)
/etc/nut/upssched-cmd: Script that runs upsmon -c fsd when timer fires (see templates/upssched-cmd.sh)

Key services: nut-driver.target, nut-server, nut-monitor — all must be active.

5.5 Shutdown Script (Proxmox Nodes)

The SHUTDOWNCMD in upsmon.conf points to /usr/local/bin/ups-shutdown.sh:

#!/bin/bash
# Stop all running VMs and CTs gracefully, then power off node
for vmid in $(qm list --running 2>/dev/null | awk 'NR>1{print $1}'); do
    qm shutdown "$vmid" --timeout 60 --forceStop 1 &
done
for ctid in $(pct list --running 2>/dev/null | awk 'NR>1{print $1}'); do
    pct shutdown "$ctid" --timeout 60 --forceStop 1 &
done
wait
sleep 3
shutdown -h now

5.6 Pitfalls

Heredoc escaping through SSH fails. Don't try to cat << 'EOF' nested configs through SSH. Write files locally, SCP them, then move into place. Alternatively, use base64 encoding.
NOTIFYTYPE is not a valid upsmon.conf directive. Use individual NOTIFYFLAG lines instead. LOWBATTERY → LOWBATT (NUT 2.8.x naming).
upssched PIPEFN/LOCKFN need /run/nut/ to exist. mkdir -p /run/nut && chown nut:nut /run/nut before starting.
Battery age matters. If the UPS battery is degraded (< 2 min runtime), the Low-Battery flag fires immediately, bypassing the upssched grace period. Replace batteries older than 3-4 years.
Store NUT credentials in 1Password. Create separate items for master and slave users.

5.7 Verification

# From each Proxmox node:
upsc apc@NUT_MASTER_IP 2>&1 | head -3    # Should show battery.charge

# On NUT master:
systemctl is-active nut-server nut-monitor
upsc apc@localhost 2>&1 | head -5

# Verify shutdown script syntax:
bash -n /usr/local/bin/ups-shutdown.sh

5.8 Battery Runtime Calibration via `upsrw`

When a UPS battery is replaced but the firmware retains old calibration data, battery.runtime may show implausibly low values (e.g. 60s for a fresh battery) and ups.status may show OL LB (Online + Low Battery) despite 100% charge.

Updating battery date resets the firmware's age-based runtime estimate:

# CRITICAL: -s FLAG MUST COME BEFORE -u/-p (otherwise silent failure)
upsrw -s battery.mfr.date=YYYY/MM/DD -u upsadmin -p "$NUT_PASS" apc@localhost

Adjusting battery.runtime.low (threshold at which LB flag triggers):

upsrw -s battery.runtime.low=30 -u upsadmin -p "$NUT_PASS" apc@localhost

⚠️ PITFALL: upsrw flag ordering. upsrw -u user -p pass -s VAR=VALUE apc@host FAILS silently with "Unexpected response from upsd: ERR PASSWORD-REQUIRED". The -s VAR=VALUE flag MUST precede -u/-p in the argument list.

Verification after calibration:

upsc apc@localhost 2>&1 | grep -E 'battery.mfr.date|battery.runtime|ups.status'
# Expected: runtime jumps from ~60s to ~285s, status changes from "OL LB" to "OL"

Note: Some UPS firmware (e.g. APC Back-UPS ES 700G) caches battery.mfr.date and may not persist the change across power cycles. The runtime recalculation triggered by the date update is the important part — it forces the firmware to recompute its degradation curve.

Section 6: Ceph OSD Management & Disk Integration

6.1 Cluster Overview Commands

ceph -s                    # Health, services, data summary
ceph osd tree              # Topology: hosts, OSDs, weights, status
ceph osd df                # Per-OSD usage
ceph df                    # Pool-level + raw storage summary
ceph osd pool ls detail    # Pool configs (replication, CRUSH rules, PGs)
pvesm status               # Proxmox storage view

6.2 Adding New OSDs to a Node (e.g., n5pro)

When integrating recycled disks from a decommissioned server into a new Proxmox node:

# On the target node (via SSH from another node or directly):
# 1. Identify available disks
lsblk -o NAME,SIZE,TYPE,FSTYPE

# 2. Zap any existing filesystem/partitions (DESTROYES data!)
ceph-volume lvm zap /dev/sdX --destroy

# 3. Create OSD via Proxmox GUI or CLI
pveceph osd create /dev/sdX

# 4. Verify it joined the cluster
ceph osd tree | grep $(hostname)

Prerequisites on the target node:

Proxmox VE installed and joined to cluster
Ceph installed via pveceph install
Ceph monitor may or may not be needed (depends on cluster redundancy)
Disks must be wiped (no ZFS, LVM, or partitions remaining)

6.3 Decommissioning OSDs (Data Redistribution)

Before removing OSDs from a node being decommissioned:

# 1. Mark OSDs as OUT (triggers data redistribution to other OSDs)
ceph osd out <osd-id>
# e.g., ceph osd out 8 9

# 2. Monitor rebalance progress (WAIT for completion before destroying)
ceph -w    # Watch for "recovery" to complete, PGs back to active+clean
ceph osd df  # Verify remaining OSDs absorbed the data

# 3. Once safe, destroy and purge
ceph osd destroy <osd-id> --yes-i-really-mean-it
ceph osd purge <osd-id> --yes-i-really-mean-it

# 4. Remove the monitor if the node hosted one
systemctl stop ceph-mon@$(hostname)
ceph mon remove $(hostname)

Capacity check before draining: Ensure remaining cluster has enough free space for redistributed data.

ceph df  # Check TOTAL AVAIL vs. amount on OSDs being removed

6.4 Removing a Node from the Cluster

# 1. Decommission all OSDs on the node (Section 6.3)
# 2. Remove Ceph monitor (if present)
# 3. Remove from Proxmox cluster
pvecm delnode <hostname>
# 4. If node is permanently gone and unreachable:
pvecm delnode <hostname> -force

6.5 Ceph Health Warnings to Investigate Before Adding Load

Before migrating data INTO a Ceph cluster, check for existing health issues:

ceph health detail

Common warnings:

Slow OSD operations — disk performance degradation, check ceph osd perf
Stalled BlueFS reads — db device (WAL/db partition) failing
Low space on a node — uneven distribution, may need CRUSH adjustment
Destroyed OSDs still in tree — osd.6 destroyed entries reduce apparent capacity, clean up with ceph osd purge
MON_DISK_LOW — Mon node root disk <20% free, can cause Mon flapping. Clean up disk space or remove excess Mons.

6.5.1 Ceph Mon Reduction (Over-provisioned Mons)

Running too many Mons (7+) increases Paxos consensus latency on every map update. Recommended: 3 for clusters up to 10 nodes, 5 for larger clusters. Mons on non-PVE nodes (e.g. standalone Ubuntu) add no value and should be removed first.

Procedure (no downtime, live):

# Check current Mons
ceph mon dump

# Remove excess Mons one at a time
ceph mon remove <name>    # e.g. ceph mon remove ubuntu

# Verify quorum after each removal
ceph -s | grep "mon:"

# Target: 3 Mons on well-distributed physical nodes
# Good placement: one Mon per rack/room, avoid co-location with OSD-heavy nodes if possible

Mon selection criteria for a 3-Mon cluster:

Spread across different physical hardware (not all on same model)
Prefer nodes with SSD root disks (Mons write leveldb/rocksdb)
Avoid nodes that are being decommissioned
At least one Mon on a stable, high-uptime node

Removing a Mon from a node (cleanup):

# Stop the mon service on the node
ssh root@<node> 'systemctl stop ceph-mon@<hostname>'
ssh root@<node> 'systemctl disable ceph-mon@<hostname>'
# The mon data dir can be left in place harmlessly

6.5.2 Ceph OSD Purge (Destroyed OSD Cleanup)

Destroyed OSDs that remain in the CRUSH tree cause confusion and reduce apparent capacity. Purge them fully:

# Check for destroyed OSDs
ceph osd tree | grep destroyed

# Purge (removes from CRUSH + OSD map)
ceph osd purge <osd-id> --yes-i-really-mean-it

# Verify removal
ceph osd tree | grep osd.<id>  # should return nothing

This triggers PG rebalancing — monitor progress with ceph -w or ceph -s | grep pgs. Recovery rate depends on osd_recovery_sleep (default 0.1 = throttled to ~10% speed). For urgent recoveries, temporarily set ceph config set osd osd_recovery_sleep 0.

6.6 Schön Homelab Ceph State (as of 2026-07-03)

Property	Value
Cluster ID	204c8171-e0b1-4f40-9de2-a7cfe4ef68d9
Health	HEALTH_WARN (2 OSDs slow BlueStore ops: osd.0, osd.1)
Total Raw	12 TB (11 HDD + 1.1 SSD)
Used	5.1 TB (42%)
Available	7.0 TB
OSDs	10 up, 10 in (osd.6 purged 2026-07-03)
Mons	3 (proxmox5, proxmox7, proxmox4) — reduced from 7 on 2026-07-03
Pools	7 (cephfs_data, cephfs_metadata, vm_disks, .mgr, rbd, hdd_disk, tm_disks)

Mon reduction (2026-07-03): Removed 4 Mons (ubuntu, proxmox1, proxmox2, proxmox3). mon.ubuntu was on 10.0.30.100 — not even a PVE cluster member. Keeping 3 Mons spread across different physical nodes. Procedure: ceph mon remove <name> per Mon, no downtime.

OSD distribution (post-purge):

proxmox1: osd.1 (HDD 3.6T, 38% full)
proxmox2: osd.0 (SSD 188G, 52%), osd.2 (SSD 233G, 57%)
proxmox3: osd.4 (SSD 233G, 67%)
proxmox4: osd.3 (SSD 238G, 67%)
proxmox5: osd.5 (SSD 238G, 61%)
proxmox6: osd.10 (HDD 982G, 82% — overfilled)
proxmox7: osd.7 (HDD 982G, 75% — overfilled)
ubuntu/10.0.30.100: osd.8 + osd.9 (HDD 2.7T each, ~28% — underutilized, planned for decommission)

OSD DB Optimization (2026-07-04): Both OSDs on 10.0.30.100 now have DB on SSD (real partitions, no loopback):

osd.8: DB on osd-8-db LV (30GB, sda4 PV) via ceph-volume lvm new-db — up
osd.9 (old): destroyed+purged — RocksDB was irreparably corrupt (MANIFEST corruption from loopback DB)
osd.6 (new, recycled ID): recreated on /dev/sdf with DB on osd-9-db LV (30GB, sda4 PV) via ceph-volume lvm create --block.db — up, backfill completed from peers (size=3, no data loss)
Swap partition (sda3) repurposed, root shrunk to 160G (automated initramfs script), sda4 (64GB LVM) holds both DB LVs
Post-shrink cleanup (2026-07-04): initramfs premount script + GRUB rescue entry removed, GRUB + initramfs regenerated, verified no shrink_root references remain

Pool-full incident (2026-07-04): 6 pools showed 100% full (MAX_AVAIL=0) despite HDD class having 6.3 TB free. Root cause: osd.10 (982GB) hit 95% full_ratio, blocking all writes to PGs on that OSD. Fixed by raising full_ratio to 0.97 + reweighting osd.10→0.5, osd.7→0.8. All pools recovered to 68-94% with 75-181 GB avail. See Section 6.12 for diagnosis/fix procedure.

Known issues:

OSD imbalance: VAR 0.20-2.13, small HDDs (osd.7/10 ~1TB) overfilled vs large HDDs (osd.6/8 ~2.8TB) underused — reweighted 2026-07-04, long-term fix = add larger OSDs
tm_disks pool: replica=2 (no redundancy — data loss risk if one OSD fails)
CephFS: 0 clients, 0 data — MDS daemons running but unused
target_size_ratio=0.9 on 3 pools (cephfs_data, cephfs_metadata, vm_disks) — misconfigured, causes PG over-allocation

Pending optimizations (not yet executed):

OSD reweight to balance fill levels
tm_disks: increase to replica=3 or decommission pool
osd_memory_target: reduce from 4GB to 2-3GB on 15GB-RAM nodes
target_size_ratio: set realistic values per pool
CephFS: shut down if unused, or document purpose
Integrate 4× 3TB HDDs from 10.0.30.100 ZFS pool into Ceph (RAIDZ2, see Section 6.9)

See references/cluster-audit-2026-07.md for full audit findings and recommendations.

6.7 Schön Homelab PVE Version State (Post-Update 2026-07-02)

All 8 nodes successfully upgraded to PVE 9.2.3 via rolling update:

Node	PVE	Kernel	Guests after migration
proxmox1	9.2.3	6.17.4-2-pve	CT 105, VM 106 (homeassistant), CT 140, CT 231
proxmox2	9.2.3	7.0.2-6-pve	VM 300 (mariadb-01), VM 311 (maxscale-02), CT 133
proxmox3	9.2.3	7.0.2-6-pve	CT 121, CT 200, VM 129-131 (stopped)
proxmox4	9.2.3	7.0.12-1-pve	VM 302 (mariadb-03), VM 310 (maxscale-01), CT 135 (openwebui)
proxmox5	9.2.3	7.0.12-1-pve	CT 114 (smb-tm-noris, running — Time Machine target)
proxmox6	9.2.3	7.0.12-1-pve	VM 301 (mariadb-02), CT 108-109, CT 112-113, CT 116-117, CT 120 (frigate), CT 134
proxmox7	9.2.3	7.0.12-1-pve	CT 100, CT 124 (litellm), CT 230 (hermes-agent), VM 99999 (traefik)
n5pro	9.2.3	7.0.12-1-pve	CT 104 (paperless-ngx)

Kernel variance is normal — proxmox1-3 had no kernel change in the 9.2.2→9.2.3 update path. proxmox4-7 and n5pro jumped from 7.0.0 to 7.0.12.

HA activation status (updated 2026-07-04): 19 HA-managed guests. CT 137 (imap), CT 114 (smb-tm), CT 136 (smb-media) all added with max_relocate=3/max_restart=2. See references/cluster-audit-2026-07.md for full session detail.

6.8 LXC Live Migration Pitfall

Proxmox does NOT support LXC live migration — this is a hard limitation in all supported versions (9.1.x, 9.2.x). Attempting pvesh create /nodes/{node}/lxc/{vmid}/migrate --target {target} --online 1 fails with:

migration aborted: lxc live migration is currently not implemented

Workarounds:

In-place update + reboot (recommended): apt full-upgrade updates host packages while CTs keep running. Reboot restarts CTs on the same node. This is the standard Proxmox approach.
Backup/restore: vzdump CT, restore on another node with vzrestore — slow for large disk images but zero downtime during backup.
Clone: pct clone <ctid> <newid> --storage <storage> --target <node> — creates a copy on the target, then swap IPs/config.

Section 6.9: ZFS-to-Ceph Disk Migration Strategy

6.9.1 RAIDZ Constraint: Cannot Shrink (CRITICAL)

ZFS RAIDZ does NOT support zpool remove for individual disks. The number of disks in a RAIDZ vdev is fixed at creation. Options:

zpool offline <disk> — Takes a disk offline (pool stays alive, reads from parity), frees the physical disk for Ceph. Pool becomes degraded but remains usable.
zpool destroy — Destroys the entire pool, freeing ALL disks simultaneously. Data must be migrated elsewhere first.

There is NO middle ground — you cannot remove disks one-by-one from a RAIDZ vdev while keeping the pool at reduced capacity. zpool remove only works for mirrors and special/log/cache vdevs, NOT RAIDZ.

6.9.2 Phased Migration: Degrade ZFS, Grow Ceph Simultaneously

Strategy when Ceph has enough free space to absorb ZFS data:

Phase	Action	ZFS Disks Active	ZFS Parity Left	Ceph OSDs Added
1	`zpool offline pool <oldest-disk>` → wipe → create Ceph OSD	N-1	RAIDZ2: 1 left	+1
2	Migrate ZFS data → Ceph (RBD/CephFS)	N-1 (shrinking usage)	1	1
3	`zpool destroy` → wipe remaining → create Ceph OSDs	0	—	+(remaining)

Safety constraint: Only offline 1 disk at a time from RAIDZ2 (keeps 1 parity). Offlining 2 disks from RAIDZ2 leaves ZERO redundancy — one sector error during migration = potential data loss.

Capacity pre-check: Verify Ceph has enough free space for ALL ZFS data before starting:

# On Ceph admin node:
ceph df | head -5    # Check AVAIL space
# On ZFS host:
zfs list -o name,used,avail <pool>   # Check used space
# Rule: Ceph AVAIL > ZFS USED

6.9.3 Disk Health Assessment Before Migration (SMART + badblocks)

Before committing any disk to Ceph, run both SMART checks AND destructive testing:

SMART quick-check (all disks at once):

for d in sda sdb sdc sdd sde sdf sdg sdh sdi; do
  echo "=== /dev/$d ==="
  smartctl -a /dev/$d 2>/dev/null | grep -E \
    "Device Model|Serial|Power_On_Hours|Reallocated_Sector|Current_Pending|Offline_Uncorrectable|UDMA_CRC|overall-health|Self-test execution" | head -12
done

Key SMART thresholds for Ceph suitability:

Attribute	Good	Caution	Reject
Reallocated_Sector_Ct (5)	0	1-10	>10
Current_Pending_Sector (197)	0	1-50	>50
Offline_Uncorrectable (198)	0	1-50	>50
UDMA_CRC_Error_Count (199)	0	1-10 (cable)	>10 (controller)
Power_On_Hours (9)	<50k	50k-90k	>90k
Self-test status	(0) Passed	—	(121) Read failure

SMART Self-test (short, 2 min):

smartctl -t short /dev/sdX
sleep 130
smartctl -a /dev/sdX | grep -A3 "Self-test"
# Watch for: "Completed: read failure" = FAIL
# "Completed without error" = PASS

Destructive test (badblocks, 9-22h per 3TB):

# DESTROYS ALL DATA on the disk — only run on disks that will be repurposed!
wipefs -a /dev/sdX
badblocks -wsv -b 4096 /dev/sdX 2>&1 | tee /tmp/badblocks_sdX.log &
# -w = write mode (4 patterns: 0xaa, 0x55, 0xff, 0x00)
# -s = show progress
# -v = verbose (prints errors)
# Estimated time: ~5h per 3TB per pattern × 4 = ~20h total
# Check progress: tail -1 /tmp/badblocks_sdX.log
# Kill early if needed: kill $(pgrep badblocks)

Decision matrix after testing:

Result	Ceph Suitable?	Action
SMART clean + badblocks 0 errors	✅ Yes	Proceed as OSD
SMART clean + badblocks finds errors	⚠️ Marginal	Reallocation may fix sectors. Risk acceptable for non-critical pools.
SMART fail (pending/uncorrectable >50)	❌ No	Retire disk
SMART Self-test read failure	❌ No	Even with few pending sectors, confirmed read failures indicate progressive degradation

6.9.4 Schön Homelab 10.0.30.100 Disk Inventory (2026-07-03)

Actual disk layout (corrected from earlier "9×2.7TB" assumption):

Disk	Model	Serial	Size	Hours	Age	SMART	Role	Ceph Ready?
sda	Crucial MX300 SSD	164914EEB886	275GB	81,871	9.3y	✅	Boot SSD	—
sdb	WD30EFRX Red+	WCC4N4VRJ5UU	3TB	81,786	9.3y	✅	ZFS pool	⚠️ Old but healthy
sdc	WD30EZRX Green	WMC1T2256499	3TB	89,262	10.2y	⛔ 371 pending, 243 uncorr	Free	❌ Defective
sdd	WD30EFRX Red+	WCC4N4SVR8JL	3TB	57,925	6.6y	⚠️ 16 pending, 15 uncorr, Self-test FAIL	Free	❌ Failed
sde	WD30EFRX Red+	WMC4N2399595	3TB	104,327	11.9y	✅	Ceph osd.8	⚠️ Very old
sdf	WD30EFRX Red+	WMC4N2400243	3TB	98,475	11.2y	✅	Ceph osd.9	⚠️ Very old
sdg	WD30EFZX Purple	WX12DA01FUA7	3TB	30,901	3.5y	✅	ZFS pool	✅ Young, healthy
sdh	WD30EFZX Purple	WX12DA0R1NFH	3TB	30,901	3.5y	✅	ZFS pool	✅ Young, healthy
sdi	WD30EFZX Purple	WX22DA0D7EP3	3TB	30,900	3.5y	✅	ZFS pool	✅ Young, healthy

ZFS Pool pool01_n2_redundant: RAIDZ2, 4 disks (sdg, sdh, sdi, sdb), 10.9TB raw, 3.63TB used, 1.87TB free, 57% fragmented. Monthly scrubs clean. Contains Docker volumes, TimeMachine (~951GB), proxmox_nfs (~963GB), HomeAssistant.

Conclusion: Neither sdc (defective) nor sdd (self-test failure) are suitable for Ceph. The 4 healthy ZFS pool disks can only be released by destroying the pool. Recommended strategy: offline sdb (oldest, 9.3y) first as initial Ceph OSD, migrate ZFS data to Ceph, then destroy pool and claim remaining 3 disks.

6.9.5 Ceph OSD Creation on Recycled Disks

Once a disk is physically available (offline from ZFS or freshly inserted):

# On the target PVE node (e.g. n5pro):
# 1. Wipe all signatures
wipefs -a /dev/sdX
# If disk was ZFS, also zap partitions
sgdisk --zap-all /dev/sdX 2>/dev/null || true

# 2. Create OSD (PVE CLI)
pveceph osd create /dev/sdX
# OR via ceph-volume directly:
ceph-volume lvm create --data /dev/sdX

# 3. Verify
ceph osd tree | grep $(hostname)
ceph osd df | head -5

Prerequisite: Ceph must be installed on the target node (pveceph install). The node must be a PVE cluster member.

6.9.6 SSH Access to 10.0.30.100

# Works with Proxmox SSH key (NOT the Galera key):
ssh -i ~/.ssh/id_ed25519_proxmox -o StrictHostKeyChecking=no root@10.0.30.100 '<commands>'

The Galera SSH key from 1Password does NOT work for 10.0.30.100 — only the Proxmox id_ed25519_proxmox key grants root access.

6.10 Adding BlueStore DB/WAL Device to Existing OSD

When an HDD-OSD was created without a separate DB (block.db) device, all RocksDB metadata lives on the slow HDD. Adding a flash DB device post-creation dramatically speeds up metadata operations.

⚠️ ceph-bluestore-tool --command bluefs-bdev-new-db DOES NOT WORK on Ceph Squid (19.2.x). It fails with "No valid bdev label found" regardless of whether the target is a raw partition or LVM LV. The correct tool is ceph-volume lvm new-db.

Prerequisites

A free flash device or partition (SSD/NVMe) on the OSD host
The flash device must be set up as LVM (PV → VG → LV). Raw partitions do NOT work with ceph-volume lvm new-db.
OSD must be stopped during the operation (brief downtime, ~1-2 min)

DB Sizing Guidance

DB Size	Use Case	Notes
10 GB	Minimum	⚠️ Spill risk with heavy metadata
20-30 GB	Practical	✅ Sufficient for normal workloads
50 GB	Conservative	What osd.7/osd.10 use on proxmox6/7
~4% of OSD size	Formula	Ensures entire RocksDB fits on flash

Procedure (Confirmed Working on Ceph 19.2.3)

# 1. Identify the OSD FSID
OSD_FSID=$(cat /var/lib/ceph/osd/ceph-<OSD_ID>/fsid)
echo "OSD FSID: $OSD_FSID"

# 2. Create LVM on the flash device (e.g. former swap partition sda3)
pvcreate /dev/sda3
vgcreate ceph-db-vg /dev/sda3
lvcreate -L 30G -n osd-<OSD_ID>-db ceph-db-vg

# 3. Stop the OSD
systemctl stop ceph-osd@<OSD_ID>

# 4. Attach DB volume (THE KEY COMMAND)
ceph-volume lvm new-db \
  --osd-id <OSD_ID> \
  --osd-fsid "$OSD_FSID" \
  --target ceph-db-vg/osd-<OSD_ID>-db

# 5. Start the OSD
systemctl start ceph-osd@<OSD_ID>

# 6. Verify (wait ~15s for OSD to come up)
sleep 15
ceph osd tree | grep osd.<OSD_ID>    # Should show "up"
ls -la /var/lib/ceph/osd/ceph-<OSD_ID>/block.db  # Should symlink to /dev/dm-*

Post-Addition Behavior

BlueFS spillover is EXPECTED: Existing RocksDB SST files remain on the HDD. New writes go to the flash DB. Over time, compaction migrates data to the flash device. The BLUEFS_SPILLOVER warning clears naturally (hours to days).
OSD may show "down" briefly after restart — it takes ~10-15s for BlueFS to initialize with the new DB device and for the OSD to report "up" to the monitors.
ceph-bluestore-tool commands fail while OSD is running — they cannot lock the fsid file (Device or resource busy). This is expected; use ceph osd metadata <id> from a monitor node instead to verify DB device assignment.

Verification from Monitor Node

# Check if DB device is recognized cluster-wide
ceph osd metadata <OSD_ID> | python3 -c "
import sys, json
d = json.load(sys.stdin)
for k in ['hostname', 'rotational', 'bluefs_db_dev', 'devices']:
    print(f'  {k} = {d.get(k, \"NOT SET\")}')
"
# devices field should list BOTH the HDD and the flash device (e.g. "sda,sde")
# bluefs_db_dev may show "NOT SET" initially — metadata propagates slowly

User Preference: NO Loopback Files for DB/WAL (CRITICAL)

The user explicitly rejects loopback files (losetup + fallocate) for Ceph DB/WAL devices. Even though loopback-on-SSD is technically functional and reboot-safe (via systemd unit), the user considers it a hack. Always propose real partitions: repurpose swap, shrink root partition, or add a physical SSD/NVMe.

DB Device Migration: Loopback → Real Partition (PITFALL — RocksDB Corruption)

When migrating an OSD's DB device from loopback to a real partition, the existing RocksDB must be readable. ceph-volume lvm new-db reads the existing DB to shard it onto the new device. If the RocksDB is corrupt, the migration fails AND leaves the OSD in an unrecoverable state.

Failure sequence observed (10.0.30.100, osd.9, 2026-07-03):

osd.9 stopped, block.db symlink removed
ceph-volume lvm new-db --target ceph-db-vg/osd-9-db-new → FAILED: Corruption: bad record length in file db/MANIFEST-060205
Loopback file deleted + loop device detached (cleanup)
Old block.db symlink restored → but loopback file gone → OSD can't open DB at all
ceph-bluestore-tool --command repair → FAILED (same MANIFEST corruption)
OSD permanently down — must be destroyed and rebuilt

Root cause: The RocksDB on the loopback device was already corrupt (likely from the initial compaction when the DB device was first added). The corruption was silent — the OSD was running but with degraded metadata.

Safe migration procedure (PREVENTS corruption loss):

# 1. STOP OSD
systemctl stop ceph-osd@<OSD_ID>

# 2. CHECK existing RocksDB health BEFORE touching anything
ceph-bluestore-tool --command bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-<OSD_ID> 2>&1
# If this succeeds, RocksDB is readable. If it errors, DO NOT proceed — the OSD
# must be destroyed and rebuilt instead of migrated.

# 3. Attach new DB device (KEEP old device intact!)
ceph-volume lvm new-db --osd-id <OSD_ID> --osd-fsid <FSID> --target <NEW_LV>

# 4. ONLY if step 3 succeeds: start OSD and verify
systemctl start ceph-osd@<OSD_ID>
sleep 15
ceph osd tree | grep osd.<OSD_ID>  # Must show "up"

# 5. ONLY after OSD is confirmed up: clean up old DB device
# (Remove loopback file, detach loop device, remove old LVM, disable systemd service)

⚠️ CRITICAL: Never delete the old DB device (loopback file, partition, LV) until the new DB is successfully attached AND the OSD is confirmed "up". Deleting the old DB before confirming the new one works leaves the OSD with no readable RocksDB — the only recovery is ceph osd destroy + rebuild from peer replicas.

When RocksDB is already corrupt (recovery procedure):

If ceph-bluestore-tool --command repair fails with MANIFEST corruption, the OSD is unrecoverable. Data is safe (Ceph size=3 replication), but the OSD must be destroyed and rebuilt:

# From a monitor node (e.g. 10.0.20.91):
CEPH="ceph --conf /etc/pve/ceph.conf --keyring /etc/pve/priv/ceph.client.admin.keyring"

# 1. Destroy the OSD (marks as destroyed, triggers backfill from peers)
#    --force is REQUIRED when OSD is down and PGs are not active+clean
#    (without --force: "Error EAGAIN: OSD(s) have no reported stats")
$CEPH osd destroy <OSD_ID> --force

# 2. On the OSD host: wipe the disk and recreate OSD with DB on real partition
ceph-volume lvm zap /dev/sdX --destroy

# 3. Create new OSD with DB on real partition
#    ⚠️ Flag is --block.db, NOT --db-dev (which is unrecognized)!
ceph-volume lvm create --data /dev/sdX --block.db /dev/ceph-db-vg/osd-<OSD_ID>-db

# 4. Purge the OLD destroyed OSD from CRUSH (also needs --force when OSD is down)
$CEPH osd purge <OLD_OSD_ID> --force

# 5. Verify new OSD joins cluster (may get a RECYCLED OSD ID — this is normal!)
$CEPH osd tree | grep osd.<NEW_ID>
# Backfill runs automatically — monitor with: $CEPH -w

⚠️ OSD ID Recycling: When you destroy an OSD and create a new one on the same disk, the new OSD may receive a different (recycled) ID — e.g. destroying osd.9 and recreating gave osd.6. This is normal Ceph behavior. The OSD ID is just a number; what matters is the disk and DB assignment. Always purge the old destroyed OSD ID from CRUSH after the new one is up.

⚠️ ceph-volume lvm create flag pitfall: The flag for specifying the DB device is --block.db, NOT --db-dev. Using --db-dev produces a cryptic error: unrecognized arguments: --db-dev. This differs from ceph-volume lvm new-db which uses --target.

⚠️ ceph osd destroy requires --force when OSD is down: Without --force, you get Error EAGAIN: OSD(s) have no reported stats, and not all PGs are active+clean. Since the OSD is down and can't report stats, --force is the only way forward. Same applies to ceph osd purge.

⚠️ rbd lock remove syntax (not lock break): To manually remove an RBD exclusive lock: rbd --id admin lock remove <pool/image> <locker_id> <cookie>. Parse the locker ID and cookie from rbd lock list --format json. However, ceph osd blacklist add <client_addr> is usually more effective — it forces the kernel RBD driver to error out, which releases any D-state processes holding the device.

Post-Shrink udev Issue: New Partition Not Visible After Boot

After an automated root partition shrink (initramfs script), the newly created partition (e.g. sda4) may appear in the partition table (fdisk -l) but NOT as a block device (ls /dev/sda4). This is a udev timing issue — the partition was created late in the boot process and udev hadn't processed it yet.

Fix (immediate, remote via SSH):

partprobe /dev/sda
udevadm trigger --subsystem-match=block
sleep 2
ls -la /dev/sda4  # Now visible

This should be added to the post-reboot verification checklist after any automated shrink.

Post-Shrink Cleanup: Remove Temporary Boot Infrastructure

After the automated shrink has been executed and verified (sda4 exists, root is 160G), the temporary boot infrastructure must be removed to prevent accidental re-triggering on future boots:

# 1. Remove initramfs premount script
rm /etc/initramfs-tools/scripts/init-premount/auto-shrink-root

# 2. Remove initramfs hook (if only used for shrink)
rm /etc/initramfs-tools/hooks/resize_tools

# 3. Remove GRUB rescue entry
rm /etc/grub.d/40_custom_rescue

# 4. Regenerate GRUB + initramfs (removes shrink_root from menu, cleans initramfs)
update-grub
update-initramfs -u -k all

# 5. Verify no shrink references remain
grep -r "shrink_root" /boot/grub/grub.cfg || echo "✓ clean"
grep -rl "shrink" /etc/initramfs-tools/scripts/ || echo "✓ clean"

# 6. Restore GRUB timeout to normal (if increased for debugging)
sed -i 's/GRUB_TIMEOUT=10/GRUB_TIMEOUT=2/' /etc/default/grub
update-grub

⚠️ Do NOT remove the GPT backup (/etc/initramfs-tools/gpt_backup.bin) — it's harmless and could be useful for future rollback. Only remove the scripts and GRUB entries that could accidentally trigger a shrink.

The partition changes (sda2=160G, sda4=LVM) are persistent and survive reboot — only the automation scripts are temporary.

If a loopback DB was already set up (mistake), tear it down and migrate to a real partition:

# 1. Stop OSD
systemctl stop ceph-osd@<OSD_ID>
# 2. Detach the loopback DB (remove symlink, ceph-volume doesn't have a clean detach)
rm /var/lib/ceph/osd/ceph-<OSD_ID>/block.db
# 3. Remove LVM structures
lvremove -y ceph-db-vg9/osd-<OSD_ID>-db
vgremove ceph-db-vg9
pvremove /dev/loop10
# 4. Detach loop device
losetup -d /dev/loop10
rm /var/lib/ceph/osd-9-db.img
# 5. Disable systemd service
systemctl disable ceph-loopback-db.service
rm /etc/systemd/system/ceph-loopback-db.service
rm -rf /etc/systemd/system/ceph-osd@<OSD_ID>.service.d
systemctl daemon-reload
# 6. Create real partition DB (see below) and start OSD

Getting Space for DB Partitions on Systems Without Free Flash

When a host has only one SSD (used for OS) and no free flash slots, space for DB partitions must be carved from the existing SSD:

Option A: Repurpose swap partition (immediate, no reboot)

Most Proxmox/Ubuntu hosts have a 16-32GB swap partition that's barely used
swapoff /dev/sdX3 → wipefs -a → pvcreate → vgcreate → lvcreate -L 30G
Only enough space for 1 OSD's DB (30GB from 32GB swap)

Option B: Shrink ext4 root partition (requires reboot, RISKY without console)

ext4 can grow online but CANNOT shrink online — must be unmounted
Root partition must be shrunk from a rescue environment (initramfs shell or Live ISO)
⚠️ Without IPMI/KVM/physical access, a failed resize = permanent loss of the machine
Two approaches:
- Manual (console required): GRUB entry with break=premount → initramfs shell → run e2fsck + resize2fs + fdisk by hand → continue
- Automated (unattended, console recommended as fallback): GRUB entry with shrink_root=1 → initramfs premount script runs e2fsck → resize2fs → sgdisk automatically → continues boot. See references/initramfs-auto-shrink-template.md for complete templates, setup sequence, and 6 pitfalls learned during development.
Risk mitigation: Hard reset during any phase of the automated shrink is survivable — all intermediate states are bootable. grub-reboot (one-time) auto-reverts to normal boot. GPT backup embedded in initramfs for rollback.
Key pitfalls (see reference for full detail):
1. initramfs hook must . /usr/share/initramfs-tools/hook-functions for copy_exec
2. GPT backup must be copied INTO initramfs (root FS not mounted in premount)
3. GRUB entry must NOT have break=premount (would stop boot instead of running script)
4. Use grub-reboot (one-time) not grub-set-default (persistent) for rescue boot

Option C: Add a physical SSD/NVMe (recommended for production)

Small 256-512GB NVMe (~40€) provides 180-300GB for DB
Supports 6-10 OSDs at 30GB each
No risk to existing data, no downtime

Repurposing Swap Partition for DB (Schön Homelab)

On 10.0.30.100, the 32GB swap partition (sda3) was repurposed for osd.8's DB device:

Swap had only 1.5GB used, 6.4GB RAM free → safe to disable
swapoff /dev/sda3 → wipefs -a /dev/sda3 → LVM → ceph-volume lvm new-db
Only enough space for ONE OSD's DB (30GB LV). osd.9's DB was initially set up via loopback (rejected by user) — requires root partition shrink or additional SSD.

Journal Cleanup to Free Root Space

Before allocating root-SSD space for DB, free up wasted space:

journalctl --vacuum-size=500M    # Freed 3.5GB on 10.0.30.100
apt-get clean                     # Freed 322MB

References

references/lxc-gpu-device-bind-rules.md — Copy-paste config snippets per GPU vendor
references/lxc-gpu-autosetup-background-script.md — Background install pattern with completion markers
references/comfyui-model-directories.md — ComfyUI, Ollama, diffusers directory layout
references/cluster-audit-2026-07.md — Cluster audit findings
references/cluster-analysis-2026-07-12.md — July 12 analysis: n5pro, Ceph osd.10 full, K8s VM balloon, perf optimizations
references/ceph-pool-full-recovery-2026-07.md — Pool-full recovery + OSD prep
references/zfs-to-ceph-migration-2026-07.md — Disk inventory, SMART data
references/ceph-pg-osd-management-2026-07.md — PG reduction, OSD reweight, concurrent I/O
references/initramfs-auto-shrink-template.md — Automated ext4 root partition shrink via initramfs (premount script, hook, GRUB entry, setup sequence, 6 pitfalls)
scripts/lxc-background-package-install.sh — Background ROCm/ComfyUI install script
templates/nut-master-ups.conf — ups.conf driver config for APC Back-UPS ES 700G
templates/nut-slave-upsmon.conf — upsmon.conf template for Proxmox NUT slave nodes
templates/nut-master-upsmon.conf — upsmon.conf template for NUT master
templates/nut-master-upsd-users.conf — upsd.users with master+slave users
templates/nut-upssched.conf — upssched.conf for timed FSD trigger
templates/upssched-cmd.sh — upssched command script
templates/ups-shutdown.sh — Guest shutdown script for Proxmox nodes

Section 7: Ceph Erasure Coding Pools & LXC Integration

7.1 Creating an EC Pool for RBD Use

Erasure Coding (EC) pools provide storage efficiency (k+m chunks instead of full replicas) but require a replicated metadata pool for RBD. The EC pool stores data blocks; the metadata pool stores RBD object maps and journaling.

EC Profile Creation

CEPH="ceph --conf /etc/pve/ceph.conf --keyring /etc/pve/priv/ceph.client.admin.keyring"

# Create EC profile: k=4 data chunks, m=1 parity chunk, HDD-only, failure_domain=osd
$CEPH osd erasure-code-profile set ec41-hdd \
  plugin=jerasure \
  technique=reed_sol_van \
  k=4 m=1 \
  crush-failure-domain=osd \
  crush-device-class=hdd

Parameter	Meaning
`k=4`	4 data chunks (stripe width)
`m=1`	1 parity chunk (survives 1 OSD failure)
`crush-failure-domain=osd`	Chunks spread across OSDs (not hosts)
`crush-device-class=hdd`	Only HDD OSDs used (excludes SSD)

⚠️ k+m is FIXED at pool creation. You cannot change it later (e.g. 4+1 → 5+1) without creating a new pool and migrating data. Adding more OSDs later improves distribution but keeps the same k+m.

Choosing failure_domain: With only 4 hosts but 5+ OSDs, failure_domain=osd is required (need ≥ k+m = 5 failure domains). failure_domain=host would need ≥5 hosts.

Pool Creation

# EC data pool (stores actual object data)
$CEPH osd pool create media_ec 128 128 erasure ec41-hdd

# Metadata pool (replicated, stores RBD metadata)
$CEPH osd pool create media_meta 32 32 replicated

# Enable RBD application on both pools
$CEPH osd pool application enable media_ec rbd
$CEPH osd pool application enable media_meta rbd

# CRITICAL: Allow EC overwrites (required for RBD write support)
$CEPH osd pool set media_ec allow_ec_overwrites true

⚠️ allow_ec_overwrites true is mandatory for RBD use. Without it, RBD images on EC pools are read-only and container creation/mount fails.

7.2 Registering EC-Backed RBD Storage in PVE

pvesm add rbd media \
  --pool media_meta \
  --data-pool media_ec \
  --username admin \
  --monhost 10.0.20.50:6789,10.0.20.70:6789,10.0.20.40:6789

--pool = metadata pool (replicated)
--data-pool = EC pool (actual data)
PVE creates RBD images on the metadata pool; data blocks are stored on the EC pool automatically

Verify:

pvesm status --storage media
# Should show "active" with available space

7.3 Adding `rootdir` Content Type to RBD Storage (CRITICAL)

Newly created RBD storages default to images content type only. LXC mount points (mp0, mp1, ...) require rootdir content type. Without it, pct set -mp0 <storage>:<size> fails with:

storage 'media' does not allow content type 'rootdir' (Container)

Fix:

pvesm set media --content rootdir
# Verify:
grep -A8 "rbd: media" /etc/pve/storage.cfg
# Should show: content rootdir

7.4 Creating LXC CT with EC-Backed Mount Point

Root Disk: Must Use Non-EC Storage

EC-backed RBD storage typically only supports rootdir (for mount points) and images (for VMs). The container root filesystem needs a storage that supports rootdir — use an existing replicated RBD storage (e.g. vm_disks, hdd_disk):

pct create 136 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --arch amd64 \
  --hostname smb-media \
  --memory 512 --swap 0 --cores 1 \
  --rootfs vm_disks:8 \
  --net0 name=eth0,bridge=vmbr0,ip=10.0.30.30/24,gw=10.0.30.1,tag=30,type=veth \
  --unprivileged 1 --features nesting=1

Mount Point: EC-Backed RBD

pct set 136 -mp0 media:500,mp=/mnt/media

⚠️ pct set -mp0 <storage>:<size> creates AND formats the RBD image. Do NOT use pvesm alloc first — it creates an unformatted RBD image, and the container will fail to start with Script exited with status 32 (pre-start hook fails because the filesystem doesn't exist).

If you already used pvesm alloc (recovery):

# 1. Remove the mp0 config
pct set 136 -delete mp0
# 2. Free the unformatted volume
pvesm free media:vm-136-disk-1
# 3. Re-add via pct set (creates + formats)
pct set 136 -mp0 media:500,mp=/mnt/media

7.5 Template Download Gotchas

Templates are per-node, not cluster-wide (unless on shared storage). Each node that creates a CT needs the template downloaded to a local or shared template storage:

# Check which storages support vztmpl on the target node
pvesm status | grep -iE "template|vztmpl"

# Download to the correct storage (NOT "local" if disabled)
pveam download hdd_templates debian-12-standard_12.12-1_amd64.tar.zst

⚠️ local storage is often disabled on nodes that use shared/ceph storage. Use the node's active template storage (e.g. hdd_templates).

7.6 CFS Lock Timeouts Under Ceph Load

When Ceph is recovering/backfilling, pct create can fail with:

unable to create CT <id> - cfs-lock 'storage-<name>' error: got lock request timeout

Mitigation: Try a different storage for the root disk, or wait for Ceph recovery to complete. The lock contention comes from PVE's pmxcfs (configuration filesystem) struggling under I/O pressure from Ceph.

7.7 Unprivileged CT Limitations: No NFS Mounts

Unprivileged LXC containers cannot mount NFS shares (mount.nfs: Operation not permitted). For data migration into an unprivileged CT:

Use rsync over SSH instead of NFS mount
Set up SSH access to the CT (see Section 7.7.1 below)
Run rsync from the source host targeting the CT's IP

7.7.1 SSH Setup in Unprivileged LXC CTs for Rsync Migration

Freshly created LXC CTs from Debian templates have no SSH server installed and no root password set. To enable rsync-over-SSH migration:

# 1. Install SSH server + rsync in the CT (via pct exec, no SSH needed yet)
pct exec <ctid> -- apt-get install -y -qq openssh-server rsync

# 2. Set root password (for initial sshpass-based rsync)
pct exec <ctid> -- bash -c "echo root:TEMPASSWORD | chpasswd"

# 3. Enable root login + password auth in sshd_config
pct exec <ctid> -- bash -c "sed -i 's/^#PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config"
pct exec <ctid> -- bash -c "sed -i 's/^PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config"
pct exec <ctid> -- bash -c "sed -i 's/^#PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config"
pct exec <ctid> -- bash -c "sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config"
pct exec <ctid> -- systemctl restart ssh

# 4. On source host: install sshpass, then rsync with password
sshpass -p "TEMPASSWORD" rsync -a --partial \
  -e "sshpass -p TEMPASSWORD ssh -o StrictHostKeyChecking=no" \
  /source/path/ root@<ct-ip>:/dest/path/

⚠️ Alternative: SSH key via pct exec (no password needed):

# Add source host's pubkey to CT authorized_keys
pct exec <ctid> -- mkdir -p /root/.ssh
pct exec <ctid> -- bash -c "echo 'ssh-ed25519 AAAA... root@source' >> /root/.ssh/authorized_keys"
pct exec <ctid> -- chmod 600 /root/.ssh/authorized_keys
# Still need PermitRootLogin yes in sshd_config!

⚠️ Bridge name pitfall: Freshly created CTs may default to bridge=vmbr1 which doesn't exist on all nodes. Check existing CTs (pct config <existing-ct> | grep net) for the correct bridge name. In the Schön homelab, all CTs use bridge=vmbr0 with tag=30 for VLAN 30.

⚠️ sshpass rsync syntax: The -e flag must wrap the entire SSH command including sshpass:

# CORRECT:
rsync -a -e "sshpass -p PASS ssh -o StrictHostKeyChecking=no" /src/ root@ip:/dst/

# WRONG (sshpass not passed through to rsync's SSH transport):
sshpass -p PASS rsync -a /src/ root@ip:/dst/

⚠️ pct push writes NULL bytes when source file is missing/unreadable (CRITICAL): If you SCP a file to the Proxmox host and the SCP fails (e.g. "Failure" on remote write), pct push will happily push a zero-byte or null-filled file to the CT — no error, exit 0. The resulting file contains thousands of \x00 bytes instead of the intended content. Always verify pct push output with pct exec <ctid> -- cat <dest> and check for null bytes.

⚠️ Shell $ characters in SHA512-CRYPT password hashes get eaten through nested bash/SSH (CRITICAL): Writing Dovecot password files containing {SHA512-CRYPT}$6$salt$hash through pct exec -- bash -c "cat > file << EOF ..." strips all $ characters because bash interprets them as variable references even inside double-quoted heredocs. The resulting hash is truncated and unusable.

Reliable method — base64 round-trip:

# 1. Write the file locally (preserves $ characters)
cat > /tmp/dovecot-users << 'LOCAL_EOF'
user@domain.com:{SHA512-CRYPT}$6$salt$hash...
LOCAL_EOF

# 2. Base64 encode
B64=$(base64 /tmp/dovecot-users | tr -d '\n')

# 3. Decode inside CT via pct exec
pct exec <ctid> -- bash -c "echo $B64 | base64 -d > /etc/dovecot/users"

# 4. Verify — check for intact $ characters
pct exec <ctid> -- cat /etc/dovecot/users
# Should show: user@domain.com:{SHA512-CRYPT}$6$salt$hash...

Why this works: Base64 encoding eliminates all special characters before transit. The base64 -d inside the CT restores the exact bytes including $, {, } characters. No shell interpretation occurs on the encoded payload.

Alternative: Write the file locally, SCP to the Proxmox host, then pct push — but verify the SCP succeeded AND the pushed content is non-null before trusting it.

7.7.2 Background Rsync with sshpass for Large Mailbox Migration

For migrating mailboxes (Dovecot maildir format) from a Docker volume to a CT:

# On source host (e.g. 10.0.30.100):
nohup bash -c "
rsync -a --partial -e \"sshpass -p TEMPASSWORD ssh -o StrictHostKeyChecking=no\" \
  /var/lib/docker/volumes/mailserver/mail/vhosts/ \
  root@<ct-ip>:/var/mail/vhosts/ 2>&1
echo VHOSTS_DONE:\$?
" > /tmp/rsync-vhosts.log 2>&1 &

Monitor: ps aux | grep rsync and sshpass -p PASS ssh root@<ct-ip> "du -sh /var/mail/vhosts/"

7.7.3 Dovecot IMAP Setup in LXC on Ceph EC

Pattern for standing up an IMAP server in an unprivileged LXC CT with mail data on Ceph EC 4+1:

# 1. Create CT with rootfs on replicated RBD, mail data on EC RBD
pct create <ctid> hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --hostname imap --cores 2 --memory 2048 --swap 512 \
  --rootfs vm_disks:8 \
  --net0 name=eth0,bridge=vmbr0,tag=30,ip=10.0.30.40/24,gw=10.0.30.1 \
  --unprivileged 1 --features nesting=1 --ostype debian

# 2. Add EC-backed mount point for mail data
pct set <ctid> -mp0 media:20,mp=/var/mail

# 3. Install Dovecot
pct exec <ctid> -- apt-get install -y -qq dovecot-imapd dovecot-pop3d

# 4. Configure maildir with vhost domain structure
pct exec <ctid> -- bash -c "cat > /etc/dovecot/conf.d/99-mail.conf << 'EOF'
mail_location = maildir:/var/mail/vhosts/%d/%n/mail
passdb {
  driver = passwd-file
  args = scheme=PLAIN username_format=%u /etc/dovecot/users
}
userdb {
  driver = static
  args = uid=5000 gid=5000 home=/var/mail/vhosts/%d/%n
}
listen = *
EOF"

# 5. Create vmail user + directory structure
pct exec <ctid> -- groupadd -g 5000 vmail
pct exec <ctid> -- useradd -u 5000 -g 5000 -d /var/mail -s /usr/sbin/nologin vmail
pct exec <ctid> -- mkdir -p /var/mail/vhosts/<domain>
pct exec <ctid> -- chown -R 5000:5000 /var/mail

# 6. Create users file (passwords TBD — set real passwords later)
pct exec <ctid> -- bash -c "cat > /etc/dovecot/users << 'EOF'
user@domain.com:{PLAIN}changeme
EOF"

# 7. Restart + verify
pct exec <ctid> -- systemctl restart dovecot
pct exec <ctid> -- ss -tlnp | grep -E "143|993"

Maildir structure mapping: Docker mailserver volumes use mail/vhosts/<domain>/<user>/mail/ — Dovecot's mail_location = maildir:/var/mail/vhosts/%d/%n/mail maps directly to this structure. %d = domain, %n = user part of email address.

7.7.4 Dovecot Password Management (Generate + Reset Hashes)

Generating a SHA512-CRYPT Hash for Dovecot Users

Use doveadm pw inside the CT to generate password hashes compatible with the passwd-file auth backend:

# Generate a single hash
pct exec <ctid> -- doveadm pw -s SHA512-CRYPT -p "PLAINTEXT_PASSWORD"
# Output: {SHA512-CRYPT}$6$salt$hash...

# Generate hash and build users file for all accounts in one step
HASH=$(pct exec <ctid> -- doveadm pw -s SHA512-CRYPT -p "PLAINTEXT_PASSWORD")
for user in dominik sarah newsletter dokumente sarah-coc honig; do
  echo "${user}@familie-schoen.com:${HASH}"
done

Writing Password Files (base64 Round-Trip — CRITICAL)

Shell $ characters in SHA512-CRYPT hashes ($6$salt$hash) get eaten through nested bash/SSH/pct-exec. The base64 round-trip from Section 7.7.1 is the reliable method:

# 1. Build the users file locally (preserves $ characters)
cat > /tmp/dovecot-users << 'LOCAL_EOF'
user@domain.com:{SHA512-CRYPT}$6$salt$hash...
LOCAL_EOF

# 2. Base64 encode
B64=$(base64 /tmp/dovecot-users | tr -d '\n')

# 3. Decode inside CT via pct exec
pct exec <ctid> -- bash -c "echo $B64 | base64 -d > /etc/dovecot/users"

# 4. Verify — check for intact $ characters
pct exec <ctid> -- cat /etc/dovecot/users
# Should show: user@domain.com:{SHA512-CRYPT}$6$salt$hash...

# 5. Restart Dovecot
pct exec <ctid> -- systemctl restart dovecot

# Quick IMAP login test via netcat
pct exec <ctid> -- bash -c \
  "echo -e '1 LOGIN user@domain.com PLAINTEXT_PASSWORD\n2 LIST \"\" \"INBOX\"\n3 LOGOUT' | timeout 5 nc localhost 143"
# Success: "1 OK ... Logged in" + "2 OK List completed"
# Failure: "1 NO [AUTHENTICATIONFAILED] Authentication failed"

Setting Uniform Password for All Accounts

To reset all mailboxes to the same password (e.g. during migration):

# 1. Generate hash once
HASH=$(pct exec <ctid> -- doveadm pw -s SHA512-CRYPT -p "NEW_PASSWORD")

# 2. Build users file locally with the same hash for all accounts
# 3. Base64 round-trip to CT (see above)
# 4. Restart Dovecot
# 5. Test login for each account

7.8 Large Data Migration: Background Rsync Pattern

For large transfers (>100GB) that exceed tool timeouts, use nohup background jobs:

# On source host:
nohup bash -c "
  rsync -a --partial /pool/source/Serien/ root@<ct-ip>:/mnt/media/Serien/
  echo SERIEN_DONE:$?
  rsync -a --partial /pool/source/XXX/ root@<ct-ip>:/mnt/media/XXX/
  echo XXX_DONE:$?
" > /tmp/rsync-media.log 2>&1 &

Monitor: tail -f /tmp/rsync-media.log or ps aux | grep rsync

⚠️ Avoid duplicate rsync processes. If a foreground rsync times out and you start a background one, kill the old process first — two rsync processes writing to the same destination can corrupt files.

7.9 Samba Setup in LXC for Media Shares

# Install
pct exec <ctid> -- apt-get install -y -qq samba avahi-daemon

# Create share directories
pct exec <ctid> -- mkdir -p /mnt/media/Filme /mnt/media/Serien /mnt/media/XXX

# Create SMB user
pct exec <ctid> -- bash -c "echo -e 'PASSWORD\nPASSWORD' | smbpasswd -a root -s"

# Write smb.conf (use tee to avoid heredoc-through-SSH escaping issues)
pct exec <ctid> -- tee /etc/samba/smb.conf > /dev/null << 'SMBCONF'
[global]
   workgroup = WORKGROUP
   server role = standalone
   security = user
   map to guest = Bad User
   min protocol = SMB2

[Media]
   path = /mnt/media
   comment = Media Share
   browseable = yes
   writable = yes
   guest ok = no
   valid users = root
   force user = root
   create mask = 0777
   directory mask = 0777
SMBCONF

# Restart and verify
pct exec <ctid> -- systemctl restart smbd nmbd
pct exec <ctid> -- systemctl is-active smbd nmbd
pct exec <ctid> -- ss -tlnp | grep -E ":445|:139"

Access: \\<ct-ip>\Media with the configured SMB user.

6.12 Ceph Pool-Full Diagnosis & OSD Reweighting (CRITICAL)

Symptom: Pools Show 100% Full Despite Free Space in Device Class

ceph df shows pools at 100% with MAX AVAIL = 0, but the raw device class (e.g. HDD) has terabytes of free space. This is caused by individual OSDs hitting full_ratio (default 95%).

Diagnosis:

ceph osd df
# Look for OSDs at 95%+ — these trigger the full condition
# Example: osd.10 at 95.00%, osd.7 at 90.26% while osd.6 at 8.92%

Mechanism: When any OSD reaches full_ratio (default 0.95), Ceph blocks ALL writes to PGs mapped to that OSD. Since nearly every pool has PGs on every OSD, ALL pools show 100% full — even though other OSDs in the same device class have plenty of space. The MAX AVAIL column drops to 0 for affected pools.

Root cause: CRUSH distributes PGs proportionally to OSD weight. Small OSDs (e.g. 1TB) fill up faster than large OSDs (e.g. 3TB) when data grows uniformly. Uneven disk sizes in the same CRUSH hierarchy cause this.

Fix: Raise full_ratio + Reweight Overloaded OSDs

# 1. Temporarily raise full_ratio to unblock writes and allow backfill
ceph osd set-full-ratio 0.97

# 2. Drain overloaded OSDs by reducing their reweight (0.0-1.0)
ceph osd reweight <osd-id> 0.5    # Aggressive drain
ceph osd reweight <osd-id> 0.8    # Moderate drain

# 3. Automatic balancing (alternative to manual reweight)
ceph osd reweight-by-utilization
# Moves PGs from over-utilized to under-utilized OSDs automatically

# 4. Monitor backfill progress
ceph -s    # Watch for "recovery" / "backfill" activity
ceph osd df  # Verify UTIL% decreasing on drained OSDs

# 5. Once balanced, restore full_ratio to default
ceph osd set-full-ratio 0.95

⚠️ backfill_toofull pitfall: If an OSD is AT full_ratio, Ceph cannot backfill data OFF that OSD (writes are blocked, including rebalance writes). Raising full_ratio to 0.97 gives the 2% headroom needed for backfill to proceed. Without this, ceph -s shows Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull.

Effect on pool availability (observed 2026-07-04):

Pool	Before (full_ratio=0.95)	After (full_ratio=0.97 + reweight)
media_ec	100%, 0 B avail	68%, 181 GB avail
vm_disks	70%, 101 GB avail	68%, 108 GB avail
hdd_disk	100%, 0 B avail	94%, 75 GB avail
tm_disks	100%, 0 B avail	74%, 113 GB avail

Long-term fix: Add more OSDs to the device class, or replace small OSDs with larger disks. Reweighting is a tactical fix — the imbalance recurs as data grows.

6.13 ZFS Dataset Destruction: Busy Processes & Async Free

Destroying a ZFS Dataset with Blocking Processes

zfs destroy -r <pool/dataset> fails with cannot unmount: pool or dataset is busy when processes have their cwd or open files in the dataset.

Find and kill blockers:

# Find processes with cwd or open files in the dataset
lsof +D /pool/dataset/         # Lists all processes accessing the dataset
fuser -mv /pool/dataset/        # Alternative: shows PIDs and access type

# Kill all blocking processes (including background monitors!)
kill -9 <PID1> <PID2> ...

# Retry destroy
zfs destroy -r <pool/dataset>

⚠️ Background monitor processes trap: If you launched a background polling loop (e.g. watching ZFS async free progress), that process has its cwd in the dataset and will block destruction. Always kill background monitors before destroying.

⚠️ zfs destroy enters D-state (uninterruptible sleep): For large datasets (hundreds of GB), zfs destroy blocks on I/O and cannot be killed (SIGKILL immune). On degraded RAIDZ2 pools with defective disks, this can take 10+ minutes. Do NOT assume it's hung — monitor with ps aux | grep "zfs destroy" and wait.

ZFS Async Free Behavior

After rm -rf on a ZFS dataset, zfs list shows high USED long after du shows the data is gone. ZFS frees blocks asynchronously via transaction group (TXG) commits.

du -sh /pool/dataset/ shows actual live data (e.g. 3 GB)
zfs list -o used <pool/dataset> shows ZFS accounting (e.g. 290 GB)
zpool sync <pool> forces a TXG sync but may hang/timeout during heavy I/O
The space is reclaimed eventually — patience is required
Destroying the dataset (zfs destroy -r) also triggers async free of all blocks

Diagnostic pattern:

# Confirm data is actually gone
du -sh /pool/dataset/           # Live data (should be small)
zfs list -o used,refer <pool>   # ZFS accounting (may lag behind)
# Gap between du and zfs = pending async free

6.14 Moving OSDs Between Nodes

There is no ceph osd move command. Three approaches exist, depending on whether the physical disk moves or only the CRUSH position changes.

Option A: CRUSH Position Change Only (Disk Stays Physically)

# Move OSD to different host in CRUSH tree (does NOT move data)
ceph osd crush set osd.X <weight> root=default host=<new-host>

⚠️ This tells Ceph the OSD is on a different host, but the disk hasn't moved. Data is NOT redistributed. Only use for CRUSH topology corrections, not physical migrations.

Option B: Physical Disk Migration (No Data Loss)

When physically moving a disk + its DB device to a new node:

# On old node:
ceph osd set noout
systemctl stop ceph-osd@X
ceph osd out osd.X
ceph osd crush remove osd.X

# Physically move disk(s) to new node

# On new node:
ceph-volume lvm scan          # Finds existing OSD on the disk
ceph-volume lvm activate X    # Reactivates OSD with same ID + data intact
ceph osd crush add osd.X <weight> root=default host=<new-node>
ceph osd unset noout

⚠️ DB device must move too. If the DB is on a separate SSD partition (e.g. ceph-db-vg/osd-X-db), that SSD must also move to the new node, or the OSD cannot start. If only the HDD moves without its DB SSD, use Option C instead.

⚠️ ceph-volume lvm scan only works if the LVM metadata on the disk is intact. If the disk was wiped or LVM headers damaged, the OSD must be rebuilt (Option C).

Option C: Destroy + Recreate (Standard, Data Via Backfill)

The officially supported method. Data is rebuilt from peer replicas via backfill:

# On monitor node:
ceph osd destroy X --force    # --force required when OSD is down
ceph osd purge X --force      # Removes from CRUSH

# On new node (disk installed):
ceph-volume lvm create --data /dev/sdX --block.db /dev/ceph-db-vg/osd-X-db

See Section 6.10 for --block.db flag details and OSD ID recycling notes.

Decision Matrix

Scenario	Option	Data Loss	Downtime
Only CRUSH topology fix	A	None	None
Physical disk + DB SSD move together	B	None	Brief (stop/start)
Only HDD moves (DB on old node's SSD)	C	None (backfill)	None (async)
Disk wiped or LVM damaged	C	None (backfill)	None (async)
Moving to node without Ceph installed	C	None (backfill)	None (async)

7.10 EC Pool Expansion: Adding More OSDs Later

EC pools automatically rebalance when new OSDs are added — no pool reconfiguration needed:

# Add new HDD OSD on n5pro (or any node)
pveceph osd create /dev/sdX

# EC pool redistributes data across all HDD OSDs automatically
# Monitor progress:
$CEPH -w  # Watch for recovery/backfill completion

The k=4, m=1 structure stays the same; only the per-OSD utilization improves. To change k+m (e.g. to 5+1), a new pool must be created and data migrated.

7.11 ZFS Pool Inventory Assessment

When evaluating a ZFS pool for migration to Ceph, gather complete dataset information:

# Pool overview
zpool list
zpool status

# Dataset tree with sizes
zfs list -o name,used,refer -r <pool>

# Key datasets to assess:
# - Docker storage (often huge with ZFS driver, 700+ subdatasets)
# - TimeMachine backups (sparsebundles, large but mostly snapshots)
# - Home directories (user data, maildirs)
# - Download/media directories (migration candidates)
# - Docker volumes (named volumes with app data)

Size interpretation:

USED includes snapshots/clones — space held by deleted data still referenced by snapshots
REFER is live data only
USED − REFER = snapshot overhead (reclaimable by destroying old snapshots)

Common findings:

Docker ZFS storage driver creates one subdataset per layer — 700+ subdatasets is a sign of layer proliferation and correlates with daemon instability
TimeMachine sparsebundles show high USED but low REFER — old snapshots hold most of the space
Dedup ratio (zfs list -o dedupratio) near 1.0× means dedup is ineffective

7.12 Ceph OSD Moving Between Nodes

See Section 6.11 for the detailed decision matrix and procedures for moving OSDs between nodes (CRUSH-only, physical disk migration, and destroy+recreate).

Key takeaway: There is no ceph osd move command. Physical disk migration with ceph-volume lvm scan/activate preserves data but requires the DB device to move too. Destroy+recreate is the standard approach when DB devices can't move — data is rebuilt from peer replicas via backfill.

98 KiB Raw Blame History Unescape Escape