hermes-skills/devops/proxmox-ve-administration/SKILL.md

---
name: proxmox-ve-administration
description: "Class-level skill for Proxmox VE administration. Covers API auth, LXC GPU acceleration, storage diagnostics, VLAN networking, UPS power management, PBS backup (local LXC + OpenStack offsite), OpenStack VM provisioning, and infrastructure monitoring (Prometheus + Grafana + Telegram alerting)."
version: 1.22.0
tags: [proxmox, lxc, gpu, storage, iscsi, zfs, rbd, vlan, networking, ceph, pbs, backup, nfs, openstack]
---

## Overview

This umbrella skill consolidates all Proxmox VE administration tasks:
- **API/Auth** — SSH key `~/.ssh/id_ed25519_proxmox`, no BatchMode. RBD LXC: see `references/lxc-creation-rbd-clone-2026-07.md`
- **Avahi Reflector** — CT 127 dual-VLAN, unidirectional. See `references/avahi-reflector-setup.md`
- **LXC GPU** — privileged CT, device pass-through, ROCm/CUDA/Intel Arc drivers
- **PVE-Native Volume Ops** — `pct move-volume`, `pct create --storage`, `pvesm alloc` + `mke2fs`, unprivileged CT permission mapping, Samba TimeMachine CT setup. See `references/pve-native-volume-ops-2026-07.md`
- **Avahi mDNS Reflector** — cross-VLAN Bonjour/TimeMachine discovery via dual-interface CT. See `references/avahi-reflector-setup.md`
- **Storage** — iSCSI/ZFS/RBD, Ceph pool redundancy, EC clone caveats, PG management, pool-full recovery, EC reweight trap, backfillfull flag, recovery tuning. See `references/lxc-creation-rbd-clone-2026-07.md`, `references/ceph-pg-management-2026-07.md`, and `references/ceph-recovery-ec-reweight-2026-07.md`
- **VLAN Bridge Setup** — correct interface creation order (VLAN first, bridge second), CT networking
- **Infrastructure Monitoring** — Prometheus + Grafana + Alertmanager + Hermes webhook (auto RCA to Telegram). PVE exporter tuning, dashboard datasource fixes, alerting rules, Ceph/Galera/Blackbox coverage. See `references/alertmanager-webhook-setup.md` and `templates/alerting_rules.yml`.

Load this skill for any Proxmox cluster operation.

---

## Section 1: API Authentication & Multi-Node SSH

### 1.1 API Token Format (CRITICAL)
```
PVEAPIToken=USERNAME!TOKENID=SECRET
```
- `USERNAME` — PVE user (e.g. `root@pam`)
- `TOKENID` — token identifier (e.g. `hermes`)
- `SECRET` — UUID secret key

Common mistake: repeating the token ID: `root@pam!hermes!hermes=...` is WRONG. Correct is `root@pam!hermes=SECRET`.

### 1.2 Multi-Node SSH Key Deployment
```bash
# 1. Discover nodes
ssh root@10.0.20.91 "pvesh get /cluster/status --output-format json"

# 2. Generate key
ssh-keygen -t ed25519 -C "hermes-agent@proxmox" -f ~/.ssh/id_ed25519_proxmox -N ""

# 3. Deploy via sshpass
for ip in 10.0.20.{10,20,30,40,50,60,70,91}; do
  sshpass -p 'PASSWORD' ssh -o StrictHostKeyChecking=no \
    root@$ip "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys" \
    < ~/.ssh/id_ed25519_proxmox.pub
done

# 4. Verify
for ip in 10.0.20.{10,20,30,40,50,60,70,91}; do
  ssh -o BatchMode=yes -i ~/.ssh/id_ed25519_proxmox root@$ip "echo OK; hostname"
done
```

### 1.3 Store SSH Key in 1Password (SECURE_NOTE workaround)
1Password's built-in "SSH Key" category silently drops private keys. Use SECURE_NOTE + CONCEALED field instead.
```bash
python3 -c "
import json, subprocess
pk=open('/home/debian/.ssh/id_ed25519_proxmox').read().strip()
puk=open('/home/debian/.ssh/id_ed25519_proxmox.pub').read().strip()
fpr=subprocess.run(['ssh-keygen','-lf','/home/debian/.ssh/id_ed25519_proxmox.pub'], capture_output=True, text=True).stdout.strip()
template={'title':'hermes-agent Proxmox SSH Key','category':'SECURE_NOTE','fields':[
  {'id':'notesPlain','type':'STRING','purpose':'NOTES','label':'notesPlain','value':f'Public key:\n{puk}\n\nFingerprint:\n{fpr}'},
  {'id':'private_key','type':'CONCEALED','label':'private key','value':pk}
]}
json.dump(template, open('/tmp/op_template.json','w'))
"
cat /tmp/op_template.json | op item create - --vault Hermes
```

### 1.4 API Response Units (CRITICAL)
- **Memory** (`/status`): BYTES → divide by `1024**3` for GB
- **Storage** (`/storage`): kB → divide by `1024**2` for GB
- **No top-level `status` key** in `/nodes/{node}/status` — the response IS the raw dict
- Use `(data or {}).get(key, 0) or 0` safe access since PVE returns `None` for missing values

### 1.5 Common Endpoints
| Endpoint | Description |
|----------|-------------|
| `/api2/json/nodes` | List nodes |
| `/api2/json/nodes/{node}/status` | Node health |
| `/api2/json/nodes/{node}/storage` | Storage list |
| `/api2/json/nodes/{node}/storage/{storage}/status` | Specific storage status |
| `/api2/json/nodes/{node}/qemu` | VMs on node |
| `/api2/json/nodes/{node}/lxc` | CTs on node |

### 1.6 Node IP Mapping (Schön Homelab)
Cluster "Homelab" — 8 nodes. IPs from `corosync.conf`:

| Node | Management IP | Role |
|------|--------------|------|
| proxmox1 | 10.0.20.10 | HA (VM 106) |
| proxmox2 | 10.0.20.20 | MariaDB-02 (VM 301) |
| proxmox3 | 10.0.20.30 | — |
| proxmox4 | 10.0.20.40 | MariaDB-03 (VM 302) |
| proxmox5 | 10.0.20.50 | — |
| proxmox6 | 10.0.20.60 | — |
| proxmox7 | 10.0.20.70 | — |
| n5pro | 10.0.20.91 | Ceph OSDs, GPU compute |

SSH: `ssh -i ~/.ssh/id_ed25519_proxmox root@<IP>`

### 1.7 Checking Updates Across All Nodes
```bash
# Running version
ssh root@NODE "pveversion"

# Pending updates count
ssh root@NODE "apt list --upgradable 2>/dev/null | grep -c '[upgradable]'"

# Key packages (PVE, QEMU, Ceph, Security, Kernel)
ssh root@NODE "apt list --upgradable 2>/dev/null | grep -E 'proxmox|pve-|qemu-server|ceph/stable|kernel|openssl|libssl'"
```

### 1.8 Node Update Procedure (ITIL Change Management)

**Check all nodes for pending updates:**
```bash
# Node IP mapping from corosync.conf
for ip_info in "10.0.20.10:proxmox1" "10.0.20.20:proxmox2" "10.0.20.30:proxmox3" \
  "10.0.20.40:proxmox4" "10.0.20.50:proxmox5" "10.0.20.60:proxmox6" \
  "10.0.20.70:proxmox7" "10.0.20.91:n5pro"; do
  ip="${ip_info%%:*}"; node="${ip_info##*:}"
  echo "=== $node ($ip) ==="
  ssh -o ConnectTimeout=5 -i ~/.ssh/id_ed25519_proxmox root@$ip \
    "pveversion | head -1; apt list --upgradable 2>/dev/null | grep -c '[upgradable]'"
done
```

**⚠️ PVE 9.1.x maintenance mode:** The API endpoint `/nodes/{node}/control/maintenance` does NOT exist in PVE 9.1.x (added in 9.2). On 9.1.x nodes, you cannot set maintenance mode. Instead:
- Live-migrate VMs (`qm migrate`) — works for KVM guests
- **LXC live migration is NOT implemented** — CTs cannot be hot-migrated. They must be stopped/moved or the node is updated in-place with CTs running
- Safe approach for CTs: `apt full-upgrade` (host packages update alongside running CTs), then `reboot` (CTs restart on the same node)

**Update one node at a time (recommended order):**
1. Compute-only nodes first (proxmox6, proxmox7) — lowest risk
2. DB-hosting nodes (proxmox4, proxmox5) — MariaDB VMs should be live-migrated off first
3. Management nodes (proxmox1-3) — HA VM on proxmox1 needs care
4. n5pro last — Ceph OSDs, last-resort storage

**Per-node procedure:**
```bash
# 1. (For 9.2.x nodes only) Set maintenance mode
# pvesh create /nodes/{node}/control/maintenance --startup=ignore
# NOTE: Fails on PVE 9.1.x — endpoint does not exist!

# 2. (Optional) Live-migrate VMs off the node for safety
# qm migrate <vmid> <target-node> --online

# 3. Update packages (CTs keep running during this step)
apt update && apt full-upgrade -y

# 4. Reboot (required for kernel changes, CTs restart automatically)
reboot

# 5. After reboot, verify
pveversion
pvecm status
pvesh get /nodes/{node}/lxc | grep running  # CTs came back up

# 6. (For 9.2.x nodes) Clear maintenance mode
# pvesh create /nodes/{node}/control/maintenance --startup=started
```

**Major version gaps (9.1.x → 9.2.x):** Expect ~170 packages including QEMU 10→11, kernel 7.0.0→7.0.12, and openssl security patches. `apt full-upgrade` is required (not `upgrade`).

### 1.9 Rolling Cluster Update: Guest Migration Strategies

When updating nodes that host running guests, guests must be evacuated first. The strategy depends on guest type and HA management status.

#### Migration Decision Matrix

| Guest Type | HA-managed? | Method | Downtime |
|------------|-------------|--------|----------|
| QEMU VM (Ceph RBD) | Yes | `qm migrate <vmid> <target> --online` (manual, NOT via HA-manager) | Zero (live) |
| QEMU VM (Ceph RBD) | No | `qm migrate <vmid> <target> --online` | Zero (live) |
| LXC CT (any storage) | Any | Cold migration: `pct stop` → `pct migrate` → `pct start` | Full stop duration |
| LXC CT (local storage) | Any | Backup/restore or clone to target | Full stop + transfer |

**⚠️ Prefer manual `qm migrate` over HA-manager migration.** The HA-manager can abort migrations unpredictably, leaving the VM in a locked state. Manual `qm migrate` gives you full control, especially critical for Galera cluster members where quorum must be preserved.

#### Cold Migration Workflow for LXC CTs (Confirmed Working)

```bash
# 1. Stop the CT
pct stop <ctid>

# 2. Migrate to target node (storage must be shared or migrated)
pct migrate <ctid> <target-node>

# 3. Start on target
pct start <ctid>

# 4. Verify
pct status <ctid>  # Should show "running"
```

#### Live Migration Workflow for QEMU VMs

```bash
# Manual live migration (works with Ceph RBD shared storage)
qm migrate <vmid> <target-node> --online

# Check migration success
qm status <vmid>  # Should show "running" on target
```

### 1.10 HA Migration Lock Recovery (CRITICAL)

When HA-managed VM migration fails or is interrupted, the VM can get stuck with a `lock: migrate` state. The VM may have actually migrated successfully, but the lock prevents further operations.

**Symptoms:**
- `qm status <vmid>` shows `lock: migrate`
- `ha-manager status` shows the service in `migrate` state
- Further `qm` commands fail with "VM is locked"

**Recovery procedure:**
```bash
# 1. Check if VM actually migrated (it may have!)
ssh root@<original-node> "qm status <vmid>"  # May show "stopped" or not-found
ssh root@<target-node> "qm status <vmid>"   # May show "running"

# 2. Clear QEMU lock
qm set <vmid> --lock none

# 3. Clear HA-manager state (if HA-managed)
ha-manager set --disable migrate qemu<vmid>  # or ct<vmid>

# 4. Verify HA state normalizes
ha-manager status | grep <vmid>
```

**Key insight:** Always verify the VM's ACTUAL location before attempting re-migration. The lock may be stale — the VM may already be running on the target node.

### 1.11 Galera Cluster Quorum Awareness During Migration

When migrating MariaDB Galera cluster members (VMs), quorum must be preserved:

- **Never migrate more than one Galera VM at a time**
- **Check `wsrep_cluster_status` after each migration step** — should show `Primary`
- **Migration order**: Migrate one node, verify quorum, then proceed to next
- **Avoid parallel migrations** of Galera members even if technically possible

In the Schön homelab: VM 300 (mariadb-01), VM 301 (mariadb-02), VM 302 (mariadb-03) form the Galera cluster. Always sequential.

### 1.12 Shell Quoting Through SSH (PITFALL)

Complex shell commands with nested quotes fail through SSH. Multiple quoting layers (local shell → SSH → remote shell) cause unexpected EOF and parse errors.

**❌ Fails — nested single quotes:**
```bash
ssh root@node "apt-get full-upgrade -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold'"
```

**✅ Works — single-quote outer, double-quote inner:**
```bash
ssh root@node 'DEBIAN_FRONTEND=noninteractive apt-get update -qq && apt-get full-upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" 2>&1 | tail -20'
```

**Rule:** Use single quotes for the outer SSH command argument, double quotes for any inner quoting needs. Avoid backticks and `$()` inside SSH command strings — use pipes instead.

### 1.13 Cluster-Wide Guest Inventory & HA Status (CRITICAL)

**⚠️ Do NOT rely on `pvesh get /cluster/resources --type vm` for HA status.** The `ha` field in that API response is unreliable — it reported all 55 guests as non-HA when 12 were actually HA-managed.

**Correct method — use `ha-manager status` for HA membership, then cross-reference with `/cluster/resources` for guest metadata:**

```bash
# Step 1: Get HA-managed service IDs from ha-manager
HA_SERVICES=$(ssh root@<any-node> 'ha-manager status' | grep '^service ' | \
  awk '{gsub(/[:;]/,"",$2); gsub(/[(:]/,"",$3); print $2":"$3}')

# Step 2: Get full guest inventory
ssh root@<any-node> 'pvesh get /cluster/resources --type vm --output-format json' | \
  python3 -c "
import json, sys, subprocess
data = json.load(sys.stdin)
# Parse ha-manager status to get HA-managed IDs
hm = subprocess.run(['ssh', '-i', '~/.ssh/id_ed25519_proxmox', 'root@10.0.20.10',
    'ha-manager status'], capture_output=True, text=True).stdout
ha_ids = set()
for line in hm.splitlines():
    if line.startswith('service '):
        parts = line.split()
        # service ct:104 (n5pro, started) → extract 'ct:104' → normalize to '104'
        sid = parts[1].rstrip(':')
        ha_ids.add(sid.replace('ct:','').replace('vm:',''))
for g in sorted(data, key=lambda x: (x['node'], x['vmid'])):
    ha = 'YES' if str(g['vmid']) in ha_ids else 'no'
    print(f\"{g['node']:12} {g['vmid']:6} {g.get('name','?'):25} {g['status']:10} HA:{ha}  type={g.get('type','?')}\")
"
```

**Simple quick-check (just HA services):**
```bash
ssh root@<any-node> 'ha-manager status' | grep '^service '
```

Output format: `service <type>:<id> (<node>, <state>)` — e.g. `service vm:300 (proxmox2, started)`.

**LRM health check (also from `ha-manager status`):**
```bash
ssh root@<any-node> 'ha-manager status' | grep '^lrm '
```
Watch for `old timestamp - dead?` — indicates the Local Resource Manager on that node is not reporting. See Section 1.15 for recovery.

### 1.14 LRM Disabled After Rolling Update (PITFALL)

During the 2026-07-02 rolling cluster update, `pve-ha-lrm` on proxmox1 was left `disabled` and `inactive (dead)`. The `ha-manager status` showed `lrm proxmox1 (old timestamp - dead?)` with a stale timestamp from March 2026. This went unnoticed because the CRM master was on proxmox4 and HA services on other nodes continued normally.

**Root cause:** The `pve-ha-lrm` service was manually disabled at some point (or a reboot cycle didn't re-enable it). Unlike `corosync` and `pvedaemon`, the LRM is not auto-enabled by Proxmox's post-boot hooks if it was previously masked/disabled.

**Detection:**
```bash
# Check all nodes for LRM health
ssh root@<any-node> 'ha-manager status | grep "^lrm "'
# Watch for: "old timestamp - dead?" or missing entries

# Direct check on suspected node
ssh root@<node-ip> 'systemctl is-enabled pve-ha-lrm; systemctl is-active pve-ha-lrm'
# Healthy: enabled / active (running) OR idle (no HA guests on that node)
# Broken: disabled / inactive (dead)
```

**Recovery:**
```bash
ssh root@<affected-node-ip> 'systemctl enable pve-ha-lrm && systemctl start pve-ha-lrm'
# Wait ~10s for lock acquisition, then verify
sleep 10
ssh root@<any-node> 'ha-manager status | grep "^lrm "'
# Affected node should now show: idle/watchdog standby (no HA guests) or active/watchdog active (has HA guests)
```

**Post-update checklist — add LRM verification to the rolling update procedure (Section 1.8):**
```bash
# After ALL nodes are updated and rebooted, verify LRM on every node
for ip_info in "10.0.20.10:proxmox1" "10.0.20.20:proxmox2" "10.0.20.30:proxmox3" \
  "10.0.20.40:proxmox4" "10.0.20.50:proxmox5" "10.0.20.60:proxmox6" \
  "10.0.20.70:proxmox7" "10.0.20.91:n5pro"; do
  ip="${ip_info%%:*}"; node="${ip_info##*:}"
  echo "=== $node ==="
  ssh -o ConnectTimeout=5 -i ~/.ssh/id_ed25519_proxmox root@$ip \
    'systemctl is-enabled pve-ha-lrm; systemctl is-active pve-ha-lrm'
done
```

**LRM states explained:**
| `ha-manager status` output | Meaning |
|----------------------------|---------|
| `active, watchdog active` | Healthy, hosting HA-managed guests |
| `idle, watchdog standby` | Healthy, no HA-managed guests on this node |
| `old timestamp - dead?` | **BROKEN** — LRM not running, needs enable+start |

**Note:** `pve-ha-crm` (CRM) only runs on the elected master node — it is normal for CRM to be `inactive` on non-master nodes. Only `pve-ha-lrm` must run on ALL nodes.

### 1.15 HA Management Operations (Add, Remove, Configure, Audit)

#### Adding a Guest to HA Management

```bash
ha-manager add <type>:<id>
# e.g. ha-manager add vm:106
# e.g. ha-manager add ct:108
```

The guest must exist and be on a node with a healthy LRM (Section 1.14). After adding, the HA-manager transitions through `request_start_balance` → `starting` → `started` (~10-25s).

#### Removing a Guest from HA Management

```bash
ha-manager remove <type>:<id>
```

Does NOT delete the guest config — only removes the HA management overlay.

#### Setting HA Parameters

```bash
ha-manager set <type>:<id> --max_relocate <N> --max_restart <N>
```

- `max_restart` — max restart attempts on the SAME node before relocating
- `max_relocate` — max relocation attempts to a DIFFERENT node before giving up

**Recommended values:**

| Guest Class | max_relocate | max_restart | Rationale |
|-------------|-------------|-------------|-----------|
| Production (DB, infra) | 3 | 2 | Balanced — local restart twice, then relocate 3× |
| Non-critical | 1 | 1 | Default — minimal retry pressure |
| Overly aggressive (avoid) | 5 | 5 | Causes restart storms on flapping nodes |

#### Inspecting HA Resource Config (CRITICAL — CLI gotcha)

**⚠️ `ha-manager config <sid>` does NOT work** — returns `400 too many arguments`. No CLI subcommand exists to view individual HA resource config.

**Use `pvesh` instead:**
```bash
# All HA resources with full config
pvesh get /cluster/ha/resources --output-format json

# Quick summary
pvesh get /cluster/ha/resources --output-format json | \
  python3 -c "
import json,sys
data=json.load(sys.stdin)
for g in sorted(data, key=lambda x: x['sid']):
    print(f\"{g['sid']:12} state={g.get('state','?'):10} reloc={g.get('max_relocate','?')} restart={g.get('max_restart','?')}\")"
```

Fields: `sid`, `state`, `type`, `max_relocate`, `max_restart`, `digest`.

#### Cleaning Up Orphaned Guest Configs from Removed Nodes

When a node is removed from the cluster (`pvecm delnode`), its guest config files remain under `/etc/pve/nodes/<removed-node>/`. These appear in `/cluster/resources` with `status=unknown` — "ghost guests".

**Detection:**
```bash
pvesh get /cluster/resources --type vm --output-format json | python3 -c "
import json,sys
data=json.load(sys.stdin)
valid_nodes={'proxmox1','proxmox2','proxmox3','proxmox4','proxmox5','proxmox6','proxmox7','n5pro'}
for g in data:
    if g['status']=='unknown' or g['node'] not in valid_nodes:
        t='ct' if g['type']=='lxc' else 'vm'
        print(f\"GHOST: {t}:{g['vmid']} node={g['node']}\")"
```

**Cleanup:**
```bash
find /etc/pve/nodes/<removed-node>/qemu-server -name "*.conf"
find /etc/pve/nodes/<removed-node>/lxc -name "*.conf"
rm /etc/pve/nodes/<removed-node>/qemu-server/<id>.conf
rm /etc/pve/nodes/<removed-node>/lxc/<id>.conf
```

Do NOT delete directories for active cluster nodes.

### 1.14 EFI Partition Cleanup Before Major Upgrades

Before upgrading across major PVE versions (especially 9.1.x → 9.2.x with kernel changes), check and clean the EFI partition:

```bash
# Check EFI usage
df -h /boot/efi

# List ESP contents
ls /boot/efi/

# Remove old kernel EFI entries if full (keep current + previous)
# Use efibootmgr to identify stale entries
efibootmgr -v
```

EFI partitions can fill up with stale kernel entries across multiple upgrades, blocking new kernel installation.

### 1.15 Root Disk Space Recovery: Remove Unused pve-data Thin Pool

PVE's default installer creates a `pve-data` LVM thin pool (`local-lvm` storage) that is often unused when all VM disks are on Ceph/RBD. This wastes 10-140 GB that could extend `pve-root`.

**Detection — check all nodes:**
```bash
for ip in 10.0.20.{10,20,30,40,50,60,70} 10.0.20.91; do
  ssh -o ConnectTimeout=5 -i ~/.ssh/id_ed25519_proxmox root@$ip \
    'lvs -o lv_name,data_percent --units g 2>/dev/null | grep -E "data|root|swap"; echo "VG Free: $(vgs pve -o vg_free --units g --noheadings 2>/dev/null)"; df -h / | tail -1'
done
```

**Recovery (online, no downtime):**
```bash
# 1. Verify pve-data is empty (0% data_percent)
lvs -o lv_name,data_percent pve/data 2>/dev/null

# 2. Remove the thin pool
lvremove -y pve/data

# 3. Extend root LV to use freed space
lvextend -l +100%FREE pve/root

# 4. Online resize ext4 filesystem
resize2fs /dev/pve/root

# 5. Verify
df -h /
lvs -o lv_name,lv_size --units g
```

**⚠️ Only remove if `local-lvm` storage is not used.** Check `pvesm status` — if `local-lvm` shows 0 used, it's safe. If VMs/CTs use local-lvm, their disks will be destroyed.

### 1.16 Comprehensive Cluster Audit Checklist

For a full cluster health audit, collect these data points:

1. **Corosync:** `pvecm status`, `corosync-cfgtool -s` (ring connectivity), `corosync-quorumtool` (quorum)
2. **HA-Manager:** `ha-manager status` (services + LRM health + CRM master)
3. **Node versions:** `pveversion` on each node (compare PVE + kernel versions)
4. **Node resources:** `pvesh get /cluster/resources --type node` (CPU/MEM/disk/uptime)
5. **Per-node details:** swap usage, CPU model, Ceph mon/OSD presence, disk listing
6. **Ceph health:** `ceph -s`, `ceph health detail`, `ceph osd tree`, `ceph osd df`, `ceph df`
7. **Ceph performance:** `ceph osd perf` (commit/apply latency per OSD)
8. **Ceph config:** `ceph config dump`, pool configs via `ceph osd pool get <pool> all`
9. **Ceph CRUSH:** `ceph osd crush rule dump` (verify class-based rules)
10. **Ceph Mon placement:** `ceph mon dump` (recommend 3 or 5, not 7+)
11. **Disk SMART:** `smartctl -a /dev/sdX` on OSD disks (health, temp, power-on hours)
12. **Network:** `ip link show` (MTU, bonds), `/proc/net/bonding/bond0` (bond mode, slave count)
13. **Datacenter config:** `/etc/pve/datacenter.cfg` (CRS, migration, replication settings)

See `references/cluster-audit-2026-07.md` for a complete worked example with findings and recommendations.

### 2.1 Privileged Container Requirement
Privileged LXC is usually required because `/dev/dri` and `/dev/kfd` are root-owned. Unprivileged works for some Intel Arc setups with proper `lxc.idmap` + `subuid/gid` mapping.

### 2.2 Container Config (`/etc/pve/lxc/<ctid>.conf`)

**AMD ROCm:**
```
features: nesting=1,keyctl=1
unprivileged: 0
lxc.cgroup2.devices.allow: c 226:* rwm    # /dev/dri
lxc.cgroup2.devices.allow: c 511:0 rwm    # /dev/kfd
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file
```

**NVIDIA CUDA:**
```
lxc.cgroup2.devices.allow: c 195:* rwm    # /dev/nvidia*
lxc.cgroup2.devices.allow: c 243:* rwm    # /dev/nvidia-uvm*
lxc.cgroup2.devices.allow: c 235:* rwm    # /dev/nvidiactl
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
```

### 2.3 ROCm Userland-Only Install (`--no-dkms`)
Do NOT install kernel modules inside the container — host already loads `amdgpu`.
```bash
wget https://repo.radeon.com/amdgpu-install/6.3.3/ubuntu/noble/amdgpu-install_6.3.60303-1_all.deb
dpkg -i amdgpu-install_6.3.60303-1_all.deb && apt-get update
amdgpu-install --usecase=rocm --no-dkms -y
```

Takes 30–90 min. Run in background:
```bash
nohup bash -c "amdgpu-install --usecase=rocm --no-dkms -y > /tmp/rocm_install.log 2>&1; echo DONE > /tmp/rocm_done" &
watch 'tail -5 /tmp/rocm_install.log'
```

### 2.4 Disk Space Management
ROCm + PyTorch + ComfyUI = 40–60 GB. Rootfs needs **80 GB minimum**.

Emergency resize:
```bash
pct exec <ctid> -- bash -c 'pip cache purge; apt-get clean; rm -rf /root/.cache/pip /tmp/pip-*'
pct resize <ctid> rootfs 80G
pct exec <ctid> -- bash -c 'resize2fs $(df / | tail -1 | awk "{print \$1}")'
```

### 2.5 RDNA3.5 Workaround (`HSA_OVERRIDE_GFX_VERSION`)
Ryzen AI 9 HX PRO 370 (gfx1150) reports `HIP error: invalid device function`. Fix:
```bash
export HSA_OVERRIDE_GFX_VERSION=11.0.0
# In systemd service:
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"
```

### 2.6 PyTorch ROCm Wheel Pitfalls
- `torchaudio` ROCm wheel links against CUDA libraries → `OSError: libcudart.so`. Fix: install CPU torchaudio or stub it out.
- `python3-rich` Debian package blocks pip upgrades → `dpkg --remove --force-depends python3-rich`
- `LD_LIBRARY_PATH` must include `/usr/local/lib/python3.12/dist-packages/torch/lib` for `libtorchaudio.so`
- After ROCm upgrade, verify PyTorch compatibility. If `hipErrorNoBinaryForGpu`, downgrade PyTorch ROCm wheel to match host's ROCm version.

### 2.7 Verification Steps
```bash
export PATH=/opt/rocm/bin:$PATH
rocminfo | grep "Name:" | head -5
rocm-smi
python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"
python3 -c "import torch; print(torch.cuda.get_device_capability())"
```

### 2.8 When Agent Is Not on the Proxmox Host
`pct`/`pveam` are unavailable locally. Use SSH/paramiko:
```python
import paramiko
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(hostname="10.0.20.91", username="root", password="...")
stdin, stdout, stderr = client.exec_command("pct list && pct exec 204 -- uname -a")
```

### 2.9 Template Download Gotcha
`pveam download` requires a **storage name**, not a node name:
```bash
# WRONG: pveam download n5pro ubuntu-24.04-standard_24.04-2_amd64.tar.zst
# CORRECT: pveam download hdd_templates ubuntu-24.04-standard_24.04-2_amd64.tar.zst
```

### 2.10 Safe HuggingFace Model Download Inside LXC
- Use **resolve** URL, not browser blob URL:
  - `https://huggingface.co/org/repo/resolve/main/path/file.safetensors`
- Always use `-O filename` with `wget`
- Resume with `wget -c`
- Single `pct exec` command for large downloads:
```bash
pct exec <ctid> -- bash -c 'cd /opt/ComfyUI/models/diffusion_models && wget -c --tries=0 --read-timeout=30 "https://huggingface.co/Comfy-Org/Ideogram-4/resolve/main/diffusion_models/ideogram4_fp8_scaled.safetensors" -O ideogram4_fp8_scaled.safetensors'
```

---

## Section 3: Storage Diagnostics

### 3.1 Identify the Outlier Node
Shared storage works on most nodes but fails on one — the most common pattern.

```python
for node in nodes:
    r = subprocess.run(["curl","-sk",f"https://{PVE}:8006/api2/json/nodes/{node}/storage/{storage_name}/status","-H",f"Authorization: {auth}"], capture_output=True, text=True, timeout=15)
    data = json.loads(r.stdout).get("data", {})
    if data:
        print(f"{node:<12} active={data.get('active')} total={data.get('total',0)/(1024**3):.0f}GB")
    else:
        print(f"{node:<12} ❌ NO DATA")
```

### 3.2 Check iSCSI Target Config
```python
r = subprocess.run(["curl","-sk",f"https://{PVE}:8006/api2/json/storage/{storage_name}","-H",f"Authorization: {auth}"], capture_output=True, text=True)
config = json.loads(r.stdout).get("data", {})
# Key fields: portal, target, pool, iscsiprovider
```

### 3.3 Common Failure Patterns
| Symptom | Cause |
|---------|-------|
| `active=0, total=0` on one node | iSCSI session not connected |
| Empty storage list | ZFS pool gone or renamed |
| "Permission denied" but config exists | Underlying resource missing |
| Memory values look impossibly large | Values are BYTES, not kB |

### 3.4 Ext4 MMP Deadlock on RBD-Backed LXC Containers (CRITICAL)

When an LXC container with an RBD (Ceph) root disk is stopped uncleanly (hard stop, host crash, or the mount process was killed mid-mount), the Ext4 filesystem's **MMP (Multi-Mount Protection)** block can be left in an active state. On the next `pct start`, the mount process enters **D-state (uninterruptible sleep)** waiting for MMP to stabilize — but MMP never stabilizes because the mount process itself holds the MMP lock. This creates a **kernel-level deadlock** that cannot be resolved by killing the process (D-state is immune to SIGKILL).

**Symptoms:**
- `pct start <ctid>` hangs indefinitely (timeout)
- `pct status <ctid>` also hangs
- `mount -o discard /dev/rbd0 /var/lib/lxc/.pve-staged-mounts/rootfs` process in `D` state
- `dmesg` shows: `EXT4-fs warning (device rbd0): ext4_multi_mount_protect:324: MMP interval 42 higher than expected, please wait.`
- `rbd unmap` fails with `Device or resource busy`
- `tune2fs -E clear_mmp` fails with `MMP: device currently active`
- `e2fsck` fails with `MMP: e2fsck being run while checking MMP block`

**Recovery Procedure (confirmed working on PVE 9.2.3 / Ceph 19.2.3):**

```bash
# PHASE 1: Break the D-state deadlock via Ceph client blacklist

# 1. Identify the stuck mount process and the RBD client
ps aux | grep "mount.*rbd" | grep -v grep
# Note the PID — it will be in D state

# 2. Find the RBD exclusive lock holder
rbd --id admin lock list <pool>/<image> --format json
# Note: locker ID, address, and cookie

# 3. Blacklist the Ceph client — this forces the kernel RBD driver
#    to error out all outstanding I/O, which releases the D-state process
ceph --conf /etc/pve/ceph.conf --keyring /etc/pve/priv/ceph.client.admin.keyring \
  osd blacklist add <client_address>
# e.g. ceph osd blacklist add 10.0.20.50:0

# 4. Wait for the D-state process to exit (usually 3-5 seconds)
sleep 5
ps aux | grep "mount.*rbd" | grep -v grep || echo "mount process freed!"

# PHASE 2: Clean up RBD state

# 5. Unmap the RBD device
rbd --id admin unmap /dev/rbd0
# If unmap fails with "Device or resource busy", blacklist again (new client ID)
# and retry. The blacklist creates a new RBD client session each time.

# 6. Remove ALL blacklists (they persist for 3600s by default)
ceph osd blacklist ls  # List all
ceph osd blacklist rm <address>  # Remove each one

# 7. Verify RBD lock is cleared
rbd --id admin lock list <pool>/<image>
# Should show no locks

# PHASE 3: Clear MMP and restart container

# 8. Map RBD fresh
rbd --id admin map <pool>/<image>

# 9. Clear MMP with FORCE flag (CRITICAL: -f required!)
tune2fs -f -E clear_mmp /dev/rbd/<pool>/<image>
# Without -f: "MMP: device currently active while trying to open"
# With -f: "Recovering journal." + success

# 10. Verify MMP is cleared
dumpe2fs /dev/rbd/<pool>/<image> | grep -i "multi-mount" || echo "MMP cleared!"

# 11. Unmap RBD (let PVE handle remapping during pct start)
rbd --id admin unmap /dev/rbd/<pool>/<image>

# 12. Start the container
pct start <ctid>
```

**Key Pitfalls:**

- **`tune2fs -E clear_mmp` WITHOUT `-f` fails** — it sees the MMP block as "active" even when no process holds the device. The `-f` (force) flag is mandatory.
- **Blacklisting creates new RBD client sessions** — each `osd blacklist add` generates a new client ID. You may need to blacklist 2-3 times as each new mapping attempt creates a new client that also gets stuck. List and remove ALL blacklists before the final clean map.
- **`rbd lock remove` syntax** — the command is `rbd lock remove <image> <locker> <cookie>`, NOT `rbd lock break`. The locker and cookie must be parsed from `rbd lock list --format json`. However, blacklisting the client is usually more effective than trying to remove the lock manually.
- **smbd may not auto-start after recovery** — after an unclean CT restart, `smbd` may remain `inactive (dead)` even though the container is running. Manually start it: `pct exec <ctid> -- systemctl start smbd`. Check with `pct exec <ctid> -- systemctl is-active smbd nmbd avahi-daemon`.
- **Do NOT reload the `rbd` kernel module** — this would disconnect ALL RBD devices on the node, affecting all VMs/CTs using Ceph storage. The blacklist approach is surgical — it only affects the specific stuck client.

**When to use this procedure:**
- LXC container with RBD root disk fails to start (hangs on `pct start`)
- `dmesg` shows MMP-related Ext4 warnings on an RBD device
- A `mount` process is stuck in D-state on an RBD device
- `pct status` for the CT hangs or times out

**Alternative (if blacklist doesn't work):** Rebooting the PVE node hosting the CT will clear all D-state processes and RBD mappings. This is heavier (affects all guests on the node) but guaranteed to work. Use only if the blacklist approach fails after 3 attempts.

---

## Section 4: VLAN Bridge Setup

### 4.1 Critical Order: VLAN FIRST, Bridge SECOND
The bridge's `bridge_ports` must reference an interface that already exists.

```bash
# 1. Create VLAN interface
curl -k -X POST https://10.0.20.10:8006/api2/json/nodes/proxmox1/network \
  -H "Cookie: PVEAuthCookie=TICKET" -H "CSRFPreventionToken: TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"iface":"bond0.30","type":"vlan","vlan-id":"30","vlan-raw-device":"bond0","autostart":1}'

# 2. Create bridge referencing the VLAN
curl -k -X POST https://10.0.20.10:8006/api2/json/nodes/proxmox1/network \
  -H "Cookie: PVEAuthCookie=TICKET" -H "CSRFPreventionToken: TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"iface":"vmbr30","type":" bridge","cidr":"10.0.30.1/24","bridge_ports":"bond0.30","autostart":1}'
```

### 4.2 Adding a NIC to a VM with VLAN Tagging

On a VLAN-aware bridge (`bridge-vlan-aware yes`), add a NIC tagged to a specific VLAN:

```bash
qm set <vmid> -net1 virtio=<MAC>,bridge=vmbr0,tag=50
# tag=50 = VLAN 50. Use a specific MAC if recreating a previously-removed NIC.
# Requires VM reboot for the guest OS to detect the new interface.
```

Use case: giving a HAOS VM access to an IoT VLAN for mDNS/Zeroconf discovery (mDNS multicast doesn't cross VLAN boundaries). After adding the NIC, configure it inside the guest OS (for HAOS: `ha network update <iface> --ipv4-method auto`).

**Multiple NICs for multiple VLANs**: The same pattern applies for additional VLANs. This homelab's HA VM 106 now has 3 NICs:
- `net0` → VLAN 30 (10.0.30.10) — management/services
- `net1` → VLAN 50 (10.0.50.127) — Shellys/IoT (added 2026-06-27)
- `net2` → VLAN 40 (10.0.40.61) — Midea AC on 10.0.40.54 (added 2026-06-27)

Each additional NIC follows the same `qm set <vmid> -netN virtio=<MAC>,bridge=vmbr0,tag=<VLAN>` + reboot + `ha network update enp0s<N> --ipv4-method auto` pattern. Interface names increment: enp0s18, enp0s19, enp0s20.

### 4.3 qm guest exec Shell Escaping (CRITICAL)

`qm guest exec <vmid> -- sh -c "..."` passes through multiple shell layers (local SSH → remote qm → guest agent → sh -c), causing severe escaping issues with special characters.

**Characters that break:**
- `!` — bash history expansion at every layer; `sed 's/!/X/'` produces literal `x21` or gets eaten
- `$` — variable expansion
- `` ` `` — backtick command substitution
- `\` — inconsistent escaping across layers

**HAOS guests have minimal tools**: `sed`, `awk`, `sh`, `curl`, `grep`, `ha` CLI. NO `python3`, `perl`, `busybox`, or `tcpdump`.

**Working approaches:**
1. **`awk` with `sprintf`** — the most reliable for special char substitution:
   ```bash
   # Produces literal ! via sprintf("%c", 33) without shell interpretation
   qm guest exec 106 -- sh -c "awk '{gsub(/x21/, sprintf(\"%c\",33)); print}' /path/file > /tmp/fixed && mv /tmp/fixed /path/file"
   ```
2. **Base64 round-trip** — for whole-file replacement: read file via `cat`, modify locally, write back via `echo BASE64 | base64 -d`. But `!` in decoded content can still get eaten if passed through `echo`.
3. **SCP** — if the guest runs sshd (HAOS doesn't by default), bypass `qm guest exec` entirely.

**HAOS redacts secrets in output**: `grep`/`cat` on files containing credentials returns `***` instead of actual values. Cannot verify password patches via grep — must test functionality instead.

### 4.4 CT Start
```bash
curl -k -X POST https://10.0.20.10:8006/api2/json/nodes/proxmox1/lxc/CTID/status/start \
  -H "Cookie: PVEAuthCookie=TICKET" -H "CSRFPreventionToken: TOKEN"
```

Verify: `status` should be `"running"`, not `"stopped"`.

---

## Section 5: NUT/UPS Power Management for Proxmox Clusters

Configure controlled shutdown on power loss: all Proxmox nodes stop their guests gracefully, then shut down. On power return, guests auto-start via `onboot=1`.

### 5.1 Architecture

```
APC UPS (USB) → NUT Master (10.0.30.100) → upsd :3493
                        ↓ upssched (FSD after 30s on battery)
                   ┌────┴────────────────┐
            NUT Slaves (all Proxmox nodes)
            upsmon receives FSD → runs shutdown script
            → qm shutdown all VMs → pct shutdown all CTs → shutdown -h now
```

- **NUT Master**: Host with USB-connected UPS. Runs `upsd` + `upsmon` (master) + `upssched`.
- **NUT Slaves**: All Proxmox nodes. Run `upsmon` (slave). On FSD, execute guest-shutdown script.
- **Shutdown order**: Slaves shut down first (guests + node), master shuts down last.
- **Restart**: Proxmox boots → `onboot=1` guests start automatically.

### 5.2 Setting onboot=1 for Auto-Restart (PITFALL)

**⚠️ `qm set` and `pct set` operate on LOCAL config files.** Running them from one node for guests on other nodes fails with `"Configuration file 'nodes/X/qemu-server/Y.conf' does not exist"`.

**Correct approach — SSH to EACH node separately:**
```bash
for ip in 10.0.20.{10,20,30,40,50,60,70} 10.0.20.91; do
  ssh -i ~/.ssh/id_ed25519_proxmox root@$ip "
    # Set onboot=1 for all running VMs
    for vmid in \$(qm list --running 2>/dev/null | awk 'NR>1{print \$1}'); do
      qm set \$vmid -onboot 1
    done
    # Set onboot=1 for all running CTs
    for ctid in \$(pct list --running 2>/dev/null | awk 'NR>1{print \$1}'); do
      pct set \$ctid -onboot 1
    done
  "
done
```

**Verify across all nodes:**
```bash
for ip in 10.0.20.{10,20,30,40,50,60,70} 10.0.20.91; do
  ssh -i ~/.ssh/id_ed25519_proxmox root@$ip "
    for ct in \$(pct list 2>/dev/null | awk 'NR>1{print \$1}'); do
      echo \"CT \$ct: \$(grep '^onboot' /etc/pve/lxc/\${ct}.conf 2>/dev/null || echo 'not-set')\"
    done
    for vm in \$(qm list 2>/dev/null | awk 'NR>1{print \$1}'); do
      echo \"VM \$vm: \$(grep '^onboot' /etc/pve/qemu-server/\${vm}.conf 2>/dev/null || echo 'not-set')\"
    done
  "
done
```

### 5.3 Deploying NUT Clients to Proxmox Nodes

Install `nut-client` and configure `upsmon` as slave on each Proxmox node:

```bash
# Prepare config files locally, then SCP + install on each node
# See templates/nut-slave-upsmon.conf and templates/ups-shutdown.sh

for ip in 10.0.20.{10,20,30,40,50,60,70} 10.0.20.91; do
  scp -i ~/.ssh/id_ed25519_proxmox \
    templates/nut-slave-upsmon.conf templates/ups-shutdown.sh \
    root@$ip:/tmp/

  ssh -i ~/.ssh/id_ed25519_proxmox root@$ip '
    export DEBIAN_FRONTEND=noninteractive
    apt-get install -y -qq nut-client
    cp /tmp/nut-slave-upsmon.conf /etc/nut/upsmon.conf
    cp /tmp/ups-shutdown.sh /usr/local/bin/ups-shutdown.sh
    chmod 755 /usr/local/bin/ups-shutdown.sh
    echo "MODE=netclient" > /etc/nut/nut.conf
    systemctl enable nut-monitor
    systemctl restart nut-monitor
    # Verify
    systemctl is-active nut-monitor
    upsc apc@NUT_MASTER_IP 2>&1 | head -2
  '
done
```

### 5.4 NUT Master Configuration

On the UPS-connected host (e.g. 10.0.30.100):

1. **`/etc/nut/nut.conf`**: `MODE=netserver`
2. **`/etc/nut/ups.conf`**: Driver section matching USB device (see `templates/nut-master-ups.conf`)
3. **`/etc/nut/upsd.conf`**: `LISTEN 0.0.0.0 3493`
4. **`/etc/nut/upsd.users`**: Both master and slave users (see `templates/nut-master-upsd-users.conf`)
5. **`/etc/nut/upsmon.conf`**: Master mode with `NOTIFYCMD /usr/sbin/upssched` (see `templates/nut-master-upsmon.conf`)
6. **`/etc/nut/upssched.conf`**: Timed FSD trigger (see `templates/nut-upssched.conf`)
7. **`/etc/nut/upssched-cmd`**: Script that runs `upsmon -c fsd` when timer fires (see `templates/upssched-cmd.sh`)

Key services: `nut-driver.target`, `nut-server`, `nut-monitor` — all must be `active`.

### 5.5 Shutdown Script (Proxmox Nodes)

The `SHUTDOWNCMD` in `upsmon.conf` points to `/usr/local/bin/ups-shutdown.sh`:

```bash
#!/bin/bash
# Stop all running VMs and CTs gracefully, then power off node
for vmid in $(qm list --running 2>/dev/null | awk 'NR>1{print $1}'); do
    qm shutdown "$vmid" --timeout 60 --forceStop 1 &
done
for ctid in $(pct list --running 2>/dev/null | awk 'NR>1{print $1}'); do
    pct shutdown "$ctid" --timeout 60 --forceStop 1 &
done
wait
sleep 3
shutdown -h now
```

### 5.6 Pitfalls

- **Heredoc escaping through SSH fails.** Don't try to `cat << 'EOF'` nested configs through SSH. Write files locally, SCP them, then move into place. Alternatively, use base64 encoding.
- **`NOTIFYTYPE` is not a valid upsmon.conf directive.** Use individual `NOTIFYFLAG` lines instead. `LOWBATTERY` → `LOWBATT` (NUT 2.8.x naming).
- **upssched PIPEFN/LOCKFN need `/run/nut/` to exist.** `mkdir -p /run/nut && chown nut:nut /run/nut` before starting.
- **Battery age matters.** If the UPS battery is degraded (< 2 min runtime), the Low-Battery flag fires immediately, bypassing the upssched grace period. Replace batteries older than 3-4 years.
- **Store NUT credentials in 1Password.** Create separate items for master and slave users.

### 5.7 Verification

```bash
# From each Proxmox node:
upsc apc@NUT_MASTER_IP 2>&1 | head -3    # Should show battery.charge

# On NUT master:
systemctl is-active nut-server nut-monitor
upsc apc@localhost 2>&1 | head -5

# Verify shutdown script syntax:
bash -n /usr/local/bin/ups-shutdown.sh
```

### 5.8 Battery Runtime Calibration via `upsrw`

When a UPS battery is replaced but the firmware retains old calibration data, `battery.runtime` may show implausibly low values (e.g. 60s for a fresh battery) and `ups.status` may show `OL LB` (Online + Low Battery) despite 100% charge.

**Updating battery date resets the firmware's age-based runtime estimate:**
```bash
# CRITICAL: -s FLAG MUST COME BEFORE -u/-p (otherwise silent failure)
upsrw -s battery.mfr.date=YYYY/MM/DD -u upsadmin -p "$NUT_PASS" apc@localhost
```

**Adjusting `battery.runtime.low`** (threshold at which LB flag triggers):
```bash
upsrw -s battery.runtime.low=30 -u upsadmin -p "$NUT_PASS" apc@localhost
```

**⚠️ PITFALL: `upsrw` flag ordering.** `upsrw -u user -p pass -s VAR=VALUE apc@host` FAILS silently with `"Unexpected response from upsd: ERR PASSWORD-REQUIRED"`. The `-s VAR=VALUE` flag MUST precede `-u`/`-p` in the argument list.

**Verification after calibration:**
```bash
upsc apc@localhost 2>&1 | grep -E 'battery.mfr.date|battery.runtime|ups.status'
# Expected: runtime jumps from ~60s to ~285s, status changes from "OL LB" to "OL"
```

Note: Some UPS firmware (e.g. APC Back-UPS ES 700G) caches `battery.mfr.date` and may not persist the change across power cycles. The runtime recalculation triggered by the date update is the important part — it forces the firmware to recompute its degradation curve.

---

---

## Section 6: Ceph OSD Management & Disk Integration

### 6.1 Cluster Overview Commands

```bash
ceph -s                    # Health, services, data summary
ceph osd tree              # Topology: hosts, OSDs, weights, status
ceph osd df                # Per-OSD usage
ceph df                    # Pool-level + raw storage summary
ceph osd pool ls detail    # Pool configs (replication, CRUSH rules, PGs)
pvesm status               # Proxmox storage view
```

### 6.2 Adding New OSDs to a Node (e.g., n5pro)

When integrating recycled disks from a decommissioned server into a new Proxmox node:

```bash
# On the target node (via SSH from another node or directly):
# 1. Identify available disks
lsblk -o NAME,SIZE,TYPE,FSTYPE

# 2. Zap any existing filesystem/partitions (DESTROYES data!)
ceph-volume lvm zap /dev/sdX --destroy

# 3. Create OSD via Proxmox GUI or CLI
pveceph osd create /dev/sdX

# 4. Verify it joined the cluster
ceph osd tree | grep $(hostname)
```

**Prerequisites on the target node**:
- Proxmox VE installed and joined to cluster
- Ceph installed via `pveceph install`
- Ceph monitor may or may not be needed (depends on cluster redundancy)
- Disks must be wiped (no ZFS, LVM, or partitions remaining)

### 6.3 Decommissioning OSDs (Data Redistribution)

Before removing OSDs from a node being decommissioned:

```bash
# 1. Mark OSDs as OUT (triggers data redistribution to other OSDs)
ceph osd out <osd-id>
# e.g., ceph osd out 8 9

# 2. Monitor rebalance progress (WAIT for completion before destroying)
ceph -w    # Watch for "recovery" to complete, PGs back to active+clean
ceph osd df  # Verify remaining OSDs absorbed the data

# 3. Once safe, destroy and purge
ceph osd destroy <osd-id> --yes-i-really-mean-it
ceph osd purge <osd-id> --yes-i-really-mean-it

# 4. Remove the monitor if the node hosted one
systemctl stop ceph-mon@$(hostname)
ceph mon remove $(hostname)
```

**Capacity check before draining**: Ensure remaining cluster has enough free space for redistributed data.
```bash
ceph df  # Check TOTAL AVAIL vs. amount on OSDs being removed
```

### 6.4 Removing a Node from the Cluster

```bash
# 1. Decommission all OSDs on the node (Section 6.3)
# 2. Remove Ceph monitor (if present)
# 3. Remove from Proxmox cluster
pvecm delnode <hostname>
# 4. If node is permanently gone and unreachable:
pvecm delnode <hostname> -force
```

### 6.5 Ceph Health Warnings to Investigate Before Adding Load

Before migrating data INTO a Ceph cluster, check for existing health issues:

```bash
ceph health detail
```

Common warnings:
- **Slow OSD operations** — disk performance degradation, check `ceph osd perf`
- **Stalled BlueFS reads** — db device (WAL/db partition) failing
- **Low space on a node** — uneven distribution, may need CRUSH adjustment
- **Destroyed OSDs still in tree** — `osd.6 destroyed` entries reduce apparent capacity, clean up with `ceph osd purge`
- **MON_DISK_LOW** — Mon node root disk <20% free, can cause Mon flapping. Clean up disk space or remove excess Mons.

### 6.5.1 Ceph Mon Reduction (Over-provisioned Mons)

Running too many Mons (7+) increases Paxos consensus latency on every map update. Recommended: 3 for clusters up to 10 nodes, 5 for larger clusters. Mons on non-PVE nodes (e.g. standalone Ubuntu) add no value and should be removed first.

**Procedure (no downtime, live):**
```bash
# Check current Mons
ceph mon dump

# Remove excess Mons one at a time
ceph mon remove <name>    # e.g. ceph mon remove ubuntu

# Verify quorum after each removal
ceph -s | grep "mon:"

# Target: 3 Mons on well-distributed physical nodes
# Good placement: one Mon per rack/room, avoid co-location with OSD-heavy nodes if possible
```

**Mon selection criteria for a 3-Mon cluster:**
- Spread across different physical hardware (not all on same model)
- Prefer nodes with SSD root disks (Mons write leveldb/rocksdb)
- Avoid nodes that are being decommissioned
- At least one Mon on a stable, high-uptime node

**Removing a Mon from a node (cleanup):**
```bash
# Stop the mon service on the node
ssh root@<node> 'systemctl stop ceph-mon@<hostname>'
ssh root@<node> 'systemctl disable ceph-mon@<hostname>'
# The mon data dir can be left in place harmlessly
```

### 6.5.2 Ceph OSD Purge (Destroyed OSD Cleanup)

Destroyed OSDs that remain in the CRUSH tree cause confusion and reduce apparent capacity. Purge them fully:

```bash
# Check for destroyed OSDs
ceph osd tree | grep destroyed

# Purge (removes from CRUSH + OSD map)
ceph osd purge <osd-id> --yes-i-really-mean-it

# Verify removal
ceph osd tree | grep osd.<id>  # should return nothing
```

This triggers PG rebalancing — monitor progress with `ceph -w` or `ceph -s | grep pgs`. Recovery rate depends on `osd_recovery_sleep` (default 0.1 = throttled to ~10% speed). For urgent recoveries, temporarily set `ceph config set osd osd_recovery_sleep 0`.

### 6.6 Schön Homelab Ceph State (as of 2026-07-03)

| Property | Value |
|----------|-------|
| Cluster ID | 204c8171-e0b1-4f40-9de2-a7cfe4ef68d9 |
| Health | HEALTH_WARN (2 OSDs slow BlueStore ops: osd.0, osd.1) |
| Total Raw | 12 TB (11 HDD + 1.1 SSD) |
| Used | 5.1 TB (42%) |
| Available | 7.0 TB |
| OSDs | 10 up, 10 in (osd.6 purged 2026-07-03) |
| Mons | 3 (proxmox5, proxmox7, proxmox4) — reduced from 7 on 2026-07-03 |
| Pools | 7 (cephfs_data, cephfs_metadata, vm_disks, .mgr, rbd, hdd_disk, tm_disks) |

**Mon reduction (2026-07-03):** Removed 4 Mons (ubuntu, proxmox1, proxmox2, proxmox3). `mon.ubuntu` was on 10.0.30.100 — not even a PVE cluster member. Keeping 3 Mons spread across different physical nodes. Procedure: `ceph mon remove <name>` per Mon, no downtime.

**OSD distribution (post-purge):**
- proxmox1: osd.1 (HDD 3.6T, 38% full)
- proxmox2: osd.0 (SSD 188G, 52%), osd.2 (SSD 233G, 57%)
- proxmox3: osd.4 (SSD 233G, 67%)
- proxmox4: osd.3 (SSD 238G, 67%)
- proxmox5: osd.5 (SSD 238G, 61%)
- proxmox6: osd.10 (HDD 982G, 82% — overfilled)
- proxmox7: osd.7 (HDD 982G, 75% — overfilled)
- ubuntu/10.0.30.100: osd.8 + osd.9 (HDD 2.7T each, ~28% — underutilized, planned for decommission)

**OSD DB Optimization (2026-07-04):** Both OSDs on 10.0.30.100 now have DB on SSD (real partitions, no loopback):
- osd.8: DB on `osd-8-db` LV (30GB, sda4 PV) via `ceph-volume lvm new-db` — up
- osd.9 (old): destroyed+purged — RocksDB was irreparably corrupt (MANIFEST corruption from loopback DB)
- osd.6 (new, recycled ID): recreated on /dev/sdf with DB on `osd-9-db` LV (30GB, sda4 PV) via `ceph-volume lvm create --block.db` — up, backfill completed from peers (size=3, no data loss)
- Swap partition (sda3) repurposed, root shrunk to 160G (automated initramfs script), sda4 (64GB LVM) holds both DB LVs
- Post-shrink cleanup (2026-07-04): initramfs premount script + GRUB rescue entry removed, GRUB + initramfs regenerated, verified no `shrink_root` references remain

**Pool-full incident (2026-07-04):** 6 pools showed 100% full (MAX_AVAIL=0) despite HDD class having 6.3 TB free. Root cause: osd.10 (982GB) hit 95% `full_ratio`, blocking all writes to PGs on that OSD. Fixed by raising `full_ratio` to 0.97 + reweighting osd.10→0.5, osd.7→0.8. All pools recovered to 68-94% with 75-181 GB avail. See Section 6.12 for diagnosis/fix procedure.

**Known issues:**
- OSD imbalance: VAR 0.20-2.13, small HDDs (osd.7/10 ~1TB) overfilled vs large HDDs (osd.6/8 ~2.8TB) underused — reweighted 2026-07-04, long-term fix = add larger OSDs
- tm_disks pool: replica=2 (no redundancy — data loss risk if one OSD fails)
- CephFS: 0 clients, 0 data — MDS daemons running but unused
- target_size_ratio=0.9 on 3 pools (cephfs_data, cephfs_metadata, vm_disks) — misconfigured, causes PG over-allocation

**Pending optimizations (not yet executed):**
1. OSD reweight to balance fill levels
2. tm_disks: increase to replica=3 or decommission pool
3. osd_memory_target: reduce from 4GB to 2-3GB on 15GB-RAM nodes
4. target_size_ratio: set realistic values per pool
5. CephFS: shut down if unused, or document purpose
6. Integrate 4× 3TB HDDs from 10.0.30.100 ZFS pool into Ceph (RAIDZ2, see Section 6.9)

See `references/cluster-audit-2026-07.md` for full audit findings and recommendations.

### 6.7 Schön Homelab PVE Version State (Post-Update 2026-07-02)

All 8 nodes successfully upgraded to PVE 9.2.3 via rolling update:

| Node | PVE | Kernel | Guests after migration |
|------|-----|--------|----------------------|
| proxmox1 | 9.2.3 | 6.17.4-2-pve | CT 105, VM 106 (homeassistant), CT 140, CT 231 |
| proxmox2 | 9.2.3 | 7.0.2-6-pve | VM 300 (mariadb-01), VM 311 (maxscale-02), CT 133 |
| proxmox3 | 9.2.3 | 7.0.2-6-pve | CT 121, CT 200, VM 129-131 (stopped) |
| proxmox4 | 9.2.3 | 7.0.12-1-pve | VM 302 (mariadb-03), VM 310 (maxscale-01), CT 135 (openwebui) |
| proxmox5 | 9.2.3 | 7.0.12-1-pve | CT 114 (smb-tm-noris, running — Time Machine target) |
| proxmox6 | 9.2.3 | 7.0.12-1-pve | VM 301 (mariadb-02), CT 108-109, CT 112-113, CT 116-117, CT 120 (frigate), CT 134 |
| proxmox7 | 9.2.3 | 7.0.12-1-pve | CT 100, CT 124 (litellm), CT 230 (hermes-agent), VM 99999 (traefik) |
| n5pro | 9.2.3 | 7.0.12-1-pve | CT 104 (paperless-ngx) |

**Kernel variance is normal** — proxmox1-3 had no kernel change in the 9.2.2→9.2.3 update path. proxmox4-7 and n5pro jumped from 7.0.0 to 7.0.12.

**HA activation status (updated 2026-07-04):** 19 HA-managed guests. CT 137 (imap), CT 114 (smb-tm), CT 136 (smb-media) all added with max_relocate=3/max_restart=2. See `references/cluster-audit-2026-07.md` for full session detail.

### 6.8 LXC Live Migration Pitfall

**Proxmox does NOT support LXC live migration** — this is a hard limitation in all supported versions (9.1.x, 9.2.x). Attempting `pvesh create /nodes/{node}/lxc/{vmid}/migrate --target {target} --online 1` fails with:

```
migration aborted: lxc live migration is currently not implemented
```

**Workarounds:**
1. **In-place update + reboot** (recommended): `apt full-upgrade` updates host packages while CTs keep running. Reboot restarts CTs on the same node. This is the standard Proxmox approach.
2. **Backup/restore**: `vzdump` CT, restore on another node with `vzrestore` — slow for large disk images but zero downtime during backup.
3. **Clone**: `pct clone <ctid> <newid> --storage <storage> --target <node>` — creates a copy on the target, then swap IPs/config.

---

## Section 6.9: ZFS-to-Ceph Disk Migration Strategy

### 6.9.1 RAIDZ Constraint: Cannot Shrink (CRITICAL)

**ZFS RAIDZ does NOT support `zpool remove` for individual disks.** The number of disks in a RAIDZ vdev is fixed at creation. Options:

1. **`zpool offline <disk>`** — Takes a disk offline (pool stays alive, reads from parity), frees the physical disk for Ceph. Pool becomes degraded but remains usable.
2. **`zpool destroy`** — Destroys the entire pool, freeing ALL disks simultaneously. Data must be migrated elsewhere first.

There is NO middle ground — you cannot remove disks one-by-one from a RAIDZ vdev while keeping the pool at reduced capacity. `zpool remove` only works for mirrors and `special`/`log`/`cache` vdevs, NOT RAIDZ.

### 6.9.2 Phased Migration: Degrade ZFS, Grow Ceph Simultaneously

Strategy when Ceph has enough free space to absorb ZFS data:

| Phase | Action | ZFS Disks Active | ZFS Parity Left | Ceph OSDs Added |
|-------|--------|-----------------|-----------------|-----------------|
| 1 | `zpool offline pool <oldest-disk>` → wipe → create Ceph OSD | N-1 | RAIDZ2: 1 left | +1 |
| 2 | Migrate ZFS data → Ceph (RBD/CephFS) | N-1 (shrinking usage) | 1 | 1 |
| 3 | `zpool destroy` → wipe remaining → create Ceph OSDs | 0 | — | +(remaining) |

**Safety constraint**: Only offline 1 disk at a time from RAIDZ2 (keeps 1 parity). Offlining 2 disks from RAIDZ2 leaves ZERO redundancy — one sector error during migration = potential data loss.

**Capacity pre-check**: Verify Ceph has enough free space for ALL ZFS data before starting:
```bash
# On Ceph admin node:
ceph df | head -5    # Check AVAIL space
# On ZFS host:
zfs list -o name,used,avail <pool>   # Check used space
# Rule: Ceph AVAIL > ZFS USED
```

### 6.9.3 Disk Health Assessment Before Migration (SMART + badblocks)

Before committing any disk to Ceph, run both SMART checks AND destructive testing:

**SMART quick-check (all disks at once)**:
```bash
for d in sda sdb sdc sdd sde sdf sdg sdh sdi; do
  echo "=== /dev/$d ==="
  smartctl -a /dev/$d 2>/dev/null | grep -E \
    "Device Model|Serial|Power_On_Hours|Reallocated_Sector|Current_Pending|Offline_Uncorrectable|UDMA_CRC|overall-health|Self-test execution" | head -12
done
```

**Key SMART thresholds for Ceph suitability**:
| Attribute | Good | Caution | Reject |
|-----------|------|---------|--------|
| Reallocated_Sector_Ct (5) | 0 | 1-10 | >10 |
| Current_Pending_Sector (197) | 0 | 1-50 | >50 |
| Offline_Uncorrectable (198) | 0 | 1-50 | >50 |
| UDMA_CRC_Error_Count (199) | 0 | 1-10 (cable) | >10 (controller) |
| Power_On_Hours (9) | <50k | 50k-90k | >90k |
| Self-test status | (0) Passed | — | (121) Read failure |

**SMART Self-test (short, 2 min)**:
```bash
smartctl -t short /dev/sdX
sleep 130
smartctl -a /dev/sdX | grep -A3 "Self-test"
# Watch for: "Completed: read failure" = FAIL
# "Completed without error" = PASS
```

**Destructive test (badblocks, 9-22h per 3TB)**:
```bash
# DESTROYS ALL DATA on the disk — only run on disks that will be repurposed!
wipefs -a /dev/sdX
badblocks -wsv -b 4096 /dev/sdX 2>&1 | tee /tmp/badblocks_sdX.log &
# -w = write mode (4 patterns: 0xaa, 0x55, 0xff, 0x00)
# -s = show progress
# -v = verbose (prints errors)
# Estimated time: ~5h per 3TB per pattern × 4 = ~20h total
# Check progress: tail -1 /tmp/badblocks_sdX.log
# Kill early if needed: kill $(pgrep badblocks)
```

**Decision matrix after testing**:
| Result | Ceph Suitable? | Action |
|--------|----------------|--------|
| SMART clean + badblocks 0 errors | ✅ Yes | Proceed as OSD |
| SMART clean + badblocks finds errors | ⚠️ Marginal | Reallocation may fix sectors. Risk acceptable for non-critical pools. |
| SMART fail (pending/uncorrectable >50) | ❌ No | Retire disk |
| SMART Self-test read failure | ❌ No | Even with few pending sectors, confirmed read failures indicate progressive degradation |

### 6.9.4 Schön Homelab 10.0.30.100 Disk Inventory (2026-07-03)

Actual disk layout (corrected from earlier "9×2.7TB" assumption):

| Disk | Model | Serial | Size | Hours | Age | SMART | Role | Ceph Ready? |
|------|-------|--------|------|-------|-----|-------|------|-------------|
| sda | Crucial MX300 SSD | 164914EEB886 | 275GB | 81,871 | 9.3y | ✅ | Boot SSD | — |
| sdb | WD30EFRX Red+ | WCC4N4VRJ5UU | 3TB | 81,786 | 9.3y | ✅ | ZFS pool | ⚠️ Old but healthy |
| sdc | WD30EZRX Green | WMC1T2256499 | 3TB | 89,262 | 10.2y | ⛔ 371 pending, 243 uncorr | Free | ❌ Defective |
| sdd | WD30EFRX Red+ | WCC4N4SVR8JL | 3TB | 57,925 | 6.6y | ⚠️ 16 pending, 15 uncorr, Self-test FAIL | Free | ❌ Failed |
| sde | WD30EFRX Red+ | WMC4N2399595 | 3TB | 104,327 | 11.9y | ✅ | Ceph osd.8 | ⚠️ Very old |
| sdf | WD30EFRX Red+ | WMC4N2400243 | 3TB | 98,475 | 11.2y | ✅ | Ceph osd.9 | ⚠️ Very old |
| sdg | WD30EFZX Purple | WX12DA01FUA7 | 3TB | 30,901 | 3.5y | ✅ | ZFS pool | ✅ Young, healthy |
| sdh | WD30EFZX Purple | WX12DA0R1NFH | 3TB | 30,901 | 3.5y | ✅ | ZFS pool | ✅ Young, healthy |
| sdi | WD30EFZX Purple | WX22DA0D7EP3 | 3TB | 30,900 | 3.5y | ✅ | ZFS pool | ✅ Young, healthy |

**ZFS Pool `pool01_n2_redundant`**: RAIDZ2, 4 disks (sdg, sdh, sdi, sdb), 10.9TB raw, 3.63TB used, 1.87TB free, 57% fragmented. Monthly scrubs clean. Contains Docker volumes, TimeMachine (~951GB), proxmox_nfs (~963GB), HomeAssistant.

**Conclusion**: Neither sdc (defective) nor sdd (self-test failure) are suitable for Ceph. The 4 healthy ZFS pool disks can only be released by destroying the pool. Recommended strategy: offline sdb (oldest, 9.3y) first as initial Ceph OSD, migrate ZFS data to Ceph, then destroy pool and claim remaining 3 disks.

### 6.9.5 Ceph OSD Creation on Recycled Disks

Once a disk is physically available (offline from ZFS or freshly inserted):

```bash
# On the target PVE node (e.g. n5pro):
# 1. Wipe all signatures
wipefs -a /dev/sdX
# If disk was ZFS, also zap partitions
sgdisk --zap-all /dev/sdX 2>/dev/null || true

# 2. Create OSD (PVE CLI)
pveceph osd create /dev/sdX
# OR via ceph-volume directly:
ceph-volume lvm create --data /dev/sdX

# 3. Verify
ceph osd tree | grep $(hostname)
ceph osd df | head -5
```

**Prerequisite**: Ceph must be installed on the target node (`pveceph install`). The node must be a PVE cluster member.

### 6.9.6 SSH Access to 10.0.30.100

```bash
# Works with Proxmox SSH key (NOT the Galera key):
ssh -i ~/.ssh/id_ed25519_proxmox -o StrictHostKeyChecking=no root@10.0.30.100 '<commands>'
```

The Galera SSH key from 1Password does NOT work for 10.0.30.100 — only the Proxmox `id_ed25519_proxmox` key grants root access.

### 6.10 Adding BlueStore DB/WAL Device to Existing OSD

When an HDD-OSD was created without a separate DB (block.db) device, all RocksDB metadata lives on the slow HDD. Adding a flash DB device post-creation dramatically speeds up metadata operations.

**⚠️ `ceph-bluestore-tool --command bluefs-bdev-new-db` DOES NOT WORK** on Ceph Squid (19.2.x). It fails with `"No valid bdev label found"` regardless of whether the target is a raw partition or LVM LV. The correct tool is `ceph-volume lvm new-db`.

#### Prerequisites
- A free flash device or partition (SSD/NVMe) on the OSD host
- The flash device must be set up as LVM (PV → VG → LV). Raw partitions do NOT work with `ceph-volume lvm new-db`.
- OSD must be stopped during the operation (brief downtime, ~1-2 min)

#### DB Sizing Guidance

| DB Size | Use Case | Notes |
|---------|----------|-------|
| 10 GB | Minimum | ⚠️ Spill risk with heavy metadata |
| 20-30 GB | Practical | ✅ Sufficient for normal workloads |
| 50 GB | Conservative | What osd.7/osd.10 use on proxmox6/7 |
| ~4% of OSD size | Formula | Ensures entire RocksDB fits on flash |

#### Procedure (Confirmed Working on Ceph 19.2.3)

```bash
# 1. Identify the OSD FSID
OSD_FSID=$(cat /var/lib/ceph/osd/ceph-<OSD_ID>/fsid)
echo "OSD FSID: $OSD_FSID"

# 2. Create LVM on the flash device (e.g. former swap partition sda3)
pvcreate /dev/sda3
vgcreate ceph-db-vg /dev/sda3
lvcreate -L 30G -n osd-<OSD_ID>-db ceph-db-vg

# 3. Stop the OSD
systemctl stop ceph-osd@<OSD_ID>

# 4. Attach DB volume (THE KEY COMMAND)
ceph-volume lvm new-db \
  --osd-id <OSD_ID> \
  --osd-fsid "$OSD_FSID" \
  --target ceph-db-vg/osd-<OSD_ID>-db

# 5. Start the OSD
systemctl start ceph-osd@<OSD_ID>

# 6. Verify (wait ~15s for OSD to come up)
sleep 15
ceph osd tree | grep osd.<OSD_ID>    # Should show "up"
ls -la /var/lib/ceph/osd/ceph-<OSD_ID>/block.db  # Should symlink to /dev/dm-*
```

#### Post-Addition Behavior

- **BlueFS spillover is EXPECTED**: Existing RocksDB SST files remain on the HDD. New writes go to the flash DB. Over time, compaction migrates data to the flash device. The `BLUEFS_SPILLOVER` warning clears naturally (hours to days).
- **OSD may show "down" briefly** after restart — it takes ~10-15s for BlueFS to initialize with the new DB device and for the OSD to report "up" to the monitors.
- **`ceph-bluestore-tool` commands fail while OSD is running** — they cannot lock the fsid file (`Device or resource busy`). This is expected; use `ceph osd metadata <id>` from a monitor node instead to verify DB device assignment.

#### Verification from Monitor Node

```bash
# Check if DB device is recognized cluster-wide
ceph osd metadata <OSD_ID> | python3 -c "
import sys, json
d = json.load(sys.stdin)
for k in ['hostname', 'rotational', 'bluefs_db_dev', 'devices']:
    print(f'  {k} = {d.get(k, \"NOT SET\")}')
"
# devices field should list BOTH the HDD and the flash device (e.g. "sda,sde")
# bluefs_db_dev may show "NOT SET" initially — metadata propagates slowly
```

#### User Preference: NO Loopback Files for DB/WAL (CRITICAL)

**The user explicitly rejects loopback files (`losetup` + `fallocate`) for Ceph DB/WAL devices.** Even though loopback-on-SSD is technically functional and reboot-safe (via systemd unit), the user considers it a hack. Always propose real partitions: repurpose swap, shrink root partition, or add a physical SSD/NVMe.

#### DB Device Migration: Loopback → Real Partition (PITFALL — RocksDB Corruption)

When migrating an OSD's DB device from loopback to a real partition, the existing RocksDB must be readable. `ceph-volume lvm new-db` reads the existing DB to shard it onto the new device. If the RocksDB is corrupt, the migration fails AND leaves the OSD in an unrecoverable state.

**Failure sequence observed (10.0.30.100, osd.9, 2026-07-03):**
1. osd.9 stopped, block.db symlink removed
2. `ceph-volume lvm new-db --target ceph-db-vg/osd-9-db-new` → FAILED: `Corruption: bad record length in file db/MANIFEST-060205`
3. Loopback file deleted + loop device detached (cleanup)
4. Old block.db symlink restored → but loopback file gone → OSD can't open DB at all
5. `ceph-bluestore-tool --command repair` → FAILED (same MANIFEST corruption)
6. OSD permanently down — must be destroyed and rebuilt

**Root cause**: The RocksDB on the loopback device was already corrupt (likely from the initial compaction when the DB device was first added). The corruption was silent — the OSD was running but with degraded metadata.

**Safe migration procedure (PREVENTS corruption loss):**

```bash
# 1. STOP OSD
systemctl stop ceph-osd@<OSD_ID>

# 2. CHECK existing RocksDB health BEFORE touching anything
ceph-bluestore-tool --command bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-<OSD_ID> 2>&1
# If this succeeds, RocksDB is readable. If it errors, DO NOT proceed — the OSD
# must be destroyed and rebuilt instead of migrated.

# 3. Attach new DB device (KEEP old device intact!)
ceph-volume lvm new-db --osd-id <OSD_ID> --osd-fsid <FSID> --target <NEW_LV>

# 4. ONLY if step 3 succeeds: start OSD and verify
systemctl start ceph-osd@<OSD_ID>
sleep 15
ceph osd tree | grep osd.<OSD_ID>  # Must show "up"

# 5. ONLY after OSD is confirmed up: clean up old DB device
# (Remove loopback file, detach loop device, remove old LVM, disable systemd service)
```

**⚠️ CRITICAL: Never delete the old DB device (loopback file, partition, LV) until the new DB is successfully attached AND the OSD is confirmed "up".** Deleting the old DB before confirming the new one works leaves the OSD with no readable RocksDB — the only recovery is `ceph osd destroy` + rebuild from peer replicas.

**When RocksDB is already corrupt (recovery procedure):**

If `ceph-bluestore-tool --command repair` fails with MANIFEST corruption, the OSD is unrecoverable. Data is safe (Ceph size=3 replication), but the OSD must be destroyed and rebuilt:

```bash
# From a monitor node (e.g. 10.0.20.91):
CEPH="ceph --conf /etc/pve/ceph.conf --keyring /etc/pve/priv/ceph.client.admin.keyring"

# 1. Destroy the OSD (marks as destroyed, triggers backfill from peers)
#    --force is REQUIRED when OSD is down and PGs are not active+clean
#    (without --force: "Error EAGAIN: OSD(s) have no reported stats")
$CEPH osd destroy <OSD_ID> --force

# 2. On the OSD host: wipe the disk and recreate OSD with DB on real partition
ceph-volume lvm zap /dev/sdX --destroy

# 3. Create new OSD with DB on real partition
#    ⚠️ Flag is --block.db, NOT --db-dev (which is unrecognized)!
ceph-volume lvm create --data /dev/sdX --block.db /dev/ceph-db-vg/osd-<OSD_ID>-db

# 4. Purge the OLD destroyed OSD from CRUSH (also needs --force when OSD is down)
$CEPH osd purge <OLD_OSD_ID> --force

# 5. Verify new OSD joins cluster (may get a RECYCLED OSD ID — this is normal!)
$CEPH osd tree | grep osd.<NEW_ID>
# Backfill runs automatically — monitor with: $CEPH -w
```

**⚠️ OSD ID Recycling**: When you destroy an OSD and create a new one on the same disk, the new OSD may receive a different (recycled) ID — e.g. destroying osd.9 and recreating gave osd.6. This is normal Ceph behavior. The OSD ID is just a number; what matters is the disk and DB assignment. Always purge the old destroyed OSD ID from CRUSH after the new one is up.

**⚠️ `ceph-volume lvm create` flag pitfall**: The flag for specifying the DB device is `--block.db`, NOT `--db-dev`. Using `--db-dev` produces a cryptic error: `unrecognized arguments: --db-dev`. This differs from `ceph-volume lvm new-db` which uses `--target`.

**⚠️ `ceph osd destroy` requires `--force` when OSD is down**: Without `--force`, you get `Error EAGAIN: OSD(s) have no reported stats, and not all PGs are active+clean`. Since the OSD is down and can't report stats, `--force` is the only way forward. Same applies to `ceph osd purge`.

**⚠️ `rbd lock remove` syntax (not `lock break`)**: To manually remove an RBD exclusive lock: `rbd --id admin lock remove <pool/image> <locker_id> <cookie>`. Parse the locker ID and cookie from `rbd lock list --format json`. However, `ceph osd blacklist add <client_addr>` is usually more effective — it forces the kernel RBD driver to error out, which releases any D-state processes holding the device.

#### Post-Shrink udev Issue: New Partition Not Visible After Boot

After an automated root partition shrink (initramfs script), the newly created partition (e.g. sda4) may appear in the partition table (`fdisk -l`) but NOT as a block device (`ls /dev/sda4`). This is a udev timing issue — the partition was created late in the boot process and udev hadn't processed it yet.

**Fix (immediate, remote via SSH):**
```bash
partprobe /dev/sda
udevadm trigger --subsystem-match=block
sleep 2
ls -la /dev/sda4  # Now visible
```

This should be added to the post-reboot verification checklist after any automated shrink.

#### Post-Shrink Cleanup: Remove Temporary Boot Infrastructure

After the automated shrink has been executed and verified (sda4 exists, root is 160G), the temporary boot infrastructure must be removed to prevent accidental re-triggering on future boots:

```bash
# 1. Remove initramfs premount script
rm /etc/initramfs-tools/scripts/init-premount/auto-shrink-root

# 2. Remove initramfs hook (if only used for shrink)
rm /etc/initramfs-tools/hooks/resize_tools

# 3. Remove GRUB rescue entry
rm /etc/grub.d/40_custom_rescue

# 4. Regenerate GRUB + initramfs (removes shrink_root from menu, cleans initramfs)
update-grub
update-initramfs -u -k all

# 5. Verify no shrink references remain
grep -r "shrink_root" /boot/grub/grub.cfg || echo "✓ clean"
grep -rl "shrink" /etc/initramfs-tools/scripts/ || echo "✓ clean"

# 6. Restore GRUB timeout to normal (if increased for debugging)
sed -i 's/GRUB_TIMEOUT=10/GRUB_TIMEOUT=2/' /etc/default/grub
update-grub
```

**⚠️ Do NOT remove the GPT backup** (`/etc/initramfs-tools/gpt_backup.bin`) — it's harmless and could be useful for future rollback. Only remove the scripts and GRUB entries that could accidentally trigger a shrink.

The partition changes (sda2=160G, sda4=LVM) are persistent and survive reboot — only the automation scripts are temporary.

If a loopback DB was already set up (mistake), tear it down and migrate to a real partition:
```bash
# 1. Stop OSD
systemctl stop ceph-osd@<OSD_ID>
# 2. Detach the loopback DB (remove symlink, ceph-volume doesn't have a clean detach)
rm /var/lib/ceph/osd/ceph-<OSD_ID>/block.db
# 3. Remove LVM structures
lvremove -y ceph-db-vg9/osd-<OSD_ID>-db
vgremove ceph-db-vg9
pvremove /dev/loop10
# 4. Detach loop device
losetup -d /dev/loop10
rm /var/lib/ceph/osd-9-db.img
# 5. Disable systemd service
systemctl disable ceph-loopback-db.service
rm /etc/systemd/system/ceph-loopback-db.service
rm -rf /etc/systemd/system/ceph-osd@<OSD_ID>.service.d
systemctl daemon-reload
# 6. Create real partition DB (see below) and start OSD
```

#### Getting Space for DB Partitions on Systems Without Free Flash

When a host has only one SSD (used for OS) and no free flash slots, space for DB partitions must be carved from the existing SSD:

**Option A: Repurpose swap partition (immediate, no reboot)**
- Most Proxmox/Ubuntu hosts have a 16-32GB swap partition that's barely used
- `swapoff /dev/sdX3` → `wipefs -a` → `pvcreate` → `vgcreate` → `lvcreate -L 30G`
- Only enough space for 1 OSD's DB (30GB from 32GB swap)

**Option B: Shrink ext4 root partition (requires reboot, RISKY without console)**
- ext4 can grow online but **CANNOT shrink online** — must be unmounted
- Root partition must be shrunk from a rescue environment (initramfs shell or Live ISO)
- **⚠️ Without IPMI/KVM/physical access, a failed resize = permanent loss of the machine**
- Two approaches:
  - **Manual** (console required): GRUB entry with `break=premount` → initramfs shell → run e2fsck + resize2fs + fdisk by hand → `continue`
  - **Automated** (unattended, console recommended as fallback): GRUB entry with `shrink_root=1` → initramfs premount script runs e2fsck → resize2fs → sgdisk automatically → continues boot. See `references/initramfs-auto-shrink-template.md` for complete templates, setup sequence, and 6 pitfalls learned during development.
- **Risk mitigation**: Hard reset during any phase of the automated shrink is survivable — all intermediate states are bootable. `grub-reboot` (one-time) auto-reverts to normal boot. GPT backup embedded in initramfs for rollback.
- **Key pitfalls** (see reference for full detail):
  1. initramfs hook must `. /usr/share/initramfs-tools/hook-functions` for `copy_exec`
  2. GPT backup must be copied INTO initramfs (root FS not mounted in premount)
  3. GRUB entry must NOT have `break=premount` (would stop boot instead of running script)
  4. Use `grub-reboot` (one-time) not `grub-set-default` (persistent) for rescue boot

**Option C: Add a physical SSD/NVMe (recommended for production)**
- Small 256-512GB NVMe (~40€) provides 180-300GB for DB
- Supports 6-10 OSDs at 30GB each
- No risk to existing data, no downtime

#### Repurposing Swap Partition for DB (Schön Homelab)

On 10.0.30.100, the 32GB swap partition (sda3) was repurposed for osd.8's DB device:
- Swap had only 1.5GB used, 6.4GB RAM free → safe to disable
- `swapoff /dev/sda3` → `wipefs -a /dev/sda3` → LVM → `ceph-volume lvm new-db`
- Only enough space for ONE OSD's DB (30GB LV). osd.9's DB was initially set up via loopback (rejected by user) — requires root partition shrink or additional SSD.

#### Journal Cleanup to Free Root Space

Before allocating root-SSD space for DB, free up wasted space:
```bash
journalctl --vacuum-size=500M    # Freed 3.5GB on 10.0.30.100
apt-get clean                     # Freed 322MB
```

---

---


## References

- `references/lxc-gpu-device-bind-rules.md` — Copy-paste config snippets per GPU vendor
- `references/lxc-gpu-autosetup-background-script.md` — Background install pattern with completion markers
- `references/comfyui-model-directories.md` — ComfyUI, Ollama, diffusers directory layout
- `references/cluster-audit-2026-07.md` — Cluster audit findings
- `references/cluster-analysis-2026-07-12.md` — July 12 analysis: n5pro, Ceph osd.10 full, K8s VM balloon, perf optimizations
- `references/ceph-pool-full-recovery-2026-07.md` — Pool-full recovery + OSD prep
- `references/zfs-to-ceph-migration-2026-07.md` — Disk inventory, SMART data
- `references/ceph-pg-osd-management-2026-07.md` — PG reduction, OSD reweight, concurrent I/O
- `references/initramfs-auto-shrink-template.md` — Automated ext4 root partition shrink via initramfs (premount script, hook, GRUB entry, setup sequence, 6 pitfalls)
- `scripts/lxc-background-package-install.sh` — Background ROCm/ComfyUI install script
- `templates/nut-master-ups.conf` — ups.conf driver config for APC Back-UPS ES 700G
- `templates/nut-slave-upsmon.conf` — upsmon.conf template for Proxmox NUT slave nodes
- `templates/nut-master-upsmon.conf` — upsmon.conf template for NUT master
- `templates/nut-master-upsd-users.conf` — upsd.users with master+slave users
- `templates/nut-upssched.conf` — upssched.conf for timed FSD trigger
- `templates/upssched-cmd.sh` — upssched command script
- `templates/ups-shutdown.sh` — Guest shutdown script for Proxmox nodes

---

## Section 7: Ceph Erasure Coding Pools & LXC Integration

### 7.1 Creating an EC Pool for RBD Use

Erasure Coding (EC) pools provide storage efficiency (k+m chunks instead of full replicas) but require a **replicated metadata pool** for RBD. The EC pool stores data blocks; the metadata pool stores RBD object maps and journaling.

#### EC Profile Creation

```bash
CEPH="ceph --conf /etc/pve/ceph.conf --keyring /etc/pve/priv/ceph.client.admin.keyring"

# Create EC profile: k=4 data chunks, m=1 parity chunk, HDD-only, failure_domain=osd
$CEPH osd erasure-code-profile set ec41-hdd \
  plugin=jerasure \
  technique=reed_sol_van \
  k=4 m=1 \
  crush-failure-domain=osd \
  crush-device-class=hdd
```

| Parameter | Meaning |
|-----------|---------|
| `k=4` | 4 data chunks (stripe width) |
| `m=1` | 1 parity chunk (survives 1 OSD failure) |
| `crush-failure-domain=osd` | Chunks spread across OSDs (not hosts) |
| `crush-device-class=hdd` | Only HDD OSDs used (excludes SSD) |

**⚠️ k+m is FIXED at pool creation.** You cannot change it later (e.g. 4+1 → 5+1) without creating a new pool and migrating data. Adding more OSDs later improves distribution but keeps the same k+m.

**Choosing failure_domain:** With only 4 hosts but 5+ OSDs, `failure_domain=osd` is required (need ≥ k+m = 5 failure domains). `failure_domain=host` would need ≥5 hosts.

#### Pool Creation

```bash
# EC data pool (stores actual object data)
$CEPH osd pool create media_ec 128 128 erasure ec41-hdd

# Metadata pool (replicated, stores RBD metadata)
$CEPH osd pool create media_meta 32 32 replicated

# Enable RBD application on both pools
$CEPH osd pool application enable media_ec rbd
$CEPH osd pool application enable media_meta rbd

# CRITICAL: Allow EC overwrites (required for RBD write support)
$CEPH osd pool set media_ec allow_ec_overwrites true
```

**⚠️ `allow_ec_overwrites true` is mandatory** for RBD use. Without it, RBD images on EC pools are read-only and container creation/mount fails.

### 7.2 Registering EC-Backed RBD Storage in PVE

```bash
pvesm add rbd media \
  --pool media_meta \
  --data-pool media_ec \
  --username admin \
  --monhost 10.0.20.50:6789,10.0.20.70:6789,10.0.20.40:6789
```

- `--pool` = metadata pool (replicated)
- `--data-pool` = EC pool (actual data)
- PVE creates RBD images on the metadata pool; data blocks are stored on the EC pool automatically

**Verify:**
```bash
pvesm status --storage media
# Should show "active" with available space
```

### 7.3 Adding `rootdir` Content Type to RBD Storage (CRITICAL)

Newly created RBD storages default to `images` content type only. **LXC mount points (`mp0`, `mp1`, ...) require `rootdir` content type.** Without it, `pct set -mp0 <storage>:<size>` fails with:

```
storage 'media' does not allow content type 'rootdir' (Container)
```

**Fix:**
```bash
pvesm set media --content rootdir
# Verify:
grep -A8 "rbd: media" /etc/pve/storage.cfg
# Should show: content rootdir
```

### 7.4 Creating LXC CT with EC-Backed Mount Point

#### Root Disk: Must Use Non-EC Storage

EC-backed RBD storage typically only supports `rootdir` (for mount points) and `images` (for VMs). The container **root filesystem** needs a storage that supports `rootdir` — use an existing replicated RBD storage (e.g. `vm_disks`, `hdd_disk`):

```bash
pct create 136 hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --arch amd64 \
  --hostname smb-media \
  --memory 512 --swap 0 --cores 1 \
  --rootfs vm_disks:8 \
  --net0 name=eth0,bridge=vmbr0,ip=10.0.30.30/24,gw=10.0.30.1,tag=30,type=veth \
  --unprivileged 1 --features nesting=1
```

#### Mount Point: EC-Backed RBD

```bash
pct set 136 -mp0 media:500,mp=/mnt/media
```

**⚠️ `pct set -mp0 <storage>:<size>` creates AND formats the RBD image.** Do NOT use `pvesm alloc` first — it creates an unformatted RBD image, and the container will fail to start with `Script exited with status 32` (pre-start hook fails because the filesystem doesn't exist).

**If you already used `pvesm alloc` (recovery):**
```bash
# 1. Remove the mp0 config
pct set 136 -delete mp0
# 2. Free the unformatted volume
pvesm free media:vm-136-disk-1
# 3. Re-add via pct set (creates + formats)
pct set 136 -mp0 media:500,mp=/mnt/media
```

### 7.5 Template Download Gotchas

**Templates are per-node, not cluster-wide** (unless on shared storage). Each node that creates a CT needs the template downloaded to a local or shared template storage:

```bash
# Check which storages support vztmpl on the target node
pvesm status | grep -iE "template|vztmpl"

# Download to the correct storage (NOT "local" if disabled)
pveam download hdd_templates debian-12-standard_12.12-1_amd64.tar.zst
```

**⚠️ `local` storage is often disabled** on nodes that use shared/ceph storage. Use the node's active template storage (e.g. `hdd_templates`).

### 7.6 CFS Lock Timeouts Under Ceph Load

When Ceph is recovering/backfilling, `pct create` can fail with:

```
unable to create CT <id> - cfs-lock 'storage-<name>' error: got lock request timeout
```

**Mitigation:** Try a different storage for the root disk, or wait for Ceph recovery to complete. The lock contention comes from PVE's pmxcfs (configuration filesystem) struggling under I/O pressure from Ceph.

### 7.7 Unprivileged CT Limitations: No NFS Mounts

Unprivileged LXC containers **cannot mount NFS shares** (`mount.nfs: Operation not permitted`). For data migration into an unprivileged CT:

1. **Use rsync over SSH** instead of NFS mount
2. Set up SSH access to the CT (see Section 7.7.1 below)
3. Run rsync from the source host targeting the CT's IP

### 7.7.1 SSH Setup in Unprivileged LXC CTs for Rsync Migration

Freshly created LXC CTs from Debian templates have **no SSH server installed** and **no root password set**. To enable rsync-over-SSH migration:

```bash
# 1. Install SSH server + rsync in the CT (via pct exec, no SSH needed yet)
pct exec <ctid> -- apt-get install -y -qq openssh-server rsync

# 2. Set root password (for initial sshpass-based rsync)
pct exec <ctid> -- bash -c "echo root:TEMPASSWORD | chpasswd"

# 3. Enable root login + password auth in sshd_config
pct exec <ctid> -- bash -c "sed -i 's/^#PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config"
pct exec <ctid> -- bash -c "sed -i 's/^PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config"
pct exec <ctid> -- bash -c "sed -i 's/^#PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config"
pct exec <ctid> -- bash -c "sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config"
pct exec <ctid> -- systemctl restart ssh

# 4. On source host: install sshpass, then rsync with password
sshpass -p "TEMPASSWORD" rsync -a --partial \
  -e "sshpass -p TEMPASSWORD ssh -o StrictHostKeyChecking=no" \
  /source/path/ root@<ct-ip>:/dest/path/
```

**⚠️ Alternative: SSH key via `pct exec` (no password needed):**
```bash
# Add source host's pubkey to CT authorized_keys
pct exec <ctid> -- mkdir -p /root/.ssh
pct exec <ctid> -- bash -c "echo 'ssh-ed25519 AAAA... root@source' >> /root/.ssh/authorized_keys"
pct exec <ctid> -- chmod 600 /root/.ssh/authorized_keys
# Still need PermitRootLogin yes in sshd_config!
```

**⚠️ Bridge name pitfall:** Freshly created CTs may default to `bridge=vmbr1` which doesn't exist on all nodes. Check existing CTs (`pct config <existing-ct> | grep net`) for the correct bridge name. In the Schön homelab, all CTs use `bridge=vmbr0` with `tag=30` for VLAN 30.

**⚠️ `sshpass` rsync syntax:** The `-e` flag must wrap the entire SSH command including sshpass:
```bash
# CORRECT:
rsync -a -e "sshpass -p PASS ssh -o StrictHostKeyChecking=no" /src/ root@ip:/dst/

# WRONG (sshpass not passed through to rsync's SSH transport):
sshpass -p PASS rsync -a /src/ root@ip:/dst/
```

**⚠️ `pct push` writes NULL bytes when source file is missing/unreadable (CRITICAL):** If you SCP a file to the Proxmox host and the SCP fails (e.g. "Failure" on remote write), `pct push` will happily push a zero-byte or null-filled file to the CT — no error, exit 0. The resulting file contains thousands of `\x00` bytes instead of the intended content. Always verify `pct push` output with `pct exec <ctid> -- cat <dest>` and check for null bytes.

**⚠️ Shell `$` characters in SHA512-CRYPT password hashes get eaten through nested bash/SSH (CRITICAL):** Writing Dovecot password files containing `{SHA512-CRYPT}$6$salt$hash` through `pct exec -- bash -c "cat > file << EOF ..."` strips all `$` characters because bash interprets them as variable references even inside double-quoted heredocs. The resulting hash is truncated and unusable.

**Reliable method — base64 round-trip:**
```bash
# 1. Write the file locally (preserves $ characters)
cat > /tmp/dovecot-users << 'LOCAL_EOF'
user@domain.com:{SHA512-CRYPT}$6$salt$hash...
LOCAL_EOF

# 2. Base64 encode
B64=$(base64 /tmp/dovecot-users | tr -d '\n')

# 3. Decode inside CT via pct exec
pct exec <ctid> -- bash -c "echo $B64 | base64 -d > /etc/dovecot/users"

# 4. Verify — check for intact $ characters
pct exec <ctid> -- cat /etc/dovecot/users
# Should show: user@domain.com:{SHA512-CRYPT}$6$salt$hash...
```

**Why this works:** Base64 encoding eliminates all special characters before transit. The `base64 -d` inside the CT restores the exact bytes including `$`, `{`, `}` characters. No shell interpretation occurs on the encoded payload.

**Alternative:** Write the file locally, SCP to the Proxmox host, then `pct push` — but verify the SCP succeeded AND the pushed content is non-null before trusting it.

### 7.7.2 Background Rsync with sshpass for Large Mailbox Migration

For migrating mailboxes (Dovecot maildir format) from a Docker volume to a CT:

```bash
# On source host (e.g. 10.0.30.100):
nohup bash -c "
rsync -a --partial -e \"sshpass -p TEMPASSWORD ssh -o StrictHostKeyChecking=no\" \
  /var/lib/docker/volumes/mailserver/mail/vhosts/ \
  root@<ct-ip>:/var/mail/vhosts/ 2>&1
echo VHOSTS_DONE:\$?
" > /tmp/rsync-vhosts.log 2>&1 &
```

Monitor: `ps aux | grep rsync` and `sshpass -p PASS ssh root@<ct-ip> "du -sh /var/mail/vhosts/"`

### 7.7.3 Dovecot IMAP Setup in LXC on Ceph EC

Pattern for standing up an IMAP server in an unprivileged LXC CT with mail data on Ceph EC 4+1:

```bash
# 1. Create CT with rootfs on replicated RBD, mail data on EC RBD
pct create <ctid> hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --hostname imap --cores 2 --memory 2048 --swap 512 \
  --rootfs vm_disks:8 \
  --net0 name=eth0,bridge=vmbr0,tag=30,ip=10.0.30.40/24,gw=10.0.30.1 \
  --unprivileged 1 --features nesting=1 --ostype debian

# 2. Add EC-backed mount point for mail data
pct set <ctid> -mp0 media:20,mp=/var/mail

# 3. Install Dovecot
pct exec <ctid> -- apt-get install -y -qq dovecot-imapd dovecot-pop3d

# 4. Configure maildir with vhost domain structure
pct exec <ctid> -- bash -c "cat > /etc/dovecot/conf.d/99-mail.conf << 'EOF'
mail_location = maildir:/var/mail/vhosts/%d/%n/mail
passdb {
  driver = passwd-file
  args = scheme=PLAIN username_format=%u /etc/dovecot/users
}
userdb {
  driver = static
  args = uid=5000 gid=5000 home=/var/mail/vhosts/%d/%n
}
listen = *
EOF"

# 5. Create vmail user + directory structure
pct exec <ctid> -- groupadd -g 5000 vmail
pct exec <ctid> -- useradd -u 5000 -g 5000 -d /var/mail -s /usr/sbin/nologin vmail
pct exec <ctid> -- mkdir -p /var/mail/vhosts/<domain>
pct exec <ctid> -- chown -R 5000:5000 /var/mail

# 6. Create users file (passwords TBD — set real passwords later)
pct exec <ctid> -- bash -c "cat > /etc/dovecot/users << 'EOF'
user@domain.com:{PLAIN}changeme
EOF"

# 7. Restart + verify
pct exec <ctid> -- systemctl restart dovecot
pct exec <ctid> -- ss -tlnp | grep -E "143|993"
```

**Maildir structure mapping:** Docker mailserver volumes use `mail/vhosts/<domain>/<user>/mail/` — Dovecot's `mail_location = maildir:/var/mail/vhosts/%d/%n/mail` maps directly to this structure. `%d` = domain, `%n` = user part of email address.

### 7.7.4 Dovecot Password Management (Generate + Reset Hashes)

#### Generating a SHA512-CRYPT Hash for Dovecot Users

Use `doveadm pw` inside the CT to generate password hashes compatible with the passwd-file auth backend:

```bash
# Generate a single hash
pct exec <ctid> -- doveadm pw -s SHA512-CRYPT -p "PLAINTEXT_PASSWORD"
# Output: {SHA512-CRYPT}$6$salt$hash...

# Generate hash and build users file for all accounts in one step
HASH=$(pct exec <ctid> -- doveadm pw -s SHA512-CRYPT -p "PLAINTEXT_PASSWORD")
for user in dominik sarah newsletter dokumente sarah-coc honig; do
  echo "${user}@familie-schoen.com:${HASH}"
done
```

#### Writing Password Files (base64 Round-Trip — CRITICAL)

Shell `$` characters in SHA512-CRYPT hashes (`$6$salt$hash`) get eaten through nested bash/SSH/pct-exec. The base64 round-trip from Section 7.7.1 is the reliable method:

```bash
# 1. Build the users file locally (preserves $ characters)
cat > /tmp/dovecot-users << 'LOCAL_EOF'
user@domain.com:{SHA512-CRYPT}$6$salt$hash...
LOCAL_EOF

# 2. Base64 encode
B64=$(base64 /tmp/dovecot-users | tr -d '\n')

# 3. Decode inside CT via pct exec
pct exec <ctid> -- bash -c "echo $B64 | base64 -d > /etc/dovecot/users"

# 4. Verify — check for intact $ characters
pct exec <ctid> -- cat /etc/dovecot/users
# Should show: user@domain.com:{SHA512-CRYPT}$6$salt$hash...

# 5. Restart Dovecot
pct exec <ctid> -- systemctl restart dovecot
```

#### Testing IMAP Login

```bash
# Quick IMAP login test via netcat
pct exec <ctid> -- bash -c \
  "echo -e '1 LOGIN user@domain.com PLAINTEXT_PASSWORD\n2 LIST \"\" \"INBOX\"\n3 LOGOUT' | timeout 5 nc localhost 143"
# Success: "1 OK ... Logged in" + "2 OK List completed"
# Failure: "1 NO [AUTHENTICATIONFAILED] Authentication failed"
```

#### Setting Uniform Password for All Accounts

To reset all mailboxes to the same password (e.g. during migration):

```bash
# 1. Generate hash once
HASH=$(pct exec <ctid> -- doveadm pw -s SHA512-CRYPT -p "NEW_PASSWORD")

# 2. Build users file locally with the same hash for all accounts
# 3. Base64 round-trip to CT (see above)
# 4. Restart Dovecot
# 5. Test login for each account
```

### 7.8 Large Data Migration: Background Rsync Pattern

For large transfers (>100GB) that exceed tool timeouts, use nohup background jobs:

```bash
# On source host:
nohup bash -c "
  rsync -a --partial /pool/source/Serien/ root@<ct-ip>:/mnt/media/Serien/
  echo SERIEN_DONE:$?
  rsync -a --partial /pool/source/XXX/ root@<ct-ip>:/mnt/media/XXX/
  echo XXX_DONE:$?
" > /tmp/rsync-media.log 2>&1 &
```

Monitor: `tail -f /tmp/rsync-media.log` or `ps aux | grep rsync`

**⚠️ Avoid duplicate rsync processes.** If a foreground rsync times out and you start a background one, kill the old process first — two rsync processes writing to the same destination can corrupt files.

### 7.9 Samba Setup in LXC for Media Shares

```bash
# Install
pct exec <ctid> -- apt-get install -y -qq samba avahi-daemon

# Create share directories
pct exec <ctid> -- mkdir -p /mnt/media/Filme /mnt/media/Serien /mnt/media/XXX

# Create SMB user
pct exec <ctid> -- bash -c "echo -e 'PASSWORD\nPASSWORD' | smbpasswd -a root -s"

# Write smb.conf (use tee to avoid heredoc-through-SSH escaping issues)
pct exec <ctid> -- tee /etc/samba/smb.conf > /dev/null << 'SMBCONF'
[global]
   workgroup = WORKGROUP
   server role = standalone
   security = user
   map to guest = Bad User
   min protocol = SMB2

[Media]
   path = /mnt/media
   comment = Media Share
   browseable = yes
   writable = yes
   guest ok = no
   valid users = root
   force user = root
   create mask = 0777
   directory mask = 0777
SMBCONF

# Restart and verify
pct exec <ctid> -- systemctl restart smbd nmbd
pct exec <ctid> -- systemctl is-active smbd nmbd
pct exec <ctid> -- ss -tlnp | grep -E ":445|:139"
```

**Access:** `\\<ct-ip>\Media` with the configured SMB user.

### 6.12 Ceph Pool-Full Diagnosis & OSD Reweighting (CRITICAL)

#### Symptom: Pools Show 100% Full Despite Free Space in Device Class

`ceph df` shows pools at 100% with `MAX AVAIL = 0`, but the raw device class (e.g. HDD) has terabytes of free space. This is caused by **individual OSDs hitting `full_ratio`** (default 95%).

**Diagnosis:**
```bash
ceph osd df
# Look for OSDs at 95%+ — these trigger the full condition
# Example: osd.10 at 95.00%, osd.7 at 90.26% while osd.6 at 8.92%
```

**Mechanism:** When any OSD reaches `full_ratio` (default 0.95), Ceph blocks ALL writes to PGs mapped to that OSD. Since nearly every pool has PGs on every OSD, ALL pools show 100% full — even though other OSDs in the same device class have plenty of space. The `MAX AVAIL` column drops to 0 for affected pools.

**Root cause:** CRUSH distributes PGs proportionally to OSD weight. Small OSDs (e.g. 1TB) fill up faster than large OSDs (e.g. 3TB) when data grows uniformly. Uneven disk sizes in the same CRUSH hierarchy cause this.

#### Fix: Raise full_ratio + Reweight Overloaded OSDs

```bash
# 1. Temporarily raise full_ratio to unblock writes and allow backfill
ceph osd set-full-ratio 0.97

# 2. Drain overloaded OSDs by reducing their reweight (0.0-1.0)
ceph osd reweight <osd-id> 0.5    # Aggressive drain
ceph osd reweight <osd-id> 0.8    # Moderate drain

# 3. Automatic balancing (alternative to manual reweight)
ceph osd reweight-by-utilization
# Moves PGs from over-utilized to under-utilized OSDs automatically

# 4. Monitor backfill progress
ceph -s    # Watch for "recovery" / "backfill" activity
ceph osd df  # Verify UTIL% decreasing on drained OSDs

# 5. Once balanced, restore full_ratio to default
ceph osd set-full-ratio 0.95
```

**⚠️ `backfill_toofull` pitfall:** If an OSD is AT `full_ratio`, Ceph cannot backfill data OFF that OSD (writes are blocked, including rebalance writes). Raising `full_ratio` to 0.97 gives the 2% headroom needed for backfill to proceed. Without this, `ceph -s` shows `Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull`.

**Effect on pool availability (observed 2026-07-04):**

| Pool | Before (full_ratio=0.95) | After (full_ratio=0.97 + reweight) |
|------|------------------------|------------------------------------|
| media_ec | 100%, 0 B avail | 68%, 181 GB avail |
| vm_disks | 70%, 101 GB avail | 68%, 108 GB avail |
| hdd_disk | 100%, 0 B avail | 94%, 75 GB avail |
| tm_disks | 100%, 0 B avail | 74%, 113 GB avail |

**Long-term fix:** Add more OSDs to the device class, or replace small OSDs with larger disks. Reweighting is a tactical fix — the imbalance recurs as data grows.

### 6.13 ZFS Dataset Destruction: Busy Processes & Async Free

#### Destroying a ZFS Dataset with Blocking Processes

`zfs destroy -r <pool/dataset>` fails with `cannot unmount: pool or dataset is busy` when processes have their cwd or open files in the dataset.

**Find and kill blockers:**
```bash
# Find processes with cwd or open files in the dataset
lsof +D /pool/dataset/         # Lists all processes accessing the dataset
fuser -mv /pool/dataset/        # Alternative: shows PIDs and access type

# Kill all blocking processes (including background monitors!)
kill -9 <PID1> <PID2> ...

# Retry destroy
zfs destroy -r <pool/dataset>
```

**⚠️ Background monitor processes trap:** If you launched a background polling loop (e.g. watching ZFS async free progress), that process has its cwd in the dataset and will block destruction. Always kill background monitors before destroying.

**⚠️ `zfs destroy` enters D-state (uninterruptible sleep):** For large datasets (hundreds of GB), `zfs destroy` blocks on I/O and cannot be killed (SIGKILL immune). On degraded RAIDZ2 pools with defective disks, this can take 10+ minutes. Do NOT assume it's hung — monitor with `ps aux | grep "zfs destroy"` and wait.

#### ZFS Async Free Behavior

After `rm -rf` on a ZFS dataset, `zfs list` shows high `USED` long after `du` shows the data is gone. ZFS frees blocks asynchronously via transaction group (TXG) commits.

- `du -sh /pool/dataset/` shows actual live data (e.g. 3 GB)
- `zfs list -o used <pool/dataset>` shows ZFS accounting (e.g. 290 GB)
- `zpool sync <pool>` forces a TXG sync but may hang/timeout during heavy I/O
- The space is reclaimed eventually — patience is required
- Destroying the dataset (`zfs destroy -r`) also triggers async free of all blocks

**Diagnostic pattern:**
```bash
# Confirm data is actually gone
du -sh /pool/dataset/           # Live data (should be small)
zfs list -o used,refer <pool>   # ZFS accounting (may lag behind)
# Gap between du and zfs = pending async free
```

### 6.14 Moving OSDs Between Nodes

There is no `ceph osd move` command. Three approaches exist, depending on whether the physical disk moves or only the CRUSH position changes.

#### Option A: CRUSH Position Change Only (Disk Stays Physically)

```bash
# Move OSD to different host in CRUSH tree (does NOT move data)
ceph osd crush set osd.X <weight> root=default host=<new-host>
```

⚠️ This tells Ceph the OSD is on a different host, but the disk hasn't moved. Data is NOT redistributed. Only use for CRUSH topology corrections, not physical migrations.

#### Option B: Physical Disk Migration (No Data Loss)

When physically moving a disk + its DB device to a new node:

```bash
# On old node:
ceph osd set noout
systemctl stop ceph-osd@X
ceph osd out osd.X
ceph osd crush remove osd.X

# Physically move disk(s) to new node

# On new node:
ceph-volume lvm scan          # Finds existing OSD on the disk
ceph-volume lvm activate X    # Reactivates OSD with same ID + data intact
ceph osd crush add osd.X <weight> root=default host=<new-node>
ceph osd unset noout
```

**⚠️ DB device must move too.** If the DB is on a separate SSD partition (e.g. `ceph-db-vg/osd-X-db`), that SSD must also move to the new node, or the OSD cannot start. If only the HDD moves without its DB SSD, use Option C instead.

**⚠️ `ceph-volume lvm scan` only works** if the LVM metadata on the disk is intact. If the disk was wiped or LVM headers damaged, the OSD must be rebuilt (Option C).

#### Option C: Destroy + Recreate (Standard, Data Via Backfill)

The officially supported method. Data is rebuilt from peer replicas via backfill:

```bash
# On monitor node:
ceph osd destroy X --force    # --force required when OSD is down
ceph osd purge X --force      # Removes from CRUSH

# On new node (disk installed):
ceph-volume lvm create --data /dev/sdX --block.db /dev/ceph-db-vg/osd-X-db
```

See Section 6.10 for `--block.db` flag details and OSD ID recycling notes.

#### Decision Matrix

| Scenario | Option | Data Loss | Downtime |
|----------|--------|-----------|----------|
| Only CRUSH topology fix | A | None | None |
| Physical disk + DB SSD move together | B | None | Brief (stop/start) |
| Only HDD moves (DB on old node's SSD) | C | None (backfill) | None (async) |
| Disk wiped or LVM damaged | C | None (backfill) | None (async) |
| Moving to node without Ceph installed | C | None (backfill) | None (async) |

### 7.10 EC Pool Expansion: Adding More OSDs Later

EC pools automatically rebalance when new OSDs are added — no pool reconfiguration needed:

```bash
# Add new HDD OSD on n5pro (or any node)
pveceph osd create /dev/sdX

# EC pool redistributes data across all HDD OSDs automatically
# Monitor progress:
$CEPH -w  # Watch for recovery/backfill completion
```

The k=4, m=1 structure stays the same; only the per-OSD utilization improves. To change k+m (e.g. to 5+1), a new pool must be created and data migrated.

### 7.11 ZFS Pool Inventory Assessment

When evaluating a ZFS pool for migration to Ceph, gather complete dataset information:

```bash
# Pool overview
zpool list
zpool status

# Dataset tree with sizes
zfs list -o name,used,refer -r <pool>

# Key datasets to assess:
# - Docker storage (often huge with ZFS driver, 700+ subdatasets)
# - TimeMachine backups (sparsebundles, large but mostly snapshots)
# - Home directories (user data, maildirs)
# - Download/media directories (migration candidates)
# - Docker volumes (named volumes with app data)
```

**Size interpretation:**
- `USED` includes snapshots/clones — space held by deleted data still referenced by snapshots
- `REFER` is live data only
- `USED − REFER` = snapshot overhead (reclaimable by destroying old snapshots)

**Common findings:**
- Docker ZFS storage driver creates one subdataset per layer — 700+ subdatasets is a sign of layer proliferation and correlates with daemon instability
- TimeMachine sparsebundles show high USED but low REFER — old snapshots hold most of the space
- Dedup ratio (`zfs list -o dedupratio`) near 1.0× means dedup is ineffective

### 7.12 Ceph OSD Moving Between Nodes

See Section 6.11 for the detailed decision matrix and procedures for moving OSDs between nodes (CRUSH-only, physical disk migration, and destroy+recreate).

**Key takeaway:** There is no `ceph osd move` command. Physical disk migration with `ceph-volume lvm scan/activate` preserves data but requires the DB device to move too. Destroy+recreate is the standard approach when DB devices can't move — data is rebuilt from peer replicas via backfill.