dominik/hermes-skills

Fork 0

Files

T

Debian e9cc106625 Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00

19 KiB

Raw Blame History

Proxmox Backup Server (PBS) in LXC — Setup & PVE Integration

When to Use

Building or rebuilding a Proxmox Backup Server inside an unprivileged LXC container on Ceph RBD storage, then connecting it to PVE nodes as a backup storage target.

Architecture Overview

PBS runs as CT 116 (hostname proxmox-backup-server, IP 10.0.30.106, VLAN 30)
Rootfs: 10 GB RBD on Ceph vm_disks pool
Backup datastores: NFS-mounted raw files (.raw) on nfs_ubuntu storage
HA-managed via ha-manager

Critical: Unprivileged LXC on Ceph RBD

Problem

pct create with --rootfs vm_disks:vm-116-disk-0,size=10G on a Ceph RBD pool fails in two ways:

If RBD doesn't exist yet: rbd error: error opening image vm-116-disk-0: (2) No such file or directory — pct create tries to create the RBD internally but fails on some Ceph configurations.
If RBD pre-created and formatted ext4: tar: ./etc: Cannot mkdir: Permission denied — the unprivileged container's UID mapping (0→100000) conflicts with the freshly formatted ext4 root directory (owned by real uid 0).

Solution: Pre-create, Format, Chown, Then pct create

# Step 1: Create RBD from a Ceph monitor node (not all PVE nodes have ceph.conf)
rbd create -p vm_disks vm-116-disk-0 --size 10G

# Step 2: Map, format ext4, chown root to 100000:100000, unmap
rbd map -p vm_disks vm-116-disk-0    # → /dev/rbdN
mkdir -p /tmp/fix-ct
mount /dev/rbdN /tmp/fix-ct
chown 100000:100000 /tmp/fix-ct
chmod 755 /tmp/fix-ct
umount /tmp/fix-ct
rbd unmap /dev/rbdN

# Step 3: Clean up any stale pct state
rm -f /etc/pve/lxc/NNN.conf

# Step 4: pct create (RBD already exists, formatted, and chowned)
pct create NNN hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --rootfs vm_disks:vm-NNN-disk-0,size=10G \
  --hostname proxmox-backup-server \
  --memory 4096 --cores 4 \
  --features keyctl=1,nesting=1 \
  --net0 name=eth0,bridge=vmbr0,gw=10.0.30.1,ip=10.0.30.106/24,tag=30,type=veth \
  --unprivileged 1 --onboot 1 --swap 512

Why This Works

lxc-usernsexec -m u:0:100000:65536 maps container root (uid 0) to host uid 100000. When tar extracts the template, it creates files as uid 0 (inside the namespace) = uid 100000 (on the host filesystem). If the ext4 root directory is owned by uid 0 on the host, the mapped uid 100000 lacks permission to create entries. Chowning the root directory to 100000:100000 before pct create grants the mapped uid write access.

Privileged Container Alternative

If the chown workaround fails or is undesirable, dropping --unprivileged 1 avoids the UID mapping entirely. However, unprivileged is recommended for security — PBS does not require privileged mode.

PBS Installation Inside LXC

Step 1: Add PBS Repository

# Inside the CT (via pct exec)
echo 'deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription' > /etc/apt/sources.list.d/pbs.list
wget -q -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg http://download.proxmox.com/debian/proxmox-release-bookworm.gpg
apt-get update -qq

Step 2: Install PBS Packages

DEBIAN_FRONTEND=noninteractive apt-get install -y -qq proxmox-backup-server proxmox-backup-client

This pulls in ZFS, LVM, thin-provisioning-tools as dependencies (normal for PBS).

Step 3: Register Existing Datastores

If the datastore paths already contain backup data (rebuild scenario), proxmox-backup-manager datastore create fails with "datastore path not empty". Instead, write the config directly:

cat > /etc/proxmox-backup/datastore.cfg << 'EOF'
datastore: s3-backup2
    path /mnt/datastore/s3-backup2

datastore: s3-backup
    path /mnt/datastore/s3-backup

datastore: noris
    path /mnt/datastore/noris
EOF

systemctl restart proxmox-backup-proxy proxmox-backup

Then verify:

proxmox-backup-manager datastore list

Step 4: Create Admin User

proxmox-backup-manager user create admin@pbs --password <PASSWORD>
proxmox-backup-manager acl update / Admin --auth-id admin@pbs

⚠️ The role name is Admin (capital A), not Administrator or admin. Using the wrong name gives a confusing "value not defined in enumeration" error.

Verify authentication:

curl -sk -d username=admin@pbs -d password=<PASSWORD> \
  https://localhost:8007/api2/json/access/ticket

Should return JSON with a ticket and CSRF token.

Step 5: Add NFS Datastore Mounts

Add mount points to the CT config (from a PVE node):

pct set 116 \
  -mp0 nfs_ubuntu:116/vm-116-disk-2.raw,mp=/mnt/datastore/s3-backup2,size=500G \
  -mp1 nfs_ubuntu:116/vm-116-disk-3.raw,mp=/mnt/datastore/s3-backup,size=192G \
  -mp2 nfs_ubuntu:116/vm-116-disk-0.raw,mp=/mnt/datastore/noris,size=500G

PVE Integration: Connecting PVE Nodes to PBS

After PBS Rebuild: Fingerprint Rotation

When PBS is rebuilt, it generates a new TLS self-signed certificate. All PVE nodes that reference the PBS storage must be updated with the new fingerprint.

# Get new fingerprint from inside the CT
pct exec 116 -- bash -c "openssl x509 -in /etc/proxmox-backup/proxy.pem -noout -fingerprint -sha256"
# → sha256 Fingerprint=A1:AF:1A:98:...

# Update PVE storage config on any PVE node (pmxcfs replicates to all)
pvesm set noris_s3 --fingerprint A1:AF:1A:98:BF:6A:8B:BA:6D:D7:1C:7B:9C:E8:66:E6:8A:72:ED:DB:7A:89:E9:47:73:07:49:FC:D6:D3:20:83
pvesm set aws_s3 --fingerprint A1:AF:1A:98:BF:6A:8B:BA:6D:D7:1C:7B:9C:E8:66:E6:8A:72:ED:DB:7A:89:E9:47:73:07:49:FC:D6:D3:20:83

Credentials

PBS requires authentication. Set username and password on each PVE PBS storage:

pvesm set noris_s3 --username admin@pbs --password <PASSWORD>
pvesm set aws_s3 --username admin@pbs --password <PASSWORD>

⚠️ If the old PBS used root@pam and the new one uses admin@pbs, BOTH fingerprint AND username must be updated. Leaving username root@pam without a valid root password on the PBS CT results in 401 Unauthorized.

Verify Connection

pvesm list noris_s3 --content backup | head -5
pvesm list aws_s3 --content backup | head -5

Should list backup volumes (format pbs-ct or pbs-vm).

PVE Storage Config Reference

pbs: noris_s3
    datastore noris
    server 10.0.30.106
    content backup
    fingerprint A1:AF:1A:98:BF:6A:8B:BA:6D:D7:1C:7B:9C:E8:66:E6:8A:72:ED:DB:7A:89:E9:47:73:07:49:FC:D6:D3:20:83
    prune-backups keep-all=1
    username admin@pbs

pbs: aws_s3
    datastore s3-backup
    server 10.0.30.106
    content backup
    fingerprint A1:AF:1A:98:BF:6A:8B:BA:6D:D7:1C:7B:9C:E8:66:E6:8A:72:ED:DB:7A:89:E9:47:73:07:49:FC:D6:D3:20:83
    prune-backups keep-all=1
    username admin@pbs

Listing & Searching PBS Backups via pvesh

To find all backups for a specific CT/VM on a PBS storage:

# On the PVE node where the CT/VM runs (or any node — PBS is shared):
pvesh get /nodes/$(hostname)/storage/noris_s3/content --content-type backup 2>&1 \
  | awk -F'│' '{print $3}' \
  | grep 'ct/111'
# → noris_s3:backup/ct/111/2026-07-02T21:00:00Z
# → noris_s3:backup/ct/111/2026-07-03T21:00:03Z
# → noris_s3:backup/ct/111/2026-07-05T21:21:37Z

⚠️ pvesh get outputs a Unicode box-drawing table (not JSON by default). The table uses │ (U+2502) as column separator. Parse with awk -F'│'. Column 3 contains the volid. Alternatively, use --output-format json for programmatic access:

pvesh get /nodes/$(hostname)/storage/noris_s3/content --output-format json \
  | python3 -c "
import json,sys
for b in json.load(sys.stdin):
    v=b.get('volid','')
    if '111' in v: print(v, b.get('ctime'), b.get('size'))"

To list ALL backup groups on a storage (see what's backed up):

pvesh get /nodes/$(hostname)/storage/noris_s3/content 2>&1 \
  | awk -F'│' '{print $3}' \
  | grep -i backup | sort -u

Selective Restore from PBS Backup

When recovering a CT, you often don't need the entire backup — just specific directories (e.g., only /opt/seafile-mysql/db/ from a Seafile CT). Options:

Full pct restore to a new temp CT, then copy needed files out. Simple but slow for large CTs.
proxmox-backup-client restore — restore specific files from a PBS backup snapshot. Requires PBS credentials and runs from inside a CT/VM with the client installed.
pct restore with --storage to the same node, then selectively rsync needed paths from the restored CT to the running one.

Pattern (selective file restore via temp CT):

# Restore CT 111 from PBS to a temp CT (e.g. 9111)
pct restore 9111 noris_s3:backup/ct/111/2026-07-02T21:00:00Z \
  --storage vm_disks --rootfs vm_disks:10G

# Copy only needed files
pct exec 111 -- docker stop seafile seafile-mysql
pct push 9111 /opt/seafile-mysql/db /tmp/mysql-restore  # or rsync
# ... then swap the restored MySQL volume into place

HA Manager Integration

Adding CT to HA

ha-manager add ct:116 --state disabled    # add in disabled state first
ha-manager set ct:116 --state enabled     # enable when ready

⚠️ Use ha-manager set to change state, NOT ha-manager update (which doesn't exist).

Relocating CT Between Nodes

ha-manager crm-command relocate ct:116 proxmox6

Verifying HA Status

ha-manager status | grep 116
# service ct:116 (n5pro, started)  ← running on n5pro

Pitfalls

pct create can't create RBD on some Ceph configs — If pct create fails with "error opening image", pre-create the RBD with rbd create from a Ceph monitor node, then retry.
Unprivileged LXC + pre-formatted RBD = Permission denied — The ext4 root directory must be chowned to 100000:100000 before pct create can extract the template. Without this, tar fails on every file with "Cannot mkdir: Permission denied".
datastore create refuses non-empty paths — When rebuilding PBS over existing data, proxmox-backup-manager datastore create errors with "datastore path not empty". Write /etc/proxmox-backup/datastore.cfg directly and restart PBS services.
PBS ACL role name is Admin — Not Administrator, not admin. The wrong casing gives a cryptic "value not defined in enumeration" error.
ha-manager update doesn't exist — Use ha-manager set <sid> --state enabled|disabled to change HA state.
Fingerprint must be uppercase colon-separated — openssl x509 -fingerprint -sha256 outputs A1:AF:1A:...; pvesm set accepts this format directly. Don't lowercase or strip colons.
PBS CT may land on a different node via HA — Creating the CT on proxmox6 doesn't guarantee it stays there. HA may migrate it to n5pro or another node. The rootfs is on shared Ceph RBD, so it works on any node — but PBS packages installed inside the CT travel with the rootfs, so no reinstall is needed.
Non-Ceph nodes can't create RBD — Nodes without ceph.conf (like n5pro) cannot run rbd commands. Create RBD images from a Ceph monitor node (proxmox4/5/6/7 in this cluster).
root@pam auth fails after rebuild — The new PBS CT has a different root password (or none set for PBS auth). Create an admin@pbs user and update PVE storage configs with the new credentials.
NFS datastore mounts use loop devices — Inside LXC, NFS-mounted .raw files appear as /dev/loopN. This is normal; df -h shows them as loop devices mounted at /mnt/datastore/.

PBS backup owner mismatch causes SILENT backup failures for MONTHS — If a backup group was initially created by root@pam (e.g. via manual vzdump) but the PVE storage config uses username admin@pbs, ALL future backups to that group fail with:

ERROR: VM 106 qmp command 'backup' failed - backup connect failed: command error: backup owner check failed (admin@pbs != root@pam)

The error appears in the PVE task log as job errors status, but the vzdump process exits cleanly — it does NOT retry or alarm. Backups can fail silently for months. The PVE task list shows status=OK for the overall job (because other VMs succeed), masking individual failures.

Diagnosis path:

# 1. Check PVE task history — look for non-OK statuses
pvesh get /nodes/<node>/tasks --limit 50 --output-format json | python3 -c "
import json,sys
for t in json.load(sys.stdin):
    if t.get('type')=='vzdump' and t.get('status')!='OK':
        print(f\"{t['starttime']} {t['status']} id={t.get('id','')}\")"

# 2. Read the task log to find the owner error
pvesh get /nodes/<node>/tasks/<UPID>/log --output-format json

# 3. Check owner files on PBS datastore
pct exec 116 -- bash -c "for d in /mnt/datastore/noris/vm/*/ /mnt/datastore/noris/ct/*/; do echo \"\$(echo \$d | sed 's|.*/noris/||;s|/\$||'): \$(cat \$d/owner)\"; done"

Fix: Overwrite the owner file for each affected group:

pct exec 116 -- bash -c '
  for d in /mnt/datastore/noris/vm/*/ /mnt/datastore/noris/ct/*/; do
    echo "admin@pbs" > "${d}owner"
  done'

Prevention: When changing PBS storage credentials in PVE (pvesm set <storage> --username admin@pbs), immediately fix all existing group owners on the PBS datastore. Always use the same username from the start.

PVE backup jobs silently skip VMs/CTs on other nodes — A cluster-level backup job that includes VMs/CTs spread across multiple PVE nodes will only back up guests on the node running the job. Others are logged as skip external VMs: 104, 108, 111, ... with no error. The job status shows OK even though most guests were skipped. To back up all guests: create per-node backup jobs, or use all mode which runs the job on every node simultaneously.
PBS password for manual proxmox-backup-client use — The PBS password is NOT in /etc/pve/storage.cfg (that file only has username and fingerprint). The plaintext password is stored at /etc/pve/priv/storage/<storage_name>.pw on each PVE node. Read it with cat /etc/pve/priv/storage/noris_s3.pw. When using proxmox-backup-client directly (outside PVE's managed pct restore), export PBS_PASSWORD, PBS_REPOSITORY, and PBS_FINGERPRINT as environment variables — the --fingerprint CLI flag does NOT exist on list/snapshots subcommands. Example:
```
PBS_PW=$(cat /etc/pve/priv/storage/noris_s3.pw)
export PBS_REPOSITORY=admin@pbs@10.0.30.106:noris \
       PBS_PASSWORD=$PBS_PW \
       PBS_FINGERPRINT=A1:AF:1A:98:BF:6A:8B:BA:6D:D7:1C:7B:9C:E8:66:E6:8A:72:ED:DB:7A:89:E9:47:73:07:49:FC:D6:D3:20:83
proxmox-backup-client snapshots | grep 111
proxmox-backup-client restore ct/111/2026-07-02T21:00:00Z root.pxar /tmp/restore --allow-existing-dirs
```
Check verification state before attempting restore — The pvesh get output table includes a verification column with JSON like {"state":"failed",...} or {"state":"ok",...}. Always check this BEFORE attempting restore. A failed verification means chunks are missing/corrupt and the backup cannot be fully restored. Parse with awk -F'│' (column 12 is verification) or use --output-format json.
Corrupted PBS chunks (.bad files) make backups unrecoverable — When PBS garbage collection encounters a corrupt chunk, it renames it to <hash>.0.bad (0 bytes) in /mnt/datastore/<store>/.chunks/<prefix>/. Any backup referencing that chunk cannot be restored. pct restore and proxmox-backup-client restore abort IMMEDIATELY at the first missing chunk with No such file or directory (os error 2) and DELETE everything extracted so far. There is no --skip-bad-chunks, --force, or --ignore-errors option. The only recourse is finding the chunk in another backup (e.g., offsite PBS) or accepting data loss for files that depended on that chunk. Diagnosis:
```
# Check for bad chunks on PBS server (CT 116):
pct exec 116 -- find /mnt/datastore/noris/.chunks -name '*.bad' -exec ls -la {} \;
# Check specific missing chunk:
pct exec 116 -- ls -la /mnt/datastore/noris/.chunks/5cf5/5cf503cc*.bad
```
Incomplete PBS backups: .tmp_didx vs .didx — A PBS snapshot directory containing only .tmp_didx files (e.g., root.pxar.tmp_didx, catalog.pcat1.tmp_didx) instead of .didx files indicates an unfinished/aborted backup. These snapshots CANNOT be restored — pct restore fails with index.json.blob not found or while reading snapshot errors. Always check the snapshot directory on the PBS server:
```
pct exec 116 -- ls -la /mnt/datastore/noris/ct/<vmid>/<snapshot>/
# .didx = complete, restorable
# .tmp_didx = incomplete, NOT restorable
```
Only snapshots with root.pxar.didx (complete dynamic index) are restorable. If the newest backup is .tmp_didx, try older snapshots.

vzdump on wrong node silently exits 0 WITHOUT backing up — If you run vzdump 108 --storage noris_offsite on proxmox1 but CT108 runs on proxmox6, vzdump exits 0 with NO output and NO backup created. There is no error, no warning, no task log entry — it silently does nothing. The backup simply doesn't appear on the PBS server. This happens because vzdump can only back up guests on the local node (unless using --all mode which runs on all nodes).

Diagnosis: If vzdump exits 0 instantly with empty output and no backup appears on PBS:

# Check which node hosts the CT
pvesh get /cluster/resources --type vm --output-format json | python3 -c "
import json,sys
for v in json.load(sys.stdin):
    if v.get('vmid')==108: print(v.get('node'), v.get('status'))"

Fix: Always run vzdump on the node where the CT/VM actually runs:

# Determine the node first
NODE=$(pvesh get /cluster/resources --type vm --output-format json | python3 -c "
import json,sys
for v in json.load(sys.stdin):
    if v.get('vmid')==108: print(v.get('node')); break")
# Run vzdump on that node
ssh root@$NODE "vzdump 108 --storage noris_offsite --mode snapshot --compress zstd"

For batch backups across multiple nodes: Run each backup on its respective node via SSH, not from a single node. Example pattern for backing up CTs spread across proxmox6, proxmox7, and n5pro:

ssh root@10.0.20.60 "vzdump 108 --storage noris_offsite --mode snapshot --compress zstd"
ssh root@10.0.20.70 "vzdump 99999 --storage noris_offsite --mode snapshot --compress zstd"
ssh root@10.0.20.91 "vzdump 104 --storage noris_offsite --mode snapshot --compress zstd"

Expanding offsite backup coverage: prioritize small critical CTs — When adding CTs to an offsite PBS backup job with limited WAN bandwidth, prioritize by size × importance:
- Tier 1 (add first): Small + critical infrastructure (Traefik ~3GB, Gitea ~1.2GB, Authelia, DNS)
- Tier 2 (add second): Medium + important (Paperless ~16GB, Litellm ~3.4GB)
- Tier 3 (evaluate separately): Large + high-value (Seafile ~558GB — consider excluding file blocks and backing up only config+DB)
- Skip: High-churn rewritable data (Frigate video ~40GB, Immich uploads)
To add CTs to an existing offsite job, edit /etc/pve/jobs.cfg:
```
# From any PVE node (pmxcfs replicates)
sed -i '/offsite-ha-daily/,/^$/{s/vmid 106/vmid 106,108,104,99999/}' /etc/pve/jobs.cfg
```
Then trigger initial full backups manually on the correct nodes (see pitfall #17).

19 KiB Raw Blame History Unescape Escape