hermes-skills/devops/proxmox-ve-administration/references/pbs-lxc-setup-2026-07.md

# Proxmox Backup Server (PBS) in LXC — Setup & PVE Integration

## When to Use

Building or rebuilding a Proxmox Backup Server inside an unprivileged LXC container on Ceph RBD storage, then connecting it to PVE nodes as a backup storage target.

## Architecture Overview

- PBS runs as CT 116 (hostname `proxmox-backup-server`, IP 10.0.30.106, VLAN 30)
- Rootfs: 10 GB RBD on Ceph `vm_disks` pool
- Backup datastores: NFS-mounted raw files (`.raw`) on `nfs_ubuntu` storage
- HA-managed via `ha-manager`

## Critical: Unprivileged LXC on Ceph RBD

### Problem

`pct create` with `--rootfs vm_disks:vm-116-disk-0,size=10G` on a Ceph RBD pool fails in two ways:

1. **If RBD doesn't exist yet**: `rbd error: error opening image vm-116-disk-0: (2) No such file or directory` — `pct create` tries to create the RBD internally but fails on some Ceph configurations.

2. **If RBD pre-created and formatted ext4**: `tar: ./etc: Cannot mkdir: Permission denied` — the unprivileged container's UID mapping (0→100000) conflicts with the freshly formatted ext4 root directory (owned by real uid 0).

### Solution: Pre-create, Format, Chown, Then pct create

```bash
# Step 1: Create RBD from a Ceph monitor node (not all PVE nodes have ceph.conf)
rbd create -p vm_disks vm-116-disk-0 --size 10G

# Step 2: Map, format ext4, chown root to 100000:100000, unmap
rbd map -p vm_disks vm-116-disk-0    # → /dev/rbdN
mkdir -p /tmp/fix-ct
mount /dev/rbdN /tmp/fix-ct
chown 100000:100000 /tmp/fix-ct
chmod 755 /tmp/fix-ct
umount /tmp/fix-ct
rbd unmap /dev/rbdN

# Step 3: Clean up any stale pct state
rm -f /etc/pve/lxc/NNN.conf

# Step 4: pct create (RBD already exists, formatted, and chowned)
pct create NNN hdd_templates:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --rootfs vm_disks:vm-NNN-disk-0,size=10G \
  --hostname proxmox-backup-server \
  --memory 4096 --cores 4 \
  --features keyctl=1,nesting=1 \
  --net0 name=eth0,bridge=vmbr0,gw=10.0.30.1,ip=10.0.30.106/24,tag=30,type=veth \
  --unprivileged 1 --onboot 1 --swap 512
```

### Why This Works

`lxc-usernsexec -m u:0:100000:65536` maps container root (uid 0) to host uid 100000. When tar extracts the template, it creates files as uid 0 (inside the namespace) = uid 100000 (on the host filesystem). If the ext4 root directory is owned by uid 0 on the host, the mapped uid 100000 lacks permission to create entries. Chowning the root directory to 100000:100000 before `pct create` grants the mapped uid write access.

### Privileged Container Alternative

If the chown workaround fails or is undesirable, dropping `--unprivileged 1` avoids the UID mapping entirely. However, unprivileged is recommended for security — PBS does not require privileged mode.

## PBS Installation Inside LXC

### Step 1: Add PBS Repository

```bash
# Inside the CT (via pct exec)
echo 'deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription' > /etc/apt/sources.list.d/pbs.list
wget -q -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg http://download.proxmox.com/debian/proxmox-release-bookworm.gpg
apt-get update -qq
```

### Step 2: Install PBS Packages

```bash
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq proxmox-backup-server proxmox-backup-client
```

This pulls in ZFS, LVM, thin-provisioning-tools as dependencies (normal for PBS).

### Step 3: Register Existing Datastores

If the datastore paths already contain backup data (rebuild scenario), `proxmox-backup-manager datastore create` fails with `"datastore path not empty"`. Instead, write the config directly:

```bash
cat > /etc/proxmox-backup/datastore.cfg << 'EOF'
datastore: s3-backup2
    path /mnt/datastore/s3-backup2

datastore: s3-backup
    path /mnt/datastore/s3-backup

datastore: noris
    path /mnt/datastore/noris
EOF

systemctl restart proxmox-backup-proxy proxmox-backup
```

Then verify:
```bash
proxmox-backup-manager datastore list
```

### Step 4: Create Admin User

```bash
proxmox-backup-manager user create admin@pbs --password <PASSWORD>
proxmox-backup-manager acl update / Admin --auth-id admin@pbs
```

⚠️ The role name is `Admin` (capital A), not `Administrator` or `admin`. Using the wrong name gives a confusing "value not defined in enumeration" error.

Verify authentication:
```bash
curl -sk -d username=admin@pbs -d password=<PASSWORD> \
  https://localhost:8007/api2/json/access/ticket
```

Should return JSON with a ticket and CSRF token.

### Step 5: Add NFS Datastore Mounts

Add mount points to the CT config (from a PVE node):

```bash
pct set 116 \
  -mp0 nfs_ubuntu:116/vm-116-disk-2.raw,mp=/mnt/datastore/s3-backup2,size=500G \
  -mp1 nfs_ubuntu:116/vm-116-disk-3.raw,mp=/mnt/datastore/s3-backup,size=192G \
  -mp2 nfs_ubuntu:116/vm-116-disk-0.raw,mp=/mnt/datastore/noris,size=500G
```

## PVE Integration: Connecting PVE Nodes to PBS

### After PBS Rebuild: Fingerprint Rotation

When PBS is rebuilt, it generates a new TLS self-signed certificate. All PVE nodes that reference the PBS storage must be updated with the new fingerprint.

```bash
# Get new fingerprint from inside the CT
pct exec 116 -- bash -c "openssl x509 -in /etc/proxmox-backup/proxy.pem -noout -fingerprint -sha256"
# → sha256 Fingerprint=A1:AF:1A:98:...

# Update PVE storage config on any PVE node (pmxcfs replicates to all)
pvesm set noris_s3 --fingerprint A1:AF:1A:98:BF:6A:8B:BA:6D:D7:1C:7B:9C:E8:66:E6:8A:72:ED:DB:7A:89:E9:47:73:07:49:FC:D6:D3:20:83
pvesm set aws_s3 --fingerprint A1:AF:1A:98:BF:6A:8B:BA:6D:D7:1C:7B:9C:E8:66:E6:8A:72:ED:DB:7A:89:E9:47:73:07:49:FC:D6:D3:20:83
```

### Credentials

PBS requires authentication. Set username and password on each PVE PBS storage:

```bash
pvesm set noris_s3 --username admin@pbs --password <PASSWORD>
pvesm set aws_s3 --username admin@pbs --password <PASSWORD>
```

⚠️ If the old PBS used `root@pam` and the new one uses `admin@pbs`, BOTH fingerprint AND username must be updated. Leaving `username root@pam` without a valid root password on the PBS CT results in `401 Unauthorized`.

### Verify Connection

```bash
pvesm list noris_s3 --content backup | head -5
pvesm list aws_s3 --content backup | head -5
```

Should list backup volumes (format `pbs-ct` or `pbs-vm`).

### PVE Storage Config Reference

```
pbs: noris_s3
    datastore noris
    server 10.0.30.106
    content backup
    fingerprint A1:AF:1A:98:BF:6A:8B:BA:6D:D7:1C:7B:9C:E8:66:E6:8A:72:ED:DB:7A:89:E9:47:73:07:49:FC:D6:D3:20:83
    prune-backups keep-all=1
    username admin@pbs

pbs: aws_s3
    datastore s3-backup
    server 10.0.30.106
    content backup
    fingerprint A1:AF:1A:98:BF:6A:8B:BA:6D:D7:1C:7B:9C:E8:66:E6:8A:72:ED:DB:7A:89:E9:47:73:07:49:FC:D6:D3:20:83
    prune-backups keep-all=1
    username admin@pbs
```

## Listing & Searching PBS Backups via pvesh

To find all backups for a specific CT/VM on a PBS storage:

```bash
# On the PVE node where the CT/VM runs (or any node — PBS is shared):
pvesh get /nodes/$(hostname)/storage/noris_s3/content --content-type backup 2>&1 \
  | awk -F'│' '{print $3}' \
  | grep 'ct/111'
# → noris_s3:backup/ct/111/2026-07-02T21:00:00Z
# → noris_s3:backup/ct/111/2026-07-03T21:00:03Z
# → noris_s3:backup/ct/111/2026-07-05T21:21:37Z
```

⚠️ `pvesh get` outputs a **Unicode box-drawing table** (not JSON by default).
The table uses `│` (U+2502) as column separator. Parse with `awk -F'│'`.
Column 3 contains the volid. Alternatively, use `--output-format json` for
programmatic access:

```bash
pvesh get /nodes/$(hostname)/storage/noris_s3/content --output-format json \
  | python3 -c "
import json,sys
for b in json.load(sys.stdin):
    v=b.get('volid','')
    if '111' in v: print(v, b.get('ctime'), b.get('size'))"
```

To list ALL backup groups on a storage (see what's backed up):

```bash
pvesh get /nodes/$(hostname)/storage/noris_s3/content 2>&1 \
  | awk -F'│' '{print $3}' \
  | grep -i backup | sort -u
```

### Selective Restore from PBS Backup

When recovering a CT, you often don't need the entire backup — just specific
directories (e.g., only `/opt/seafile-mysql/db/` from a Seafile CT). Options:

1. **Full `pct restore`** to a new temp CT, then copy needed files out.
   Simple but slow for large CTs.
2. **`proxmox-backup-client restore`** — restore specific files from a PBS
   backup snapshot. Requires PBS credentials and runs from inside a CT/VM
   with the client installed.
3. **`pct restore` with `--storage`** to the same node, then selectively
   rsync needed paths from the restored CT to the running one.

Pattern (selective file restore via temp CT):
```bash
# Restore CT 111 from PBS to a temp CT (e.g. 9111)
pct restore 9111 noris_s3:backup/ct/111/2026-07-02T21:00:00Z \
  --storage vm_disks --rootfs vm_disks:10G

# Copy only needed files
pct exec 111 -- docker stop seafile seafile-mysql
pct push 9111 /opt/seafile-mysql/db /tmp/mysql-restore  # or rsync
# ... then swap the restored MySQL volume into place
```

## HA Manager Integration

### Adding CT to HA

```bash
ha-manager add ct:116 --state disabled    # add in disabled state first
ha-manager set ct:116 --state enabled     # enable when ready
```

⚠️ Use `ha-manager set` to change state, NOT `ha-manager update` (which doesn't exist).

### Relocating CT Between Nodes

```bash
ha-manager crm-command relocate ct:116 proxmox6
```

### Verifying HA Status

```bash
ha-manager status | grep 116
# service ct:116 (n5pro, started)  ← running on n5pro
```

## Pitfalls

1. **`pct create` can't create RBD on some Ceph configs** — If `pct create` fails with "error opening image", pre-create the RBD with `rbd create` from a Ceph monitor node, then retry.

2. **Unprivileged LXC + pre-formatted RBD = Permission denied** — The ext4 root directory must be chowned to 100000:100000 before `pct create` can extract the template. Without this, tar fails on every file with "Cannot mkdir: Permission denied".

3. **`datastore create` refuses non-empty paths** — When rebuilding PBS over existing data, `proxmox-backup-manager datastore create` errors with "datastore path not empty". Write `/etc/proxmox-backup/datastore.cfg` directly and restart PBS services.

4. **PBS ACL role name is `Admin`** — Not `Administrator`, not `admin`. The wrong casing gives a cryptic "value not defined in enumeration" error.

5. **`ha-manager update` doesn't exist** — Use `ha-manager set <sid> --state enabled|disabled` to change HA state.

6. **Fingerprint must be uppercase colon-separated** — `openssl x509 -fingerprint -sha256` outputs `A1:AF:1A:...`; `pvesm set` accepts this format directly. Don't lowercase or strip colons.

7. **PBS CT may land on a different node via HA** — Creating the CT on proxmox6 doesn't guarantee it stays there. HA may migrate it to n5pro or another node. The rootfs is on shared Ceph RBD, so it works on any node — but PBS packages installed inside the CT travel with the rootfs, so no reinstall is needed.

8. **Non-Ceph nodes can't create RBD** — Nodes without `ceph.conf` (like n5pro) cannot run `rbd` commands. Create RBD images from a Ceph monitor node (proxmox4/5/6/7 in this cluster).

9. **`root@pam` auth fails after rebuild** — The new PBS CT has a different root password (or none set for PBS auth). Create an `admin@pbs` user and update PVE storage configs with the new credentials.

10. **NFS datastore mounts use loop devices** — Inside LXC, NFS-mounted `.raw` files appear as `/dev/loopN`. This is normal; `df -h` shows them as loop devices mounted at `/mnt/datastore/`.

11. **PBS backup owner mismatch causes SILENT backup failures for MONTHS** — If a backup group was initially created by `root@pam` (e.g. via manual `vzdump`) but the PVE storage config uses `username admin@pbs`, ALL future backups to that group fail with:
    ```
    ERROR: VM 106 qmp command 'backup' failed - backup connect failed: command error: backup owner check failed (admin@pbs != root@pam)
    ```
    The error appears in the PVE task log as `job errors` status, but the vzdump process exits cleanly — it does NOT retry or alarm. Backups can fail silently for months. The PVE task list shows `status=OK` for the overall job (because other VMs succeed), masking individual failures.
    
    **Diagnosis path:**
    ```bash
    # 1. Check PVE task history — look for non-OK statuses
    pvesh get /nodes/<node>/tasks --limit 50 --output-format json | python3 -c "
    import json,sys
    for t in json.load(sys.stdin):
        if t.get('type')=='vzdump' and t.get('status')!='OK':
            print(f\"{t['starttime']} {t['status']} id={t.get('id','')}\")"
    
    # 2. Read the task log to find the owner error
    pvesh get /nodes/<node>/tasks/<UPID>/log --output-format json
    
    # 3. Check owner files on PBS datastore
    pct exec 116 -- bash -c "for d in /mnt/datastore/noris/vm/*/ /mnt/datastore/noris/ct/*/; do echo \"\$(echo \$d | sed 's|.*/noris/||;s|/\$||'): \$(cat \$d/owner)\"; done"
    ```
    
    **Fix:** Overwrite the owner file for each affected group:
    ```bash
    pct exec 116 -- bash -c '
      for d in /mnt/datastore/noris/vm/*/ /mnt/datastore/noris/ct/*/; do
        echo "admin@pbs" > "${d}owner"
      done'
    ```
    
    **Prevention:** When changing PBS storage credentials in PVE (`pvesm set <storage> --username admin@pbs`), immediately fix all existing group owners on the PBS datastore. Always use the same username from the start.

12. **PVE backup jobs silently skip VMs/CTs on other nodes** — A cluster-level backup job that includes VMs/CTs spread across multiple PVE nodes will only back up guests on the node running the job. Others are logged as `skip external VMs: 104, 108, 111, ...` with no error. The job status shows `OK` even though most guests were skipped. To back up all guests: create per-node backup jobs, or use `all` mode which runs the job on every node simultaneously.

13. **PBS password for manual `proxmox-backup-client` use** — The PBS password is NOT in `/etc/pve/storage.cfg` (that file only has `username` and `fingerprint`). The plaintext password is stored at `/etc/pve/priv/storage/<storage_name>.pw` on each PVE node. Read it with `cat /etc/pve/priv/storage/noris_s3.pw`. When using `proxmox-backup-client` directly (outside PVE's managed `pct restore`), export `PBS_PASSWORD`, `PBS_REPOSITORY`, and `PBS_FINGERPRINT` as environment variables — the `--fingerprint` CLI flag does NOT exist on `list`/`snapshots` subcommands. Example:
    ```bash
    PBS_PW=$(cat /etc/pve/priv/storage/noris_s3.pw)
    export PBS_REPOSITORY=admin@pbs@10.0.30.106:noris \
           PBS_PASSWORD=$PBS_PW \
           PBS_FINGERPRINT=A1:AF:1A:98:BF:6A:8B:BA:6D:D7:1C:7B:9C:E8:66:E6:8A:72:ED:DB:7A:89:E9:47:73:07:49:FC:D6:D3:20:83
    proxmox-backup-client snapshots | grep 111
    proxmox-backup-client restore ct/111/2026-07-02T21:00:00Z root.pxar /tmp/restore --allow-existing-dirs
    ```

14. **Check verification state before attempting restore** — The `pvesh get` output table includes a `verification` column with JSON like `{"state":"failed",...}` or `{"state":"ok",...}`. Always check this BEFORE attempting restore. A `failed` verification means chunks are missing/corrupt and the backup cannot be fully restored. Parse with `awk -F'│'` (column 12 is verification) or use `--output-format json`.

15. **Corrupted PBS chunks (`.bad` files) make backups unrecoverable** — When PBS garbage collection encounters a corrupt chunk, it renames it to `<hash>.0.bad` (0 bytes) in `/mnt/datastore/<store>/.chunks/<prefix>/`. Any backup referencing that chunk cannot be restored. `pct restore` and `proxmox-backup-client restore` abort IMMEDIATELY at the first missing chunk with `No such file or directory (os error 2)` and DELETE everything extracted so far. There is no `--skip-bad-chunks`, `--force`, or `--ignore-errors` option. The only recourse is finding the chunk in another backup (e.g., offsite PBS) or accepting data loss for files that depended on that chunk. Diagnosis:
    ```bash
    # Check for bad chunks on PBS server (CT 116):
    pct exec 116 -- find /mnt/datastore/noris/.chunks -name '*.bad' -exec ls -la {} \;
    # Check specific missing chunk:
    pct exec 116 -- ls -la /mnt/datastore/noris/.chunks/5cf5/5cf503cc*.bad
    ```

16. **Incomplete PBS backups: `.tmp_didx` vs `.didx`** — A PBS snapshot directory containing only `.tmp_didx` files (e.g., `root.pxar.tmp_didx`, `catalog.pcat1.tmp_didx`) instead of `.didx` files indicates an unfinished/aborted backup. These snapshots CANNOT be restored — `pct restore` fails with `index.json.blob not found` or `while reading snapshot` errors. Always check the snapshot directory on the PBS server:
    ```bash
    pct exec 116 -- ls -la /mnt/datastore/noris/ct/<vmid>/<snapshot>/
    # .didx = complete, restorable
    # .tmp_didx = incomplete, NOT restorable
    ```
    Only snapshots with `root.pxar.didx` (complete dynamic index) are restorable. If the newest backup is `.tmp_didx`, try older snapshots.

17. **`vzdump` on wrong node silently exits 0 WITHOUT backing up** — If you run `vzdump 108 --storage noris_offsite` on proxmox1 but CT108 runs on proxmox6, vzdump exits 0 with NO output and NO backup created. There is no error, no warning, no task log entry — it silently does nothing. The backup simply doesn't appear on the PBS server. This happens because vzdump can only back up guests on the local node (unless using `--all` mode which runs on all nodes).

    **Diagnosis:** If vzdump exits 0 instantly with empty output and no backup appears on PBS:
    ```bash
    # Check which node hosts the CT
    pvesh get /cluster/resources --type vm --output-format json | python3 -c "
    import json,sys
    for v in json.load(sys.stdin):
        if v.get('vmid')==108: print(v.get('node'), v.get('status'))"
    ```

    **Fix:** Always run vzdump on the node where the CT/VM actually runs:
    ```bash
    # Determine the node first
    NODE=$(pvesh get /cluster/resources --type vm --output-format json | python3 -c "
    import json,sys
    for v in json.load(sys.stdin):
        if v.get('vmid')==108: print(v.get('node')); break")
    # Run vzdump on that node
    ssh root@$NODE "vzdump 108 --storage noris_offsite --mode snapshot --compress zstd"
    ```

    **For batch backups across multiple nodes:** Run each backup on its respective node via SSH, not from a single node. Example pattern for backing up CTs spread across proxmox6, proxmox7, and n5pro:
    ```bash
    ssh root@10.0.20.60 "vzdump 108 --storage noris_offsite --mode snapshot --compress zstd"
    ssh root@10.0.20.70 "vzdump 99999 --storage noris_offsite --mode snapshot --compress zstd"
    ssh root@10.0.20.91 "vzdump 104 --storage noris_offsite --mode snapshot --compress zstd"
    ```

18. **Expanding offsite backup coverage: prioritize small critical CTs** — When adding CTs to an offsite PBS backup job with limited WAN bandwidth, prioritize by size × importance:
    - Tier 1 (add first): Small + critical infrastructure (Traefik ~3GB, Gitea ~1.2GB, Authelia, DNS)
    - Tier 2 (add second): Medium + important (Paperless ~16GB, Litellm ~3.4GB)
    - Tier 3 (evaluate separately): Large + high-value (Seafile ~558GB — consider excluding file blocks and backing up only config+DB)
    - Skip: High-churn rewritable data (Frigate video ~40GB, Immich uploads)

    To add CTs to an existing offsite job, edit `/etc/pve/jobs.cfg`:
    ```bash
    # From any PVE node (pmxcfs replicates)
    sed -i '/offsite-ha-daily/,/^$/{s/vmid 106/vmid 106,108,104,99999/}' /etc/pve/jobs.cfg
    ```

    Then trigger initial full backups manually on the correct nodes (see pitfall #17).