# PBS Offsite Backup Jobs: Adding Guests & Running on Correct Nodes

## When to Use

When expanding offsite PBS backup coverage to include more CTs/VMs beyond
the initial HA-only job, considering bandwidth constraints of the remote
site.

## Key Lessons

### 1. vzdump Must Run on the Node Hosting the CT

**Critical pitfall:** `vzdump <vmid> --storage noris_offsite` run on
proxmox1 (10.0.20.10) will silently exit 0 for any CT hosted on a different
node — no backup is created, no error is logged. The CT config file
(`/etc/pve/lxc/<vmid>.conf`) only exists on the node where the CT runs.

**Finding which node hosts a CT:**
```bash
ssh root@10.0.20.10 "pvesh get /cluster/resources --type vm --output-format json" | \
  python3 -c "
import sys,json
data=json.load(sys.stdin)
for v in data:
    if v.get('vmid') == TARGET_VMid:
        print(f\"node={v['node']} status={v['status']}\")
"
```

**Running backup on the correct node:**
```bash
# CT108 on proxmox6 (10.0.20.60)
ssh root@10.0.20.60 "vzdump 108 --storage noris_offsite --mode snapshot --compress zstd"

# CT99999 on proxmox7 (10.0.20.70)
ssh root@10.0.20.70 "vzdump 99999 --storage noris_offsite --mode snapshot --compress zstd"

# CT104 on n5pro (10.0.20.91)
ssh root@10.0.20.91 "vzdump 104 --storage noris_offsite --mode snapshot --compress zstd"
```

### 2. Updating the Offsite Backup Job

The offsite job config lives in `/etc/pve/jobs.cfg` on any PVE node (shared
filesystem). Edit with sed:

```bash
ssh root@10.0.20.10 "
sed -i '/offsite-ha-daily/,/^$/{s/vmid 106/vmid 106,108,104,99999/}' /etc/pve/jobs.cfg
"
```

Job config after update:
```
vzdump: offsite-ha-daily
	compress zstd
	enabled 1
	mode snapshot
	schedule 02:00
	storage noris_offsite
	vmid 106,108,104,99999
```

### 3. Bandwidth-Conscious Guest Selection

The remote PBS at noris (213.95.54.60) has limited upload bandwidth.
PBS dedup means only changed chunks are transferred after the initial
full backup, but the first backup of each guest is a full upload.

**Selection criteria for offsite:**

| Criterion | Include | Exclude |
|-----------|---------|---------|
| Size < 20 GB | ✅ Small enough for initial full | |
| Size > 100 GB | | ❌ Too large for WAN |
| Change rate low (config, code) | ✅ Daily delta is tiny | |
| Change rate high (video, media) | | ❌ Daily delta too large |
| Infrastructure-critical | ✅ Traefik, Gitea, HA | |
| Replaceable data | | ❌ Frigate recordings |

**This homelab's offsite selection (2026-07-08):**
- VM106 HA (70 GB) — already in job, critical
- CT108 Gitea (1.2 GB) — added, critical code
- CT99999 Traefik (3 GB) — added, critical infra
- CT104 Paperless (16 GB) — added, documents

**Excluded from offsite:**
- CT111 Seafile (558 GB) — too large, needs separate strategy
- CT120 Frigate (40 GB) — replaceable video data
- CT115 Immich (20 GB) — photos, needs separate strategy
- CT121/124/127 — low priority, small

### 4. Verifying Offsite Backups

After running manual backups, verify on the remote PBS:

```bash
ssh root@10.0.20.10 "pvesh get /nodes/proxmox1/storage/noris_offsite/content --output-format json" | \
  python3 -c "
import sys,json,datetime
data=json.load(sys.stdin)
for b in sorted(data, key=lambda x: x.get('ctime',0), reverse=True)[:10]:
    volid=b.get('volid','')
    sz=b.get('size',0)//1024//1024
    dt=datetime.datetime.fromtimestamp(b.get('ctime',0)).strftime('%Y-%m-%d %H:%M')
    print(f'{volid:55s} {sz:>8} MB  {dt}')
"
```

**Note:** PBS content listing may lag by a few minutes after upload.
If backups don't appear immediately, wait and re-query.

### 5. Lock Timeout on Concurrent vzdump

If two vzdump jobs run simultaneously on different nodes targeting the
same PBS storage, one may fail with:
```
ERROR: can't acquire lock '/var/run/vzdump.lock' - got timeout
```

This is a global lock across the PVE cluster. Retry the failed backup
after the first one completes. For manual triggers, run sequentially.

## Session Reference

- **2026-07-08**: Added CT108 (Gitea), CT99999 (Traefik), CT104 (Paperless)
  to offsite-ha-daily job. Initial full backups run manually on correct
  nodes. CT104 failed with lock timeout (concurrent with scheduled job),
  retried successfully.