# PBS Sync Pipeline: Local → Remote with Verify + Prune

## When to Use

When you need a multi-stage backup pipeline: backup locally for fast restores,
async-push to offsite PBS, verify on both sides, then prune local aggressively
while retaining remote long-term. This is the "tiered backup" pattern.

## ⚠️ Important: Direct vzdump May Be Better Than Sync

**Confirmed 2026-07-05**: The sync pipeline sounds elegant but the initial
push sync over WAN is **impractically slow**. In practice:

- A 650 GB initial sync at measured 2.7 KB/s actual throughput (not the
  theoretical 5-7 MB/s) would take **weeks**, not hours.
- PBS push sync has significant protocol overhead — many small chunk
  round-trips over WAN make it far slower than raw vzdump.
- Meanwhile, **direct vzdump to the remote PBS** already populated the
  remote datastore with the same deduplicated chunks.
- When the sync job finally ran, it reported `"found no new data to push"`
  because the remote already had identical chunks from direct vzdump.

### Recommended Strategy

| Pattern | When to Use | Why |
|---------|------------|-----|
| **Direct vzdump → remote PBS** | Always, for ongoing offsite backups | Dedup handles incrementals naturally, no sync overhead |
| **Local PBS (separate job)** | For fast local restores | Independent schedule + retention |
| **Push sync job** | Only if local PBS has backups the remote DOESN'T have | E.g. migrating to a new remote PBS |

Don't set up the sync pipeline expecting it to be the primary offsite mechanism.
Set up **two independent vzdump jobs** (local + remote) with different retention.
The sync job is a migration tool, not a daily driver.

### ⚠️ Sync "found no new data to push" — Check for Owner Mismatch

If `sync-job run` immediately returns `"found no new data to push"` but you
know the remote is missing backups that exist locally, **the local backups
may have been failing silently for months due to an owner mismatch** (see
`references/pbs-lxc-setup-2026-07.md` pitfall #11).

Debugging path (discovered 2026-07-05):

1. Check PVE task history for the local backup job — look for non-OK statuses:
   ```bash
   pvesh get /nodes/<node>/tasks --limit 50 --output-format json | python3 -c "
   import json,sys
   for t in json.load(sys.stdin):
       if t.get('type')=='vzdump':
           print(f\"{t['starttime']} {t['status']} id={t.get('id','')}\")"
   ```
   Status `job errors` (not `OK`) means individual VM backups failed.

2. Read the task log to find the owner error:
   ```bash
   pvesh get /nodes/<node>/tasks/<UPID>/log --output-format json
   ```
   Look for: `backup owner check failed (admin@pbs != root@pam)`

3. Check owner files on the PBS datastore:
   ```bash
   pct exec 116 -- bash -c "
     for d in /mnt/datastore/noris/vm/*/ /mnt/datastore/noris/ct/*/; do
       echo \"$(echo $d | sed 's|.*/noris/||;s|/$||'): $(cat ${d}owner)\"
     done"
   ```

4. Fix: overwrite all owner files to match the PVE storage config username:
   ```bash
   pct exec 116 -- bash -c '
     for d in /mnt/datastore/noris/vm/*/ /mnt/datastore/noris/ct/*/; do
       echo "admin@pbs" > "${d}owner"
     done'
   ```

5. After fixing, trigger a manual backup to confirm it works:
   ```bash
   # On the PVE node where the VM runs
   vzdump 106 --storage noris_s3 --mode snapshot --quiet 1
   ```
   Watch the PBS task log for progress (not the PVE log — PVE `--quiet`
   suppresses progress output):
   ```bash
   pct exec 116 -- find /var/log/proxmox-backup/tasks -name "*<timestamp>*" -exec tail -5 {} \;
   ```

**Root cause explained**: The local backup job was failing every night
with `backup owner check failed`, but the overall job status showed
`job errors` (not a hard crash), and other VMs in the same job succeeded.
Meanwhile, the remote PBS already had backups from the direct `offsite-ha-daily`
vzdump job at 02:00. So when the sync job ran, the remote already had the
same deduplicated chunks — hence "no new data to push". The sync wasn't
broken; the **local backups were never being written**.

### Measuring Actual WAN Throughput

Don't trust the PBS task log — it goes silent during transfer. Measure
actual network throughput with `/proc/net/dev` deltas:

```bash
# On local PBS (CT 116):
T1=$(ssh root@pve-node "pct exec 116 -- cat /proc/net/dev | grep eth0" | awk '{print $2}')
sleep 5
T2=$(ssh root@pve-node "pct exec 116 -- cat /proc/net/dev | grep eth0" | awk '{print $2}')
echo "RX rate: $(( (T2 - T1) / 5 )) bytes/sec"

# On remote PBS:
T1=$(ssh ubuntu@<REMOTE_IP> 'cat /proc/net/dev | grep ens3' | awk '{print $2}')
sleep 5
T2=$(ssh ubuntu@<REMOTE_IP> 'cat /proc/net/dev | grep ens3' | awk '{print $2}')
echo "RX rate: $(( (T2 - T1) / 5 )) bytes/sec"
```

If RX rate is < 5 KB/s while a sync task shows "running", the sync is
effectively stalled — the PBS sync protocol is doing negligible work.
Kill it and use direct vzdump instead.

## Architecture

```
  23:00  vzdump → lokaler PBS (noris datastore)
     ↓
  02:30  verify-noris      (local integrity check)
     ↓
  03:00  sync-to-offsite   (push last backup → remote PBS)
  03:00  prune-offsite      (remote: keep-last=3, weekly=4, monthly=6)
     ↓
  04:00  verify-offsite    (remote integrity check)
     ↓
  05:00  prune-noris        (local: keep-last=1 only — frees space)
```

### Key Design Decisions

1. **Push direction**: Local PBS pushes to remote (not pull). Remote PBS
   doesn't initiate connections to the local network.
2. **`--transfer-last 1`**: Only sync the most recent snapshot per group.
   Reduces bandwidth on daily sync.
3. **Local keep-last=1**: After sync+verify, only the latest backup stays
   local for quick restore. Older backups are pruned.
4. **Remote long retention**: keep-last=3 + keep-weekly=4 + keep-monthly=6
   provides ~6 months of offsite history.
5. **PBS has no native "delete-local-after-remote-verify"**: The pipeline
   approximates this with staggered schedules. Between 03:00 (sync) and
   05:00 (local prune), the backup exists on both sides.

## Prerequisites

- Local PBS (e.g. CT 116) with datastore `noris`
- Remote PBS (e.g. OpenStack VM) with datastore `offsite`
- Remote PBS registered on local PBS as a "remote"

## Step-by-Step Setup

### 1. Register Remote PBS on Local PBS

```bash
# On local PBS (inside CT 116 via pct exec)
proxmox-backup-manager remote create noris-offsite \
  --host <REMOTE_IP> \
  --port 8007 \
  --auth-id admin@pbs \
  --password <PASSWORD>

# CRITICAL: Add fingerprint — push will fail with SSL cert error without it
# Get fingerprint from remote PBS:
#   ssh ubuntu@<REMOTE_IP> 'sudo proxmox-backup-manager cert info | grep Fingerprint'
proxmox-backup-manager remote update noris-offsite \
  --fingerprint <SHA256_FINGERPRINT>

# Verify
proxmox-backup-manager remote list
```

### 2. Create Push Sync Job

```bash
proxmox-backup-manager sync-job create sync-to-offsite \
  --remote noris-offsite \
  --remote-store offsite \
  --store noris \
  --sync-direction push \
  --schedule 03:00 \
  --transfer-last 1
```

### 3. Create Local Verify Job

```bash
proxmox-backup-manager verify-job create verify-noris \
  --store noris \
  --schedule 02:30
```

### 4. Create Local Prune Job (Aggressive)

```bash
proxmox-backup-manager prune-job create prune-noris \
  --store noris \
  --schedule 05:00 \
  --keep-last 1 \
  --comment "Minimal local retention after offsite sync"
```

### 5. Create Remote Verify + Prune Jobs

On the remote PBS:

```bash
# Verify job
proxmox-backup-manager verify-job create verify-offsite \
  --store offsite \
  --schedule 04:00

# Prune job (long retention)
proxmox-backup-manager prune-job create prune-offsite \
  --store offsite \
  --schedule 03:00 \
  --keep-last 3 \
  --keep-weekly 4 \
  --keep-monthly 6
```

### 6. Run Initial Sync Manually

The first sync transfers all data (full, not incremental). Run manually
to avoid timing out the scheduled job:

```bash
# On local PBS
proxmox-backup-manager push noris noris-offsite offsite --transfer-last 1
```

This may take a long time (hours for hundreds of GB). Subsequent syncs
are incremental (only new chunks transferred).

### 7. Verify All Jobs

```bash
# Local PBS
proxmox-backup-manager verify-job list
proxmox-backup-manager prune-job list
proxmox-backup-manager sync-job list    # ⚠️ see pitfall #1

# Remote PBS
proxmox-backup-manager verify-job list
proxmox-backup-manager prune-job list
```

## Manual Verification

Run a one-time verify on any PBS datastore:

```bash
proxmox-backup-manager verify <datastore>
```

Example output (successful):
```
verify datastore offsite
found 2 groups
verify group offsite:ct/116 (1 snapshots)
  verified 382.77/967.64 MiB in 1.89 seconds (0 errors)
verify group offsite:vm/106 (2 snapshots)
  verified 17222.64/52576.00 MiB in 103.62 seconds (0 errors)
TASK OK
```

## Monitoring Running Tasks

```bash
# List running/completed tasks
proxmox-backup-manager task list

# View task log
proxmox-backup-manager task log <UPID>

# Stop a running task
proxmox-backup-manager task stop <UPID>
```

## Sync-Job Config File Location

Sync job configs are stored at `/etc/proxmox-backup/sync.cfg`:

```
sync: sync-to-offsite
    comment Push last backup to offsite
    remote noris-offsite
    remote-store offsite
    schedule 03:00
    store noris
    sync-direction push
    transfer-last 1
```

## Pitfalls

1. **`sync-job list` may not display push jobs** — In PBS 3.4.8, `sync-job list`
   returns empty even when a push sync job exists. The job IS created and
   stored in `/etc/proxmox-backup/sync.cfg`. Verify by reading the config file
   directly. The scheduled job still runs correctly.

2. **Remote fingerprint required before push** — `proxmox-backup-manager push`
   fails with `SSL routines:tls_post_process_server_certificate:certificate
   verify failed` if the remote's fingerprint isn't registered. Always run
   `remote update --fingerprint` after `remote create`.

3. **`prune-job create` rejects keep-*=0** — PBS enforces minimum value of 1
   for all keep-* parameters. To effectively disable a retention dimension,
   simply omit it (don't set it to 0). E.g. `--keep-last 1` alone keeps only
   1 backup with no weekly/monthly/yearly retention.

4. **Initial sync is VERY slow — plan for hours, not minutes** — First push
   transfers all deduplicated chunks. For ~650 GB at ~5-7 MB/s WAN speed, expect
   **25-30+ hours** for the initial sync. The 300s timeout is nowhere near
   enough. Run manually with a long timeout or let the scheduled job handle it
   overnight (or over multiple nights). Subsequent daily syncs are incremental
   and transfer only new/changed chunks (typically MB, not GB).

   **User reality check**: "in einer stunde dürfte das nicht hochgeladen sein" —
   correct. At 5 MB/s, 650 GB takes ~36 hours. Always calculate:
   `hours = total_GB * 1024 / (speed_MBps * 3600)`.

5. **Push sync task log shows NO progress** — The task log only records the
   initial lines ("Found 9 groups to sync", "skipped: N snapshot(s)") and then
   goes silent for the entire transfer. No progress bars, no chunk counts.
   To monitor actual progress, check the **remote side**:
   ```bash
   # On remote PBS — watch disk usage grow
   ssh ubuntu@<REMOTE_IP> 'df -h /mnt/datastore/offsite; sudo du -sh /mnt/datastore/offsite/.chunks/'
   ```
   Compare before/after to estimate transfer rate. The `.chunks/` directory
   grows as chunks arrive.

6. **`--transfer-last 1` with many groups still moves lots of data** —
   `--transfer-last 1` syncs 1 snapshot PER GROUP. With 9 groups (multiple
   VMs/CTs), that's 9 snapshots total. Large VMs (e.g. 50+ GB HA VM) dominate
   the transfer. Consider filtering groups with `--group-filter` to sync only
   critical VMs first, then expand.

7. **GC (garbage collection) runs by default** — PBS enables daily GC at
   midnight by default. No explicit configuration needed. GC reclaims
   space from pruned chunks.

6. **Schedule staggering matters** — Ensure verify runs BEFORE sync (so you
   don't push corrupt data) and local prune runs AFTER sync+remote-verify
   (so you don't delete local before remote is confirmed good).

7. **`--transfer-last N` limits per-group** — `--transfer-last 1` syncs only
   the most recent snapshot PER backup group (per VM/CT). If you have 9
   groups, it syncs 9 snapshots (one per group), not 1 total.

## Related References

- `references/openstack-offsite-pbs-2026-07.md` — Remote PBS provisioning on OpenStack, PVE integration, HA backup options
- `references/pbs-lxc-setup-2026-07.md` — Local PBS in LXC setup, RBD/Ceph workarounds, PVE integration, **owner-mismatch pitfall (#11)** and **skip-external-VMs pitfall (#12)**
- `references/infra-monitoring-2026-07.md` — Prometheus + Grafana monitoring stack for PVE/Ceph/PBS/Galera
- `templates/openstack-pbs-setup.sh` — Reusable PBS install script for OpenStack VMs