dominik/hermes-skills

Fork 0

Files

T

Debian e9cc106625 Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00

13 KiB

Raw Blame History

PBS Sync Pipeline: Local → Remote with Verify + Prune

When to Use

When you need a multi-stage backup pipeline: backup locally for fast restores, async-push to offsite PBS, verify on both sides, then prune local aggressively while retaining remote long-term. This is the "tiered backup" pattern.

⚠️ Important: Direct vzdump May Be Better Than Sync

Confirmed 2026-07-05: The sync pipeline sounds elegant but the initial push sync over WAN is impractically slow. In practice:

A 650 GB initial sync at measured 2.7 KB/s actual throughput (not the theoretical 5-7 MB/s) would take weeks, not hours.
PBS push sync has significant protocol overhead — many small chunk round-trips over WAN make it far slower than raw vzdump.
Meanwhile, direct vzdump to the remote PBS already populated the remote datastore with the same deduplicated chunks.
When the sync job finally ran, it reported "found no new data to push" because the remote already had identical chunks from direct vzdump.

Recommended Strategy

Pattern	When to Use	Why
Direct vzdump → remote PBS	Always, for ongoing offsite backups	Dedup handles incrementals naturally, no sync overhead
Local PBS (separate job)	For fast local restores	Independent schedule + retention
Push sync job	Only if local PBS has backups the remote DOESN'T have	E.g. migrating to a new remote PBS

Don't set up the sync pipeline expecting it to be the primary offsite mechanism. Set up two independent vzdump jobs (local + remote) with different retention. The sync job is a migration tool, not a daily driver.

⚠️ Sync "found no new data to push" — Check for Owner Mismatch

If sync-job run immediately returns "found no new data to push" but you know the remote is missing backups that exist locally, the local backups may have been failing silently for months due to an owner mismatch (see references/pbs-lxc-setup-2026-07.md pitfall #11).

Debugging path (discovered 2026-07-05):

Check PVE task history for the local backup job — look for non-OK statuses:

pvesh get /nodes/<node>/tasks --limit 50 --output-format json | python3 -c "
import json,sys
for t in json.load(sys.stdin):
    if t.get('type')=='vzdump':
        print(f\"{t['starttime']} {t['status']} id={t.get('id','')}\")"

Status job errors (not OK) means individual VM backups failed.

Read the task log to find the owner error:
```
pvesh get /nodes/<node>/tasks/<UPID>/log --output-format json
```
Look for: backup owner check failed (admin@pbs != root@pam)

Check owner files on the PBS datastore:

pct exec 116 -- bash -c "
  for d in /mnt/datastore/noris/vm/*/ /mnt/datastore/noris/ct/*/; do
    echo \"$(echo $d | sed 's|.*/noris/||;s|/$||'): $(cat ${d}owner)\"
  done"

Fix: overwrite all owner files to match the PVE storage config username:

pct exec 116 -- bash -c '
  for d in /mnt/datastore/noris/vm/*/ /mnt/datastore/noris/ct/*/; do
    echo "admin@pbs" > "${d}owner"
  done'

After fixing, trigger a manual backup to confirm it works:

# On the PVE node where the VM runs
vzdump 106 --storage noris_s3 --mode snapshot --quiet 1

Watch the PBS task log for progress (not the PVE log — PVE --quiet suppresses progress output):

pct exec 116 -- find /var/log/proxmox-backup/tasks -name "*<timestamp>*" -exec tail -5 {} \;

Root cause explained: The local backup job was failing every night with backup owner check failed, but the overall job status showed job errors (not a hard crash), and other VMs in the same job succeeded. Meanwhile, the remote PBS already had backups from the direct offsite-ha-daily vzdump job at 02:00. So when the sync job ran, the remote already had the same deduplicated chunks — hence "no new data to push". The sync wasn't broken; the local backups were never being written.

Measuring Actual WAN Throughput

Don't trust the PBS task log — it goes silent during transfer. Measure actual network throughput with /proc/net/dev deltas:

# On local PBS (CT 116):
T1=$(ssh root@pve-node "pct exec 116 -- cat /proc/net/dev | grep eth0" | awk '{print $2}')
sleep 5
T2=$(ssh root@pve-node "pct exec 116 -- cat /proc/net/dev | grep eth0" | awk '{print $2}')
echo "RX rate: $(( (T2 - T1) / 5 )) bytes/sec"

# On remote PBS:
T1=$(ssh ubuntu@<REMOTE_IP> 'cat /proc/net/dev | grep ens3' | awk '{print $2}')
sleep 5
T2=$(ssh ubuntu@<REMOTE_IP> 'cat /proc/net/dev | grep ens3' | awk '{print $2}')
echo "RX rate: $(( (T2 - T1) / 5 )) bytes/sec"

If RX rate is < 5 KB/s while a sync task shows "running", the sync is effectively stalled — the PBS sync protocol is doing negligible work. Kill it and use direct vzdump instead.

Architecture

  23:00  vzdump → lokaler PBS (noris datastore)
     ↓
  02:30  verify-noris      (local integrity check)
     ↓
  03:00  sync-to-offsite   (push last backup → remote PBS)
  03:00  prune-offsite      (remote: keep-last=3, weekly=4, monthly=6)
     ↓
  04:00  verify-offsite    (remote integrity check)
     ↓
  05:00  prune-noris        (local: keep-last=1 only — frees space)

Key Design Decisions

Push direction: Local PBS pushes to remote (not pull). Remote PBS doesn't initiate connections to the local network.
--transfer-last 1: Only sync the most recent snapshot per group. Reduces bandwidth on daily sync.
Local keep-last=1: After sync+verify, only the latest backup stays local for quick restore. Older backups are pruned.
Remote long retention: keep-last=3 + keep-weekly=4 + keep-monthly=6 provides ~6 months of offsite history.
PBS has no native "delete-local-after-remote-verify": The pipeline approximates this with staggered schedules. Between 03:00 (sync) and 05:00 (local prune), the backup exists on both sides.

Prerequisites

Local PBS (e.g. CT 116) with datastore noris
Remote PBS (e.g. OpenStack VM) with datastore offsite
Remote PBS registered on local PBS as a "remote"

Step-by-Step Setup

1. Register Remote PBS on Local PBS

# On local PBS (inside CT 116 via pct exec)
proxmox-backup-manager remote create noris-offsite \
  --host <REMOTE_IP> \
  --port 8007 \
  --auth-id admin@pbs \
  --password <PASSWORD>

# CRITICAL: Add fingerprint — push will fail with SSL cert error without it
# Get fingerprint from remote PBS:
#   ssh ubuntu@<REMOTE_IP> 'sudo proxmox-backup-manager cert info | grep Fingerprint'
proxmox-backup-manager remote update noris-offsite \
  --fingerprint <SHA256_FINGERPRINT>

# Verify
proxmox-backup-manager remote list

2. Create Push Sync Job

proxmox-backup-manager sync-job create sync-to-offsite \
  --remote noris-offsite \
  --remote-store offsite \
  --store noris \
  --sync-direction push \
  --schedule 03:00 \
  --transfer-last 1

3. Create Local Verify Job

proxmox-backup-manager verify-job create verify-noris \
  --store noris \
  --schedule 02:30

4. Create Local Prune Job (Aggressive)

proxmox-backup-manager prune-job create prune-noris \
  --store noris \
  --schedule 05:00 \
  --keep-last 1 \
  --comment "Minimal local retention after offsite sync"

5. Create Remote Verify + Prune Jobs

On the remote PBS:

# Verify job
proxmox-backup-manager verify-job create verify-offsite \
  --store offsite \
  --schedule 04:00

# Prune job (long retention)
proxmox-backup-manager prune-job create prune-offsite \
  --store offsite \
  --schedule 03:00 \
  --keep-last 3 \
  --keep-weekly 4 \
  --keep-monthly 6

6. Run Initial Sync Manually

The first sync transfers all data (full, not incremental). Run manually to avoid timing out the scheduled job:

# On local PBS
proxmox-backup-manager push noris noris-offsite offsite --transfer-last 1

This may take a long time (hours for hundreds of GB). Subsequent syncs are incremental (only new chunks transferred).

7. Verify All Jobs

# Local PBS
proxmox-backup-manager verify-job list
proxmox-backup-manager prune-job list
proxmox-backup-manager sync-job list    # ⚠️ see pitfall #1

# Remote PBS
proxmox-backup-manager verify-job list
proxmox-backup-manager prune-job list

Manual Verification

Run a one-time verify on any PBS datastore:

proxmox-backup-manager verify <datastore>

Example output (successful):

verify datastore offsite
found 2 groups
verify group offsite:ct/116 (1 snapshots)
  verified 382.77/967.64 MiB in 1.89 seconds (0 errors)
verify group offsite:vm/106 (2 snapshots)
  verified 17222.64/52576.00 MiB in 103.62 seconds (0 errors)
TASK OK

Monitoring Running Tasks

# List running/completed tasks
proxmox-backup-manager task list

# View task log
proxmox-backup-manager task log <UPID>

# Stop a running task
proxmox-backup-manager task stop <UPID>

Sync-Job Config File Location

Sync job configs are stored at /etc/proxmox-backup/sync.cfg:

sync: sync-to-offsite
    comment Push last backup to offsite
    remote noris-offsite
    remote-store offsite
    schedule 03:00
    store noris
    sync-direction push
    transfer-last 1

Pitfalls

sync-job list may not display push jobs — In PBS 3.4.8, sync-job list returns empty even when a push sync job exists. The job IS created and stored in /etc/proxmox-backup/sync.cfg. Verify by reading the config file directly. The scheduled job still runs correctly.
Remote fingerprint required before push — proxmox-backup-manager push fails with SSL routines:tls_post_process_server_certificate:certificate verify failed if the remote's fingerprint isn't registered. Always run remote update --fingerprint after remote create.
prune-job create rejects keep-*=0 — PBS enforces minimum value of 1 for all keep-* parameters. To effectively disable a retention dimension, simply omit it (don't set it to 0). E.g. --keep-last 1 alone keeps only 1 backup with no weekly/monthly/yearly retention.
Initial sync is VERY slow — plan for hours, not minutes — First push transfers all deduplicated chunks. For ~650 GB at ~5-7 MB/s WAN speed, expect 25-30+ hours for the initial sync. The 300s timeout is nowhere near enough. Run manually with a long timeout or let the scheduled job handle it overnight (or over multiple nights). Subsequent daily syncs are incremental and transfer only new/changed chunks (typically MB, not GB).

User reality check: "in einer stunde dürfte das nicht hochgeladen sein" — correct. At 5 MB/s, 650 GB takes ~36 hours. Always calculate: hours = total_GB * 1024 / (speed_MBps * 3600).
Push sync task log shows NO progress — The task log only records the initial lines ("Found 9 groups to sync", "skipped: N snapshot(s)") and then goes silent for the entire transfer. No progress bars, no chunk counts. To monitor actual progress, check the remote side:
```
# On remote PBS — watch disk usage grow
ssh ubuntu@<REMOTE_IP> 'df -h /mnt/datastore/offsite; sudo du -sh /mnt/datastore/offsite/.chunks/'
```
Compare before/after to estimate transfer rate. The .chunks/ directory grows as chunks arrive.
--transfer-last 1 with many groups still moves lots of data — --transfer-last 1 syncs 1 snapshot PER GROUP. With 9 groups (multiple VMs/CTs), that's 9 snapshots total. Large VMs (e.g. 50+ GB HA VM) dominate the transfer. Consider filtering groups with --group-filter to sync only critical VMs first, then expand.
GC (garbage collection) runs by default — PBS enables daily GC at midnight by default. No explicit configuration needed. GC reclaims space from pruned chunks.
Schedule staggering matters — Ensure verify runs BEFORE sync (so you don't push corrupt data) and local prune runs AFTER sync+remote-verify (so you don't delete local before remote is confirmed good).
--transfer-last N limits per-group — --transfer-last 1 syncs only the most recent snapshot PER backup group (per VM/CT). If you have 9 groups, it syncs 9 snapshots (one per group), not 1 total.

references/openstack-offsite-pbs-2026-07.md — Remote PBS provisioning on OpenStack, PVE integration, HA backup options
references/pbs-lxc-setup-2026-07.md — Local PBS in LXC setup, RBD/Ceph workarounds, PVE integration, owner-mismatch pitfall (#11) and skip-external-VMs pitfall (#12)
references/infra-monitoring-2026-07.md — Prometheus + Grafana monitoring stack for PVE/Ceph/PBS/Galera
templates/openstack-pbs-setup.sh — Reusable PBS install script for OpenStack VMs

13 KiB Raw Blame History