Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,359 @@
+# PBS Sync Pipeline: Local → Remote with Verify + Prune
+
+## When to Use
+
+When you need a multi-stage backup pipeline: backup locally for fast restores,
+async-push to offsite PBS, verify on both sides, then prune local aggressively
+while retaining remote long-term. This is the "tiered backup" pattern.
+
+## ⚠️ Important: Direct vzdump May Be Better Than Sync
+
+**Confirmed 2026-07-05**: The sync pipeline sounds elegant but the initial
+push sync over WAN is **impractically slow**. In practice:
+
+- A 650 GB initial sync at measured 2.7 KB/s actual throughput (not the
+  theoretical 5-7 MB/s) would take **weeks**, not hours.
+- PBS push sync has significant protocol overhead — many small chunk
+  round-trips over WAN make it far slower than raw vzdump.
+- Meanwhile, **direct vzdump to the remote PBS** already populated the
+  remote datastore with the same deduplicated chunks.
+- When the sync job finally ran, it reported `"found no new data to push"`
+  because the remote already had identical chunks from direct vzdump.
+
+### Recommended Strategy
+
+| Pattern | When to Use | Why |
+|---------|------------|-----|
+| **Direct vzdump → remote PBS** | Always, for ongoing offsite backups | Dedup handles incrementals naturally, no sync overhead |
+| **Local PBS (separate job)** | For fast local restores | Independent schedule + retention |
+| **Push sync job** | Only if local PBS has backups the remote DOESN'T have | E.g. migrating to a new remote PBS |
+
+Don't set up the sync pipeline expecting it to be the primary offsite mechanism.
+Set up **two independent vzdump jobs** (local + remote) with different retention.
+The sync job is a migration tool, not a daily driver.
+
+### ⚠️ Sync "found no new data to push" — Check for Owner Mismatch
+
+If `sync-job run` immediately returns `"found no new data to push"` but you
+know the remote is missing backups that exist locally, **the local backups
+may have been failing silently for months due to an owner mismatch** (see
+`references/pbs-lxc-setup-2026-07.md` pitfall #11).
+
+Debugging path (discovered 2026-07-05):
+
+1. Check PVE task history for the local backup job — look for non-OK statuses:
+   ```bash
+   pvesh get /nodes/<node>/tasks --limit 50 --output-format json | python3 -c "
+   import json,sys
+   for t in json.load(sys.stdin):
+       if t.get('type')=='vzdump':
+           print(f\"{t['starttime']} {t['status']} id={t.get('id','')}\")"
+   ```
+   Status `job errors` (not `OK`) means individual VM backups failed.
+
+2. Read the task log to find the owner error:
+   ```bash
+   pvesh get /nodes/<node>/tasks/<UPID>/log --output-format json
+   ```
+   Look for: `backup owner check failed (admin@pbs != root@pam)`
+
+3. Check owner files on the PBS datastore:
+   ```bash
+   pct exec 116 -- bash -c "
+     for d in /mnt/datastore/noris/vm/*/ /mnt/datastore/noris/ct/*/; do
+       echo \"$(echo $d | sed 's|.*/noris/||;s|/$||'): $(cat ${d}owner)\"
+     done"
+   ```
+
+4. Fix: overwrite all owner files to match the PVE storage config username:
+   ```bash
+   pct exec 116 -- bash -c '
+     for d in /mnt/datastore/noris/vm/*/ /mnt/datastore/noris/ct/*/; do
+       echo "admin@pbs" > "${d}owner"
+     done'
+   ```
+
+5. After fixing, trigger a manual backup to confirm it works:
+   ```bash
+   # On the PVE node where the VM runs
+   vzdump 106 --storage noris_s3 --mode snapshot --quiet 1
+   ```
+   Watch the PBS task log for progress (not the PVE log — PVE `--quiet`
+   suppresses progress output):
+   ```bash
+   pct exec 116 -- find /var/log/proxmox-backup/tasks -name "*<timestamp>*" -exec tail -5 {} \;
+   ```
+
+**Root cause explained**: The local backup job was failing every night
+with `backup owner check failed`, but the overall job status showed
+`job errors` (not a hard crash), and other VMs in the same job succeeded.
+Meanwhile, the remote PBS already had backups from the direct `offsite-ha-daily`
+vzdump job at 02:00. So when the sync job ran, the remote already had the
+same deduplicated chunks — hence "no new data to push". The sync wasn't
+broken; the **local backups were never being written**.
+
+### Measuring Actual WAN Throughput
+
+Don't trust the PBS task log — it goes silent during transfer. Measure
+actual network throughput with `/proc/net/dev` deltas:
+
+```bash
+# On local PBS (CT 116):
+T1=$(ssh root@pve-node "pct exec 116 -- cat /proc/net/dev | grep eth0" | awk '{print $2}')
+sleep 5
+T2=$(ssh root@pve-node "pct exec 116 -- cat /proc/net/dev | grep eth0" | awk '{print $2}')
+echo "RX rate: $(( (T2 - T1) / 5 )) bytes/sec"
+
+# On remote PBS:
+T1=$(ssh ubuntu@<REMOTE_IP> 'cat /proc/net/dev | grep ens3' | awk '{print $2}')
+sleep 5
+T2=$(ssh ubuntu@<REMOTE_IP> 'cat /proc/net/dev | grep ens3' | awk '{print $2}')
+echo "RX rate: $(( (T2 - T1) / 5 )) bytes/sec"
+```
+
+If RX rate is < 5 KB/s while a sync task shows "running", the sync is
+effectively stalled — the PBS sync protocol is doing negligible work.
+Kill it and use direct vzdump instead.
+
+## Architecture
+
+```
+  23:00  vzdump → lokaler PBS (noris datastore)
+     ↓
+  02:30  verify-noris      (local integrity check)
+     ↓
+  03:00  sync-to-offsite   (push last backup → remote PBS)
+  03:00  prune-offsite      (remote: keep-last=3, weekly=4, monthly=6)
+     ↓
+  04:00  verify-offsite    (remote integrity check)
+     ↓
+  05:00  prune-noris        (local: keep-last=1 only — frees space)
+```
+
+### Key Design Decisions
+
+1. **Push direction**: Local PBS pushes to remote (not pull). Remote PBS
+   doesn't initiate connections to the local network.
+2. **`--transfer-last 1`**: Only sync the most recent snapshot per group.
+   Reduces bandwidth on daily sync.
+3. **Local keep-last=1**: After sync+verify, only the latest backup stays
+   local for quick restore. Older backups are pruned.
+4. **Remote long retention**: keep-last=3 + keep-weekly=4 + keep-monthly=6
+   provides ~6 months of offsite history.
+5. **PBS has no native "delete-local-after-remote-verify"**: The pipeline
+   approximates this with staggered schedules. Between 03:00 (sync) and
+   05:00 (local prune), the backup exists on both sides.
+
+## Prerequisites
+
+- Local PBS (e.g. CT 116) with datastore `noris`
+- Remote PBS (e.g. OpenStack VM) with datastore `offsite`
+- Remote PBS registered on local PBS as a "remote"
+
+## Step-by-Step Setup
+
+### 1. Register Remote PBS on Local PBS
+
+```bash
+# On local PBS (inside CT 116 via pct exec)
+proxmox-backup-manager remote create noris-offsite \
+  --host <REMOTE_IP> \
+  --port 8007 \
+  --auth-id admin@pbs \
+  --password <PASSWORD>
+
+# CRITICAL: Add fingerprint — push will fail with SSL cert error without it
+# Get fingerprint from remote PBS:
+#   ssh ubuntu@<REMOTE_IP> 'sudo proxmox-backup-manager cert info | grep Fingerprint'
+proxmox-backup-manager remote update noris-offsite \
+  --fingerprint <SHA256_FINGERPRINT>
+
+# Verify
+proxmox-backup-manager remote list
+```
+
+### 2. Create Push Sync Job
+
+```bash
+proxmox-backup-manager sync-job create sync-to-offsite \
+  --remote noris-offsite \
+  --remote-store offsite \
+  --store noris \
+  --sync-direction push \
+  --schedule 03:00 \
+  --transfer-last 1
+```
+
+### 3. Create Local Verify Job
+
+```bash
+proxmox-backup-manager verify-job create verify-noris \
+  --store noris \
+  --schedule 02:30
+```
+
+### 4. Create Local Prune Job (Aggressive)
+
+```bash
+proxmox-backup-manager prune-job create prune-noris \
+  --store noris \
+  --schedule 05:00 \
+  --keep-last 1 \
+  --comment "Minimal local retention after offsite sync"
+```
+
+### 5. Create Remote Verify + Prune Jobs
+
+On the remote PBS:
+
+```bash
+# Verify job
+proxmox-backup-manager verify-job create verify-offsite \
+  --store offsite \
+  --schedule 04:00
+
+# Prune job (long retention)
+proxmox-backup-manager prune-job create prune-offsite \
+  --store offsite \
+  --schedule 03:00 \
+  --keep-last 3 \
+  --keep-weekly 4 \
+  --keep-monthly 6
+```
+
+### 6. Run Initial Sync Manually
+
+The first sync transfers all data (full, not incremental). Run manually
+to avoid timing out the scheduled job:
+
+```bash
+# On local PBS
+proxmox-backup-manager push noris noris-offsite offsite --transfer-last 1
+```
+
+This may take a long time (hours for hundreds of GB). Subsequent syncs
+are incremental (only new chunks transferred).
+
+### 7. Verify All Jobs
+
+```bash
+# Local PBS
+proxmox-backup-manager verify-job list
+proxmox-backup-manager prune-job list
+proxmox-backup-manager sync-job list    # ⚠️ see pitfall #1
+
+# Remote PBS
+proxmox-backup-manager verify-job list
+proxmox-backup-manager prune-job list
+```
+
+## Manual Verification
+
+Run a one-time verify on any PBS datastore:
+
+```bash
+proxmox-backup-manager verify <datastore>
+```
+
+Example output (successful):
+```
+verify datastore offsite
+found 2 groups
+verify group offsite:ct/116 (1 snapshots)
+  verified 382.77/967.64 MiB in 1.89 seconds (0 errors)
+verify group offsite:vm/106 (2 snapshots)
+  verified 17222.64/52576.00 MiB in 103.62 seconds (0 errors)
+TASK OK
+```
+
+## Monitoring Running Tasks
+
+```bash
+# List running/completed tasks
+proxmox-backup-manager task list
+
+# View task log
+proxmox-backup-manager task log <UPID>
+
+# Stop a running task
+proxmox-backup-manager task stop <UPID>
+```
+
+## Sync-Job Config File Location
+
+Sync job configs are stored at `/etc/proxmox-backup/sync.cfg`:
+
+```
+sync: sync-to-offsite
+    comment Push last backup to offsite
+    remote noris-offsite
+    remote-store offsite
+    schedule 03:00
+    store noris
+    sync-direction push
+    transfer-last 1
+```
+
+## Pitfalls
+
+1. **`sync-job list` may not display push jobs** — In PBS 3.4.8, `sync-job list`
+   returns empty even when a push sync job exists. The job IS created and
+   stored in `/etc/proxmox-backup/sync.cfg`. Verify by reading the config file
+   directly. The scheduled job still runs correctly.
+
+2. **Remote fingerprint required before push** — `proxmox-backup-manager push`
+   fails with `SSL routines:tls_post_process_server_certificate:certificate
+   verify failed` if the remote's fingerprint isn't registered. Always run
+   `remote update --fingerprint` after `remote create`.
+
+3. **`prune-job create` rejects keep-*=0** — PBS enforces minimum value of 1
+   for all keep-* parameters. To effectively disable a retention dimension,
+   simply omit it (don't set it to 0). E.g. `--keep-last 1` alone keeps only
+   1 backup with no weekly/monthly/yearly retention.
+
+4. **Initial sync is VERY slow — plan for hours, not minutes** — First push
+   transfers all deduplicated chunks. For ~650 GB at ~5-7 MB/s WAN speed, expect
+   **25-30+ hours** for the initial sync. The 300s timeout is nowhere near
+   enough. Run manually with a long timeout or let the scheduled job handle it
+   overnight (or over multiple nights). Subsequent daily syncs are incremental
+   and transfer only new/changed chunks (typically MB, not GB).
+
+   **User reality check**: "in einer stunde dürfte das nicht hochgeladen sein" —
+   correct. At 5 MB/s, 650 GB takes ~36 hours. Always calculate:
+   `hours = total_GB * 1024 / (speed_MBps * 3600)`.
+
+5. **Push sync task log shows NO progress** — The task log only records the
+   initial lines ("Found 9 groups to sync", "skipped: N snapshot(s)") and then
+   goes silent for the entire transfer. No progress bars, no chunk counts.
+   To monitor actual progress, check the **remote side**:
+   ```bash
+   # On remote PBS — watch disk usage grow
+   ssh ubuntu@<REMOTE_IP> 'df -h /mnt/datastore/offsite; sudo du -sh /mnt/datastore/offsite/.chunks/'
+   ```
+   Compare before/after to estimate transfer rate. The `.chunks/` directory
+   grows as chunks arrive.
+
+6. **`--transfer-last 1` with many groups still moves lots of data** —
+   `--transfer-last 1` syncs 1 snapshot PER GROUP. With 9 groups (multiple
+   VMs/CTs), that's 9 snapshots total. Large VMs (e.g. 50+ GB HA VM) dominate
+   the transfer. Consider filtering groups with `--group-filter` to sync only
+   critical VMs first, then expand.
+
+7. **GC (garbage collection) runs by default** — PBS enables daily GC at
+   midnight by default. No explicit configuration needed. GC reclaims
+   space from pruned chunks.
+
+6. **Schedule staggering matters** — Ensure verify runs BEFORE sync (so you
+   don't push corrupt data) and local prune runs AFTER sync+remote-verify
+   (so you don't delete local before remote is confirmed good).
+
+7. **`--transfer-last N` limits per-group** — `--transfer-last 1` syncs only
+   the most recent snapshot PER backup group (per VM/CT). If you have 9
+   groups, it syncs 9 snapshots (one per group), not 1 total.
+
+## Related References
+
+- `references/openstack-offsite-pbs-2026-07.md` — Remote PBS provisioning on OpenStack, PVE integration, HA backup options
+- `references/pbs-lxc-setup-2026-07.md` — Local PBS in LXC setup, RBD/Ceph workarounds, PVE integration, **owner-mismatch pitfall (#11)** and **skip-external-VMs pitfall (#12)**
+- `references/infra-monitoring-2026-07.md` — Prometheus + Grafana monitoring stack for PVE/Ceph/PBS/Galera
+- `templates/openstack-pbs-setup.sh` — Reusable PBS install script for OpenStack VMs