hermes-skills/devops/seafile-api/references/seafile-immich-backup-strategy.md

# Seafile & Immich Backup Strategy Analysis

## When to Use

When planning offsite/backup strategy for Seafile (558 GB) or Immich
on bandwidth-limited remote PBS, and deciding what to back up vs.
what to leave on Ceph-only redundancy.

## Seafile Backup Strategies (6 Options)

### Context

- Seafile Pro 13.0.19 MC Docker edition on CT111
- File blocks: `/opt/seafile-data/seafile/seafile-data/storage/blocks/` (intrinsically deduplicated, content-addressable)
- Config: `/opt/seafile/.env`, `seafile.conf`, `seahub_settings.py`
- DB: MySQL (`seafile_db`, `ccnet_db`, `seahub_db`) — currently in `seafile-mysql` container, migration to Galera planned
- Total size: ~558 GB (mostly file blocks)
- Ceph redundancy: blocks stored on Ceph EC4+1 (5 HDD OSDs, 1-failure tolerance)

### Option Matrix

| # | Strategy | Offsite Volume | Pros | Cons |
|---|----------|---------------|------|------|
| 1 | **PBS Full (current)** | 558 GB init, then delta | Simple, everything included | Huge for offsite, slow restore |
| 2 | **Config+DB only** | <1 GB | Fast daily offsite, trivial restore | Blocks only in Ceph — total Ceph loss = all files gone |
| 3 | **SeafGC + PBS exclude blocks** | ~2 GB | Clean, regularly garbage-collected | Blocks only in Ceph, needs GC discipline |
| 4 | **Seafile Native Backup** | Variable | Official way, incremental | Needs second Seafile server (not S3) |
| 5 | **RBD Snapshot + PBS** | 558 GB init, delta like #1 | Point-in-time consistent | Ceph pool must support snapshots |
| 6 | **Blocks on S3 (Pro feature)** | Rootfs ~2 GB offsite, blocks on S3 | Clean separation, Ceph relieved | Migration effort, S3 costs |

### Recommendation

- **Short-term**: Option 2 (Config+DB offsite) + regular `seaf-gc.sh`. Blocks
  are redundant in Ceph EC4+1. For DR against total Ceph loss: Option 4 as
  second pillar.
- **Long-term**: Option 6 (S3 storage backend) — moves blocks out of Ceph
  entirely, making PBS offsite trivial (~2 GB) and Ceph only hosts config+DB.

## Seafile Pro S3 Storage Backend

Seafile Pro supports S3 as **primary storage backend** (not backup target).
All file blocks are stored directly on S3 instead of local disk/Ceph.

### Configuration (in `seafile.conf`)

```ini
[storage_backend]
name = s3
bucket = seafile-data
key_id = XXX
key = XXX
region = eu-central-1
endpoint = https://s3.example.com
```

### Key Distinctions

- **Primary storage** — replaces local/Ceph block storage entirely
- **NOT a backup target** — S3 is where live data lives, not a backup copy
- **Migration required** — existing blocks must be migrated to S3
- **Ceph relief** — once on S3, Ceph only holds config+DB (~2 GB), making
  PBS offsite backup trivial and fast

### When to Choose S3 Backend

- Ceph capacity is constrained (OSDs near full)
- Offsite backup of 558 GB blocks is impractical (bandwidth-limited)
- S3 storage is cheaper than maintaining Ceph OSDs
- Want clean separation: config+DB on Ceph (small, fast), blocks on S3 (large, cheap)

## Seafile Native Backup (`seaf-backup-cmd`)

### How It Works

- Uses Seafile's internal RPC API to synchronize libraries to a **second
  Seafile server**
- Incremental — only changed commits/blocks are transferred
- The second server must be a full Seafile server (same version)
- Commands: `seaf-backup-cmd status`, `seaf-backup-cmd sync <repo-id>`

### Limitations

- **Needs a second Seafile server** — not a simple file/rsync target
- **Not S3-compatible** — target must be Seafile server with storage backend
- **Setup complexity** — must configure backup server in `seafile.conf`
- **Per-library sync** — no global "backup everything" by default

### Location

```bash
# In the seafile container:
docker exec seafile /opt/seafile/seafile-server-latest/seahub/scripts/seaf-backup-cmd.sh
# Error without proper SEAFILE_DATA_DIR env — needs server running
```

### When to Choose

- Need true DR replica (second running Seafile instance)
- Have infrastructure for a second server
- Want incremental, deduplicated replication (not full copies)

## Immich Backup Considerations

### Architecture

- CT115 (immich): stopped, 20 GB rootfs — upload directory location unclear
- CT117 (immich-postgresql): running, 914 MB DB on proxmox6
- Photos/videos are NOT deduplicated (unlike Seafile blocks)

### Strategy Options

| # | Strategy | Offsite Volume | Notes |
|---|----------|---------------|-------|
| 1 | PBS Full both CTs | ~21 GB | Simple but may grow |
| 2 | DB dump + upload-dir rsync | DB ~1 GB + uploads | Selective |
| 3 | DB only offsite | <1 GB | Photos only in Ceph |
| 4 | Uploads on Ceph, DB on Galera | DB offsite <1 GB | Clean separation |

### Key Question

Where are the actual photo/video uploads stored? CT115 is stopped —
uploads may be on a mount point, external volume, or within the
PostgreSQL CT. Must identify before choosing a strategy.

## Bandwidth-Aware Offsite Backup Prioritization

When adding services to a bandwidth-limited offsite PBS:

### Tier 1: Small + Critical Infrastructure (add first)

- Traefik (~3 GB) — reverse proxy config, all routes
- Gitea (~1.2 GB) — code repositories
- Authelia, DNS — tiny but critical
- Paperless (~16 GB) — documents (medium size, high value)

### Tier 2: Medium + Important

- Litellm (~3.4 GB) — API config
- Seafile config+DB (~2 GB) — without blocks

### Tier 3: Evaluate Separately

- Seafile blocks (~558 GB) — consider S3 backend or native backup
- Immich uploads — identify storage location first

### Skip

- Frigate (~40 GB) — rewritable video, low value
- High-churn data that's easily regenerated

## PBS Offsite Job Configuration

### Adding CTs to Offsite Job

```bash
# Edit /etc/pve/jobs.cfg from any PVE node (pmxcfs replicates)
# Current: vmid 106
# Target: vmid 106,108,104,99999
sed -i '/offsite-ha-daily/,/^$/{s/vmid 106/vmid 106,108,104,99999/}' /etc/pve/jobs.cfg
```

### Initial Full Backup (Manual)

Must run `vzdump` on the node where each CT actually runs (see
`proxmox-ve-administration` → `references/pbs-lxc-setup-2026-07.md`
pitfall #17 for the silent-skip issue):

```bash
# Determine node for each CT first
pvesh get /cluster/resources --type vm --output-format json | python3 -c "
import json,sys
for v in json.load(sys.stdin):
    if v.get('vmid') in [108,99999,104]:
        print(f\"CT{v['vmid']:5d} node={v['node']}\")"

# Run on correct nodes
ssh root@10.0.20.60 "vzdump 108 --storage noris_offsite --mode snapshot --compress zstd"
ssh root@10.0.20.70 "vzdump 99999 --storage noris_offsite --mode snapshot --compress zstd"
ssh root@10.0.20.91 "vzdump 104 --storage noris_offsite --mode snapshot --compress zstd"
```

### Verification

```bash
# Check backups arrived at PBS (may need content cache refresh)
pvesh get /nodes/proxmox1/storage/noris_offsite/content --output-format json | \
  python3 -c "
import json,sys,datetime
for b in sorted(json.load(sys.stdin), key=lambda x: x.get('ctime',0), reverse=True)[:10]:
    print(f\"{b['volid']:55s} {b.get('size',0)//1024//1024:>8} MB  {datetime.datetime.fromtimestamp(b.get('ctime',0)).strftime('%Y-%m-%d %H:%M')}\")"
```

## Session Context (2026-07-07)

- Offsite job expanded: `vmid 106` → `vmid 106,108,104,99999`
- Initial full backups triggered manually on correct nodes
- CT111 (Seafile) backup showed 0 MB on noris_s3 — empty/failed backup
  (previous valid: 558 GB on 2026-07-06)
- Frigate excluded from all backups per user decision
- Seafile backup strategy discussion: user exploring alternatives to
  558 GB full backup, considering S3 storage backend (Pro feature)
- Immich: CT115 stopped, upload location unidentified