dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Hermes Agent Skills collection

Browse Source
This commit is contained in:
Debian
2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
devops/docker-host-administration/references/10.0.30.100-decommission.md
+126
View File
@@ -0,0 +1,126 @@
# 10.0.30.100 Decommission — Session Detail (2026-06-30)
## Host Info
- **Host**: 10.0.30.100 (Ubuntu, hostname `ubuntu`)
- **User**: dominik (password auth initially, SSH key deployed)
- **Docker**: 28.1.1, ZFS backend (`pool01_n2_redundant/docker` dataset, 2.7TB)
- **SSH Key**: `~/.ssh/hermes_pve_agent` deployed to dominik's authorized_keys
## SUDO_ASKPASS Setup
- Password: provided in chat (do not persist)
- Askpass script deployed to `/tmp/askpass.sh` on remote host
- Pattern: `SUDO_ASKPASS=/tmp/askpass.sh sudo -A COMMAND`
## Services Removed (10 initial services)
### Containers
None were running — only images and volumes remained. All 10 services had been stopped previously.
### Images Removed (~28 GB)
| Service | Images | Approx Size |
|---------|--------|-------------|
| Frigate | 0.16.0-beta2, beta4, stable | ~8.5 GB |
| Immich | server + ML + ML-openvino | ~3.0 GB |
| Paperless | 6 versions (2.9.0–2.17.1) | ~8.3 GB |
| Scrypted | latest | 2.2 GB |
| Homegear | stable | 1.5 GB |
| ioBroker | latest | 1.3 GB |
| Homebridge | ubuntu-no-avahi | 0.9 GB |
| openHAB | 3.4.5 | 0.7 GB |
| Node-RED | latest | 0.6 GB |
| Jenkins | jekyll-docker | 0.5 GB |
### Volumes Removed (20 volumes)
frigate_config, frigate_storage, frigate_postgres-data, frigate_double-take, homebridge, homegear-data-etc, homegear-data-lib, homegear-data-log, immich_model-cache, immich_pgdata, iobrokerdata, jenkins_home, nodered, openhab4_openhab4_addons, openhab4_openhab4_conf, openhab4_openhab4_userdata, paperless_data, paperless_media, paperless_redisdata, scrypted
### Bind Mounts Cleaned
- `/pool01_n2_redundant/HomeAssistant/frigate` — deleted (clips, recordings)
- `/pool01_n2_redundant/homes/dominik/.homebridge` — deleted (config, accessories)
- `/pool01_n2_redundant/Frigate/` — emptied (ZFS dataset left, "dataset is busy")
## GitLab + Runner Removal (additional)
### Containers Removed
- `development-gitlab-1` (GitLab CE 15.8.1)
- `gitlab-runner` (gitlab-runner:alpine)
- `development-register-runner-1` (registration container, already exited)
- `development-dind-1` (Docker-in-Docker, **was causing restart-loop**)
### Images Removed
- gitlab/gitlab-ce:15.8.1-ce.0
- gitlab/gitlab-runner:alpine
- docker:20-dind
- docker:dind
### Volumes Removed
- gitlab_config, gitlab_data, gitlab_logs
- development_gitlab-runner
- 2× runner cache volumes (runner-xtxmd-6p-project-11-concurrent-0-cache-*)
## InfluxDB + Home Assistant Volume Removal
### Volumes Deleted (~273 GB freed)
| Volume | Size | Content | Data Range |
|--------|------|---------|------------|
| `506f355d...` (anonymous) | 270 GB | InfluxDB engine (data + wal) | Oct 2022 – Aug 2025 |
| `homeassistant_home_assistant_config` | 2.7 GB | HA config (duplicate, HA runs on 10.0.30.10) | — |
## Remaining Volumes on Host (analysis done, not yet acted on)
### Large Volumes
| Volume | Size | Content | Data Range |
|--------|------|---------|------------|
| seafile_data | 246 GB | Seafile (logs, nginx, seafile-conf) | — |
| seafile_custom_data | 226 GB | Seafile storage (blocks, commits, fs) | Mar 2015 – May 2025 |
| registry_lib | 1.4 GB | Docker Registry | — |
### Medium Volumes
| Volume | Size | Content |
|--------|------|---------|
| embyconfig | 860 MB | Emby Media Server |
| trivy_cache | 433 MB | Trivy Security Scanner |
| jellyfin_jellyfin_config | 314 MB | Jellyfin |
| matomo | 134 MB | Matomo Analytics |
| monitoring_prometheus_data | 146 MB | Prometheus |
| monitoring_grafana_db | 61 MB | Grafana |
| portainer_data | 63 MB | Portainer |
| pgdata | 46 MB | PostgreSQL |
| ghost-imkerei | 43 MB | Ghost Blog |
| openproject_postgresql_data | 25 MB | OpenProject |
### Small Volumes (<10 MB)
grafana, vicare, handbrake_config, openproject_assets, spoolman, proxy_conf, traefik_acme, telegraf, sabnzbd, bitmonero, ethereum, bienen_proxy, ghost_proxy, xenia_*, ~60 anonymous hash volumes (mostly 7 KB each, one 943 KB)
## Docker Daemon Instability
### Root Cause
`development-dind-1` container had `restart: always` and crashed on boot, creating a restart-loop. This made the Docker daemon extremely slow/unresponsive:
- `docker images` hung or returned empty (false verification!)
- `docker rmi` timed out
- `docker volume rm` timed out
- `docker ps` worked but slowly
### Resolution
1. `kill -9 $(pgrep dockerd)` — force kill daemon
2. `systemctl start docker` — restart
3. Wait 30-60s for ZFS backend to initialize
4. Quickly `docker rm -f development-dind-1` before it restarts
5. After removal, daemon became stable
### False Verification Incident
Initial verification showed `docker images | grep -i frigate` returning empty → concluded "all images deleted." After daemon restart, images were visible again. Lesson: always verify daemon stability before trusting `docker images` output.
## Remaining Containers After Cleanup
- portainer (Up)
- Traefik (Up)
- spoolman-spoolman-1 (Exited)
- website-imkerei-bienen-proxy-1 (Up)
- website-imkerei-proxy-1 (Up)
- website-imkerei-website-1 (Up)
## Model ID Drift (noris.ai)
- Old model ID: `vllm/gemma-4-31b-it-dynamo` (no longer available)
- New model ID: `vllm/gemma-4-31b-it` (verified via `/v1/models` endpoint)
- Also available: `vllm/prod/gemma-4-31b-it`
- Fixed in: `recipe_scraper_v8_intl.py` line 108
- Updated: cronjob prompt, memory entry
devops/docker-host-administration/references/10.0.30.100-docker-analysis-2026-07.md
+207
View File
@@ -0,0 +1,207 @@
# Docker Analysis: 10.0.30.100 (2026-07-04)
## Host Info
- **Host**: ubuntu (10.0.30.100)
- **Storage Driver**: ZFS on `pool01_n2_redundant/docker`
- **Docker Root**: `/var/lib/docker`
- **Dataset USED**: 811 GB (includes snapshot overhead)
- **Dataset REFER**: 491 GB (live data)
- **Snapshot Overhead**: ~320 GB (deleted layers held by ZFS snapshots)
- **ZFS Subdatasets**: 735 (was 744 before 2 paperless tags removed)
- **Images**: 80 (before partial cleanup)
- **Containers**: 5 running (47 config dirs total — many dead/stopped)
- **Volumes**: 117 named + anonymous
## Daemon State: PARTIALLY RECOVERED
Docker daemon was `active` but effectively non-functional:
- `docker system df` crashed: `error getting build cache usage: failed to get usage for HASH: layer does not exist`
- `docker ps` sometimes worked (intermittently), sometimes timed out
- `docker images --format` returned empty output with exit 0 (silent failure)
- `docker system prune` failed: `a prune operation is already running`
- `docker rmi` for some images worked (2 paperless tags removed), most timed out
- `journalctl -u docker` showed: `error creating zfs mount: mount pool/docker/HASH: no such file or directory` and `Error unmounting /var/lib/docker/zfs/graph/HASH: invalid argument`
## Recovery Attempt & Outcome
1. `systemctl restart docker` — hung in `deactivating` for 60s, then `stop-sigterm` timeout, SIGKILL
2. `dockerd` became zombie (`<defunct>`)
3. Killed leftover `containerd-shim-runc-v2` process
4. `systemctl restart containerd` — succeeded
5. `systemctl start docker` — entered `activating` state
6. After 2-3 minutes, daemon reached `active` (ZFS graphdriver scanned 735 subdatasets)
7. After restart: `docker ps` worked (5 containers up), but `docker images` returned empty and `docker system df` timed out
8. Triggered `docker image prune -a` — spawned `zfs destroy -r` for each layer, each entering D-state due to I/O contention with concurrent rsync
9. `zfs destroy` processes hung in D-state (uninterruptible) — blocked all subsequent Docker operations until each completed sequentially
10. Daemon remained `active` throughout — `docker ps` continued to work, but `docker images` stayed empty and `docker system df` timed out
**Outcome**: Daemon IS up and serving containers, but graphdriver is degraded. `docker ps` works; `docker images` returns empty; `docker system df` times out. The `zfs destroy` cascade from `image prune -a` is still running sequentially in the background.
## `zfs destroy` D-State Under I/O Contention
When Docker's ZFS graphdriver cleans up layers, it spawns `zfs destroy -r pool/docker/<hash>` for each removed layer. Under heavy pool I/O (e.g. concurrent rsync reading 300+ GB from the same ZFS pool), each `zfs destroy` enters **D-state (uninterruptible sleep)** and can take minutes per layer.
**Symptoms:**
- `ps aux | grep "zfs destroy"` shows process in `D` state
- Docker CLI commands (`docker images`, `docker system df`) time out while destroy runs
- `zpool iostat` shows high read throughput (rsync) but near-zero writes (destroy blocked)
- The dataset being destroyed may already report "does not exist" in `zfs list` — the destroy is stuck in kernel metadata flush, not actual data deletion
**Key insight:** This is NOT a permanent hang — `zfs destroy` eventually completes, but each one is serialized and I/O-starved. Docker appears broken during this window but is actually just waiting for ZFS to finish cleaning up.
**Recovery:** Patience. Don't kill the `zfs destroy` process (D-state is immune to SIGKILL anyway). Wait for I/O contention to ease (e.g. let rsync finish), then Docker operations resume. If urgent, reduce concurrent I/O on the pool.
## Root Cause
735 ZFS subdatasets for Docker layers, several with corrupt ZFS mounts (`no such file or directory`). The ZFS graphdriver tries to mount each subdataset during daemon startup; corrupt ones cause indefinite hangs. The daemon cannot complete initialization.
## Complete Volume Size Inventory
### Named Volumes (all significant sizes)
| Volume | Size | Purpose |
|--------|------|---------|
| embyconfig | 860 MB | Emby Media Server config |
| jellyfin_jellyfin_config | 314 MB | Jellyfin config |
| monitoring_prometheus_data | 146 MB | Prometheus metrics |
| matomo | 134 MB | Matomo analytics |
| monitoring_grafana_db | 61 MB | Grafana dashboards |
| ghost-imkerei | 43 MB | Ghost blog (Imkerei) |
| jellyfin_jellyfin_cache | 18 MB | Jellyfin cache |
| grafana | 7.2 MB | Grafana (legacy?) |
| handbrake_config | 3 MB | Handbrake config |
| openproject_openproject_assets | 912 KB | OpenProject |
| monitoring_alertmanager | 14 KB | Alertmanager config |
| ethereum | 14 KB | Ethereum node |
| monitoring_prometheus_alerts | 13 KB | Prometheus alerts |
| monitoring_promtail_config | 7 KB | Promtail config |
| monitoring_loki_config | 7 KB | Loki config |
| bitmonero | 180 KB | Monero miner |
| bienen_proxy | 59 KB | nginx proxy (Imkerei) |
| ghost_proxy | 59 KB | nginx proxy (Ghost) |
**Total named volumes: ~1.5 GB.** Volumes are NOT the space problem.
### Anonymous Volumes (hash names)
~100 volumes, all < 1 MB each. Total < 5 MB. Negligible.
### Container Config Dirs
47 directories à ~100-200 KB (logs/metadata). Total ~5 MB.
### Where the 811 GB Actually Lives
| Area | Size | % of total |
|------|------|------------|
| ZFS Image Layers (735 subdatasets) | ~491 GB | 60% |
| Snapshot/Clone Overhead | ~320 GB | 40% |
| Named Volumes | ~1.5 GB | <1% |
| Anonymous Volumes | ~5 MB | ~0% |
| Container Configs | ~5 MB | ~0% |
## Seafile Stack Inventory
### Docker Volumes (11 total!)
| Volume | Content | Notes |
|--------|---------|-------|
| seafile_data | logs, nginx, seafile core | Main Seafile data |
| seafile_custom_data | commits, fs, storage, thumbnails, avatars | **Actual file storage** |
| seafile-conf | Seafile config | |
| seafile_custom_conf | Custom config | |
| seafile_custom_proxy_conf | Proxy config | |
| seafile_custom_mariadb | MariaDB data (22 entries) | Active DB? |
| seafile-mysql | MySQL data (28 entries) | Active DB? |
| seafile_mysql | MySQL data (24 entries) | Another DB instance |
| seafile-mysql.crashed | Crashed MySQL (50 entries) | ⚠️ Old crash, never cleaned |
| seafile_seafile-mysql-2023 | 2023 backup MySQL (16 entries) | Old backup |
**3 MySQL/MariaDB instances** + 1 crashed + 1 old 2023 backup = 5 database volumes for one Seafile installation.
### Legacy Seafile (non-Docker)
- `/pool01_n2_redundant/homes/dominik/seafile-server/` — Original 2014 Seafile install (src + seahub only, no data)
### Seafile Container Status
NOT running. No Seafile containers in `docker ps`.
## Mailserver Stack Inventory
### Docker Volume: `mailserver`
| Subdir | Content | Notes |
|--------|---------|-------|
| mail/ | Postfächer (UID 1024, since 2018) | Actual mail data |
| mysql/ | Mailserver MySQL DB | |
| mysql_bak/ | MySQL backup | |
| rainloop/ | Rainloop webmail | UID 991 |
| redis/ | Redis cache | |
| traefik/ | Traefik config | |
### ZFS Maildirs
| Path | Size | Owner |
|------|------|-------|
| `/pool01_n2_redundant/homes/dominik/.Maildir` | 540 MB | Dominik |
| `/pool01_n2_redundant/homes/sarah/.Maildir` | 2 GB | Sarah |
### Mailserver Container Status
NOT running. No mailserver container in `docker ps`.
## ZFS Homes Overview
| Dataset | USED | REFER |
|---------|------|-------|
| pool01_n2_redundant/homes | 456 GB | 455 GB |
Homes contain: dominik (72 entries), sarah (31 entries). Includes .Maildir, seafile-server legacy, and many personal directories.
## Running Containers (after recovery)
| Name | Image | Status |
|------|-------|--------|
| Traefik | traefik:latest | Up |
| website-imkerei-bienen-proxy-1 | nginx:alpine | Up |
| website-imkerei-proxy-1 | nginx:alpine | Up |
| website-imkerei-website-1 | ghost:5.3 | Up |
| portainer | portainer/portainer-ee:lts | Up |
spoolman-spoolman-1 was stuck in "Removal In Progress" — cleared after daemon restart.
## Top Images by Size (before crash)
| Size | Image |
|------|-------|
| 4.08 GB | exadel/compreface-core:1.2.0 |
| 2.2 GB | koush/scrypted:latest |
| 1.98 GB | home-assistant:stable |
| 1.67 GB | gotenberg/gotenberg:8.9 |
| 1.55 GB | thecodingmachine/gotenberg:latest |
| 1.54 GB | homegear/homegear:stable |
| 1.38 GB | seafile-mc:11.0-latest |
| 1.37 GB | seafile-mc:latest |
| 1.37 GB | paperless-ngx:2.17.1 |
| 1.36 GB | paperless-ngx:2.15.1 |
| 1.29 GB | immich-machine-learning:openvino |
| 1.26 GB | iobroker:latest |
| 1.09 GB | spoolman:latest |
| 1.07 GB | double-take:latest |
| 1.04 GB | jellyfin:latest |
## Duplicate Images Identified
| Image | Versions | Est. Waste |
|-------|----------|------------|
| paperless-ngx | 2.9.0, 2.12, 2.12.0, 2.15.1, 2.17.1 + 2× ghcr | ~6 GB |
| seafile-mc | 11.0-latest, latest | ~2.7 GB |
| gotenberg | 8.9, latest | ~3.2 GB |
| portainer-ee | lts, 2.21.5, `<none>` | ~0.7 GB |
| memcached | 1.6, latest, `<none>` | ~0.25 GB |
| postgres | 13, 16 | ~0.85 GB |
| Dangling | 3× `<none>` | ~0.3 GB |
## Recommendation
Docker ZFS storage driver is irreparably corrupt. Options:
1. **Rebuild (recommended)**: Stop docker, rename `/var/lib/docker`, restart with `overlay2` on a separate ext4/xfs volume, redeploy containers. Volumes on ZFS survive.
2. **Manual ZFS cleanup**: Find and `zfs destroy` corrupt subdatasets. Risky, time-consuming, may not fix all issues.
## All Unique Repos (60)
alpine, apache/tika, bitnami/mongodb, bitnami/redis, buanet/iobroker, containrrr/watchtower, dschoen/jenkins-jekyll-docker, eclipse-mosquitto, emby/embyserver, exadel/compreface-{admin,api,core,fe}, flaresolverr/flaresolverr, gcr.io/cadvisor/cadvisor, ghcr.io/donkie/spoolman, ghcr.io/home-assistant/home-assistant, ghcr.io/immich-app/immich-machine-learning, ghcr.io/paperless-ngx/paperless-ngx, ghost, giansalex/monero-miner, gotenberg/gotenberg, grafana/grafana, grafana/loki, grafana/promtail, homegear/homegear, hon95/prometheus-nut-exporter, influxdb, jakowenko/double-take, jellyfin/jellyfin, jlesage/handbrake, koush/scrypted, lanrat/hass-screenshot, lscr.io/linuxserver/bookstack, lscr.io/linuxserver/mariadb, mailserver2/mailserver, mailserver2/postfixadmin, mailserver2/rainloop, mariadb, matomo, memcached, mitmproxy/mitmproxy, mysql, nginx, openhab/openhab, oznu/homebridge, paperlessngx/paperless-ngx, portainer/portainer-ee, postgres, prom/alertmanager, prom/node-exporter, prom/prometheus, redis, registry, schoendominik/sma-emeter, seafileltd/seafile-mc, telegraf, thecodingmachine/gotenberg, traefik, vxcontrol/rancher-letsencrypt, willfarrell/autoheal
devops/docker-host-administration/references/10.0.30.100-seafile-mailserver-inventory-2026-07.md
+95
View File
@@ -0,0 +1,95 @@
# Seafile & Mailserver Volume Inventory — 10.0.30.100 (2026-07-04)
## Seafile Stack
11 Docker volumes related to Seafile. Only `seafile_custom_data` is significant.
| Volume | Size | Content |
|--------|------|---------|
| seafile_custom_data | 226 GB | Seafile file storage |
| └ storage/blocks | 221 GB | Actual file blocks |
| └ storage/fs | 5.3 GB | Filesystem metadata |
| └ storage/commits | 396 MB | Commit history |
| └ thumbnail | 4.5 MB | Thumbnails |
| seafile_custom_mariadb | 272 MB | MariaDB (active?) |
| seafile-mysql | 226 MB | MySQL (old) |
| seafile_mysql | 260 MB | MySQL (old) |
| seafile-mysql.crashed | 375 MB | Crashed MySQL (never cleaned up) |
| seafile_seafile-mysql-2023 | 235 MB | 2023 backup MySQL |
| seafile-conf | < 1 MB | Config |
| seafile_custom_conf | < 1 MB | Custom config |
| seafile_custom_proxy_conf | < 1 MB | Proxy config |
| seafile_data | UNKNOWN (du timeout) | Logs, nginx, seafile core |
**Cleanup potential:** 3 old MySQL + 1 crashed + 1 2023 backup = ~1.1 GB reclaimable.
**Legacy non-Docker Seafile:** `/pool01_n2_redundant/homes/dominik/seafile-server/` — source from 2014, only `src/` and `seahub/` dirs, no data.
## Mailserver Stack
Single Docker volume `mailserver` — unusual structure (no `_data/` directory, data in volume root).
### Component Sizes
| Subdir | Size | Content |
|--------|------|---------|
| mail/clamav | 251 MB | ClamAV virus definitions |
| mail/rspamd | 20 MB | Spam filter data |
| mail/postfix | 477 KB | Postfix config |
| mail/dovecot | 50 KB | Dovecot config |
| mail/dkim | 80 KB | DKIM keys |
| mail/sieve | 14 KB | Sieve rules |
| mail/ssl | 28 KB | SSL certs |
| mail/opendkim | 512 B | OpenDKIM |
| mail/vhosts | 13.5 GB | Mailboxes (see below) |
| redis | 569 MB | Redis cache |
| rainloop | 6.5 MB | Rainloop webmail |
| mysql | 2.9 MB | Mailserver MySQL |
| mysql_bak | 3.0 MB | MySQL backup |
| traefik | 1 KB | Traefik config |
**Total mailserver:** ~14.3 GB
### vhosts Breakdown (measured via find -printf after rsync completed)
| Domain | Mailbox | Files | Size |
|--------|---------|-------|------|
| familie-schoen.com | dominik | 65,493 | 3,975 MB |
| familie-schoen.com | sarah | 9,685 | 3,791 MB |
| famille-schoen.com | sarah-coc | 9,976 | 4,585 MB |
| familie-schoen.com | newsletter | 14,200 | 421 MB |
| familie-schoen.com | dokumente | 173 | 716 MB |
| familie-schoen.com | honig | 159 | 7 MB |
| dominikschoen.de | — | 0 | empty |
| famschoen.eu | — | 0 | empty |
| schoen.sytes.net | — | 0 | empty |
| grafiniert.de | — | 0 | empty |
**Total vhosts:** ~13.5 GB (87,043 files)
Largest individual emails: 92 MB, 92 MB, 61 MB, 48 MB — typical large attachments.
## ZFS Maildirs (not in Docker)
| Path | Size |
|------|------|
| /pool01_n2_redundant/homes/dominik/.Maildir | 540 MB |
| /pool01_n2_redundant/homes/sarah/.Maildir | 2.0 GB |
## Measurement Techniques Used
1. **Per-volume `timeout 5 du -sb`** — works for volumes < 500 MB
2. **`find -type f -printf "%s\n" | awk`** — fallback for large volumes where `du` hangs (vhosts)
3. **Per-subdirectory `du -sh`** — for known directory structures (storage/blocks, storage/fs)
4. **ZFS-native `zfs list -o refer`** — for overall Docker dataset size (not per-volume)
All measurements taken post-rsync (ZFS pool I/O was clear by then).
## Migration to Ceph EC (2026-07-04)
vhosts migrated via `rsync -a --partial -e "sshpass -p PASS ssh"` from 10.0.30.100 to CT 137 (10.0.30.40, imap), `/var/mail/vhosts/` on Ceph EC 4+1 RBD. Dovecot configured with `mail_location = maildir:/var/mail/vhosts/%d/%n/mail`.
## Container Status
Neither Seafile nor Mailserver containers were running at time of inventory.
Only 5 containers active on 10.0.30.100: Traefik, 2× nginx (imkerei proxies), Ghost (imkerei blog), Portainer.
devops/docker-host-administration/references/docker-service-reconstruction-2026-07.md
+168
View File
@@ -0,0 +1,168 @@
# Docker Service Reconstruction After Config Loss — 2026-07-05
## Scenario
CT 111 (Proxmox LXC container, 10.0.30.99) had its root filesystem partially cleared — all Docker compose files, `.env` files, and container state were deleted. However, **data on bind-mount volumes survived** because they were on separate RBD-backed mount points. This reference covers rebuilding Docker services from surviving data without losing anything.
## Key Principles
1. **NEVER delete or overwrite existing data directories** — only create new compose/config files
2. **Inspect surviving data before writing anything** — read existing config files (seahub_settings.py, postgresql.conf, etc.) to extract credentials and settings
3. **Bind mounts survive root FS clearing** — data on separate partitions/RBDs is safe
4. **Docker images often survive** — check `docker images` before pulling
5. **Postgres data is self-contained** — `POSTGRES_PASSWORD` env var is only used for initialization; existing DBs keep their own auth. Reset password with `ALTER USER` after starting.
## File Transfer to Proxmox CTs
### Problem: `scp` to PVE nodes can hang or get blocked by security scanners
### Solution: base64-over-SSH (reliable, no scp needed)
```bash
# Write file locally, then transfer via base64 encoding
B64=$(base64 -w0 /tmp/compose-file.yml)
ssh root@<pve-node> "pct exec <CT_ID> -- bash -c 'echo $B64 | base64 -d > /path/to/file'"
```
### Solution: `pct push` (writes file into CT from PVE host)
```bash
# Copy file to PVE host first (via base64 if scp fails)
B64=$(base64 -w0 /tmp/file)
ssh root@<pve-node> "echo $B64 | base64 -d > /tmp/file"
# Then push into CT
ssh root@<pve-node> "pct push <CT_ID> /tmp/file /path/inside/ct/file"
```
### Pitfall: Shell variable expansion eats `${VAR}` in heredocs
When writing Docker compose files via `bash -c 'cat > file << EOF'`, shell expands `${VARIABLE}` references. The resulting file has empty values where compose variables should be.
**Fix**: Write files locally with `write_file`, then transfer via base64 or `pct push`. Never use heredocs for compose files with `${VAR}` interpolation.
## Multi-line RSA Keys in Docker .env Files
### Problem
Docker Compose `.env` files parse each line as `KEY=value`. Multi-line RSA private keys (PEM format with `-----BEGIN PRIVATE KEY-----` headers) break the parser because:
- Newlines split the value across multiple lines
- `+` characters in base64 trigger "unexpected character in variable name" errors
- Quoting doesn't help — the parser still chokes on `+`
### Solution: Escape newlines as literal `\n` (two chars: backslash + n)
```python
import pathlib
pem = pathlib.Path('/tmp/jwt_private_key.pem').read_text()
single_line = pem.strip().replace('\n', '\\n')
# Write to .env with the escaped key
env_content = f"""JWT_PRIVATE_KEY={single_line}
OTHER_VAR=value
"""
pathlib.Path('/opt/app/.env').write_text(env_content)
```
Docker Compose will interpolate `\n` back to actual newlines when passing the env var to the container. The container receives a proper multi-line PEM key.
### Pitfall: `sed` fails on keys containing `/`
Using `sed "s|^JWT_PRIVATE_KEY=.*|$KEY|"` fails because RSA keys contain `/` characters which conflict with sed's delimiter. Use Python for safe string manipulation.
## Postgres Recovery with Existing Data
### Starting Postgres with existing data directory
When the Postgres data directory survives (on a bind mount), the `POSTGRES_PASSWORD` env var is **ignored** — it's only used for initial database creation on empty directories. Postgres will start with whatever passwords are already in the cluster.
```bash
# Remove stale postmaster.pid (leftover from unclean shutdown)
rm -f /mnt/data/postgres/postmaster.pid
# Start postgres container — it will detect existing data and skip init
docker compose up -d database
# Logs show: "PostgreSQL Database directory appears to contain a database; Skipping initialization"
# Then: "database system was not properly shut down; automatic recovery in progress"
# Then: "database system is ready to accept connections"
```
### Resetting the postgres password after start
Since we don't know the original password, reset it to match the `.env`:
```bash
# Use psql inside the container — quoting hell through nested SSH
# This pattern works:
ssh root@<node> 'pct exec <CT> -- bash -c "docker exec <pg_container> psql -U postgres -c \"ALTER USER postgres WITH PASSWORD '"'"'NewPassword!'"'"';\""'
# Output: ALTER ROLE
```
## Immich Recovery Pattern
### Surviving Data
- `/mnt/data/immich-library/` — photo library (261 GB, years 2004-2026)
- `/mnt/data/immich-postgres/` — Postgres 14 data directory
- `immich_model-cache` Docker volume — ML model cache
### Steps
1. Download official Immich docker-compose.yml from GitHub releases
2. Create `.env` with: `UPLOAD_LOCATION=/mnt/data/immich-library`, `DB_DATA_LOCATION=/mnt/data/immich-postgres`, `DB_PASSWORD=<new_password>`
3. Set `model-cache` volume as `external: true` (existing volume)
4. Start database first: `docker compose up -d database`
5. Remove stale `postmaster.pid` if present
6. Reset postgres password: `ALTER USER postgres WITH PASSWORD '<new_password>'`
7. Start full stack: `docker compose up -d`
8. Verify: `curl -s -o /dev/null -w "%{http_code}" http://localhost:2283` → 200
### Immich Compose Template (v2.6.3, OpenVINO ML)
Key compose elements for recovery:
- Server image: `ghcr.io/immich-app/immich-server:release`
- ML image: `ghcr.io/immich-app/immich-machine-learning:release-openvino` (Intel Arc/OpenVINO)
- Postgres image: `ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0` (pinned by digest)
- Redis: `docker.io/valkey/valkey:9` (pinned by digest)
- Model cache volume: `external: true, name: immich_model-cache`
## Seafile Recovery Pattern
### Surviving Data
- `/opt/seafile-data/` — file storage (blocks, commits, fs directories) — 249 GB
- `/opt/seafile-data/seafile/conf/` — seahub_settings.py, seafile.conf, gunicorn.conf.py
- Docker images: `seafileltd/seafile-mc:12.0-latest`, `mariadb:10.11`, `memcached:1.6.29`
### Lost Data
- `/opt/seafile-mysql/db` — MariaDB database (was on root FS)
- `/opt/seadoc-data/` — SeaDoc data
- All compose files and `.env`
### Steps
1. Download official Seafile CE 12.0 compose templates (seafile-server.yml, caddy.yml, seadoc.yml)
2. Extract credentials from surviving config: `cat /opt/seafile-data/seafile/conf/seafile.conf` → DB password
3. Extract domain from `seahub_settings.py` → `SERVICE_URL = "https://seafile.familie-schoen.com"`
4. Generate new JWT_PRIVATE_KEY (RSA 2048): `openssl genpkey -algorithm RSA -out jwt.pem -pkeyopt rsa_keygen_bits:2048`
5. Convert to single-line `\n`-escaped format for .env (see above)
6. Create missing directories: `mkdir -p /opt/seafile-mysql/db /opt/seafile-caddy /opt/seadoc-data`
7. Start services: `docker compose up -d`
8. MariaDB initializes fresh — Seafile will re-init DB but file storage is preserved
### Post-Recovery Issue
The MariaDB database is freshly initialized. File storage (blocks/commits/fs) survives but the DB doesn't know about the libraries. SeaDoc data is lost. Libraries need to be reconstructed using `seaf-fsck` or manually re-created and linked to existing storage.
## Root FS Space Management
When pulling many Docker images on a CT with limited root FS:
```bash
# Check space BEFORE pulling
df -h /
# If root FS is >90% full, prune before pulling
docker image prune -f
docker builder prune -f
# Monitor during pulls
watch -n 10 'df -h / | tail -1'
```
CT 111 had 49 GB root FS at 93% after pulling all images for Immich + Seafile. Consider `docker image prune` after services are running.