Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,498 @@
+# Seafile Docker Recovery: MySQL Volume Loss & Selective PBS Restore
+
+## Scenario
+
+Seafile MC Docker edition on CT 111 (n5pro, 10.0.20.91). The `.env` was changed
+(image downgraded from `seafile-mc:13.0-latest` to `seafile-mc:12.0-latest`,
+hostname changed from `cloud.familie-schoen.com` to `seafile.familie-schoen.com`),
+which triggered MySQL volume reinitialization — wiping all Seafile databases
+(`seafile_db`, `ccnet_db`, `seahub_db`) and the `seafile` MySQL user. The file
+block storage (`/opt/seafile-data/`) remained intact. A PBS backup of CT 111 from
+2026-07-02 existed on `noris_s3` storage. The original `/opt/seafile/` config
+directory was preserved as a ZIP by the user.
+
+## Symptoms
+
+### Symptom 1: Container "Up" but not actually serving
+
+```bash
+docker ps                    # → seafile Up 7 hours
+docker logs seafile --tail 50 # → "waiting for mysql server to be ready: mysql is not ready" (repeating forever)
+```
+
+The container's entrypoint (`/scripts/start.py`) calls `wait_for_mysql()` before
+proceeding to `init_seafile_server()`. If MySQL connection fails, it loops forever.
+The container stays "Up" (PID 1 alive) but Seafile never starts.
+
+### Symptom 2: seaf-fsck warns about DB connection
+
+```bash
+docker exec seafile bash -lc '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'
+# → [WARNING] Failed to connect to MySQL: Access denied for user 'seafile'@'172.18.0.5'
+# → seaf-fsck run done
+# → Done.
+```
+
+Fsck exits 0 but didn't actually check anything — the DB connection failed.
+
+### Symptom 3: MySQL has no Seafile databases
+
+```bash
+docker exec seafile-mysql mysql -uroot -p'PASSWORD' -e 'SHOW DATABASES;'
+# → information_schema, mysql, performance_schema, sys  (NO seafile_db, ccnet_db, seahub_db)
+```
+
+### Symptom 4: No seafile MySQL user
+
+```bash
+docker exec seafile-mysql mysql -uroot -p'PASSWORD' \
+  -e "SELECT user,host FROM mysql.user WHERE user LIKE '%sea%';"
+# → Empty set
+```
+
+## Root Cause Analysis
+
+### How `wait_for_mysql()` works
+
+File: `/scripts/utils.py` inside the seafile container:
+
+```python
+def wait_for_mysql():
+    db_host = get_conf('DB_HOST', '127.0.0.1')
+    db_user = 'root'
+    db_passwd = get_conf('DB_ROOT_PASSWD', '')
+
+    # BUT: if seafile.conf exists, it overrides with the seafile user!
+    seafile_conf_path = join(topdir, 'conf', 'seafile.conf')
+    if exists(seafile_conf_path):
+        cp = ConfigParser()
+        cp.read(seafile_conf_path)
+        db_host = cp.get('database', 'host')    # → 'db'
+        db_user = cp.get('database', 'user')    # → 'seafile'
+        db_passwd = cp.get('database', 'password')
+        db_port = int(cp.get('database', 'port'))
+
+    while True:
+        try:
+            connection = pymysql.connect(host=db_host, port=db_port,
+                                         user=db_user, passwd=db_passwd)
+        except Exception as e:
+            # prints "mysql is not ready" and sleeps
+```
+
+**Critical insight**: Once `seafile.conf` exists (from a prior successful install),
+`wait_for_mysql()` uses the `seafile` user — NOT root. If the MySQL volume was
+wiped, the `seafile` user doesn't exist, and this loop runs forever.
+
+### Why `init_seafile_server()` doesn't fix it
+
+Even after manually recreating the `seafile` user and databases, restarting the
+container produces:
+
+```
+Skip running setup-seafile-mysql.py because there is existing seafile-data folder.
+```
+
+The bootstrap script (`/scripts/bootstrap.py`) skips DB initialization when
+`seafile-data/` already exists. So the MySQL tables are never created, and fsck
+reports `Table 'seafile_db.Repo' doesn't exist`.
+
+## Recovery Procedure
+
+### Option A: Full PBS Restore of MySQL Volume (Preferred)
+
+When a PBS backup exists, restoring the entire MySQL data directory is cleanest
+because it includes all users, grants, schemas, and data.
+
+⚠️ **Before attempting PBS restore, verify the backup is restorable** (learned
+the hard way 2026-07-06):
+
+1. Check verification state in `pvesh get` output — if `{"state":"failed"}`,
+   the backup has missing/corrupt chunks and may not restore.
+2. Check snapshot directory on PBS server for `.didx` vs `.tmp_didx` —
+   only `.didx` snapshots are complete and restorable.
+3. Check for `.bad` chunk files on PBS server — if any chunk referenced by
+   the backup is `.bad` (0 bytes), restore aborts immediately with no option
+   to skip. Files dependent on that chunk are permanently lost.
+
+See `proxmox-ve-administration` skill → `references/pbs-lxc-setup-2026-07.md`
+pitfalls #13–16 for PBS credential lookup, verification checking, corrupted
+chunk diagnosis, and incomplete backup detection.
+
+If PBS restore fails due to corrupted chunks, fall back to Option B (manual
+recreation) and use `seaf-fsck.sh --repair` to rebuild repo metadata from
+the intact `/opt/seafile-data/` file blocks.
+
+#### Step 1: Locate the PBS backup
+
+```bash
+# On the PVE node hosting the CT (n5pro = 10.0.20.91):
+pvesh get /nodes/$(hostname)/storage/noris_s3/content --content-type backup 2>&1 \
+  | awk -F'│' '{print $3}' \
+  | grep 'ct/111'
+# → noris_s3:backup/ct/111/2026-07-02T21:00:00Z
+```
+
+**Note**: `pvesh get` outputs a Unicode box-drawing table, not JSON. Parse with
+`awk -F'│'` (Unicode pipe character U+2502). Column 3 contains the volid.
+
+#### Step 2: Stop Seafile containers
+
+```bash
+# On the PVE node:
+pct exec 111 -- docker stop seafile seafile-mysql seafile-memcached
+```
+
+#### Step 3: Rename broken MySQL volume
+
+```bash
+pct exec 111 -- mv /opt/seafile-mysql/db /opt/seafile-mysql/db.broken-$(date +%Y%m%d)
+```
+
+#### Step 4: Restore from PBS
+
+Use `pct restore` with `--storage` pointing to the CT's root storage, then
+selectively extract only `/opt/seafile-mysql/db/` from the restored backup.
+
+Alternatively, use `proxmox-backup-client` to restore specific files:
+
+```bash
+# Map the PBS backup to a temp location
+# PBS restore maps the CT filesystem
+pct exec 111 -- bash -c '
+  mkdir -p /tmp/restore
+  # Use pb-client to restore just the mysql db path
+  # (requires PBS credentials)
+'
+```
+
+#### Step 5: Verify MySQL data
+
+```bash
+pct exec 111 -- docker start seafile-mysql
+sleep 10
+pct exec 111 -- docker exec seafile-mysql mysql -uroot -p'PASSWORD' \
+  -e "SELECT user,host FROM mysql.user WHERE user='seafile'; SHOW DATABASES;"
+# Should show: seafile@%, seafile_db, ccnet_db, seahub_db
+```
+
+#### Step 6: Start Seafile
+
+```bash
+pct exec 111 -- docker start seafile
+sleep 30
+pct exec 111 -- docker logs seafile --tail 20
+# Should show: "Seafile server started", "Seahub is started"
+```
+
+#### Step 7: Run fsck
+
+```bash
+pct exec 111 -- docker exec seafile bash -lc \
+  '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'
+```
+
+With restored DB, fsck should connect and check all repos.
+
+### Option B: Manual DB Init via `docker run` Setup Script (Preferred over SQL import)
+
+When PBS restore fails and you need fresh DBs, the cleanest approach is running
+the official `setup-seafile-mysql.sh` in a throwaway container with proper env
+vars. This creates all schemas, users, and config files correctly.
+
+**Critical**: The `bootstrap.py` in the main container skips `init_seafile_server()`
+when `/shared/seafile/seafile-data/` exists. To force fresh DB init:
+
+1. Stop the seafile container
+2. Rename `seafile-data` temporarily: `mv .../seafile-data .../seafile-data.bak`
+3. Clear `conf/`: `rm -rf .../conf/*`
+4. Drop any half-created databases: `DROP DATABASE ccnet_db, seafile_db, seahub_db`
+5. Run setup in a throwaway container:
+
+```bash
+docker run --rm --name seafile-setup --network seafile-net \
+  -e SERVER_NAME=seafile \
+  -e SERVER_IP=cloud.familie-schoen.com \
+  -e MYSQL_USER=seafile \
+  -e MYSQL_USER_PASSWD='SEAFILE_PW' \
+  -e MYSQL_USER_HOST=% \
+  -e MYSQL_HOST=db \
+  -e MYSQL_PORT=3306 \
+  -e MYSQL_ROOT_PASSWD='ROOT_PW' \
+  -e CCNET_DB=ccnet_db \
+  -e SEAFILE_DB=seafile_db \
+  -e SEAHUB_DB=seahub_db \
+  -e SEAFILE_SERVER_HOSTNAME=cloud.familie-schoen.com \
+  -e SEAFILE_SERVER_PROTOCOL=https \
+  -e INIT_SEAFILE_ADMIN_EMAIL=admin@example.com \
+  -e INIT_SEAFILE_ADMIN_PASSWORD=password \
+  -e TIME_ZONE=Europe/Berlin \
+  -v /opt/seafile-data:/shared \
+  seafileltd/seafile-mc:13.0-latest \
+  /opt/seafile/seafile-server-13.0.19/setup-seafile-mysql.sh auto -n seafile
+```
+
+⚠️ The env var names matter: `MYSQL_ROOT_PASSWD` (not `MYSQL_ROOT_PASSWORD`),
+`MYSQL_USER_PASSWD` (not `MYSQL_USER_PASSWORD`). The setup script reads
+`get_param_val(args.mysql_root_passwd, 'MYSQL_ROOT_PASSWD')` from `os.environ`.
+
+6. Restore original `seafile-data` and `conf`:
+```bash
+rm -rf .../seafile-data && mv .../seafile-data.bak .../seafile-data
+# Restore conf from backup (contains correct DB credentials, SERVICE_URL, etc.)
+cp -a .../conf.bak/* .../conf/
+```
+
+7. Update `SERVICE_URL` in `seahub_settings.py` if hostname changed:
+```bash
+sed -i 's|old.domain.com|new.domain.com|g' .../conf/seahub_settings.py
+```
+
+8. Start the container normally — it will run upgrade scripts (12.0→13.0 etc.)
+   and start Seafile.
+
+### Option B2: Manual SQL Schema Import (Legacy Approach)
+
+If `docker run` setup doesn't work, manually recreate databases and import schemas.
+
+#### Step 1: Create databases and user
+
+```bash
+docker exec seafile-mysql mysql -uroot -p'ROOT_PW' -e "
+  CREATE DATABASE IF NOT EXISTS ccnet_db CHARACTER SET utf8 COLLATE utf8_general_ci;
+  CREATE DATABASE IF NOT EXISTS seafile_db CHARACTER SET utf8 COLLATE utf8_general_ci;
+  CREATE DATABASE IF NOT EXISTS seahub_db CHARACTER SET utf8 COLLATE utf8_general_ci;
+  CREATE USER 'seafile'@'%' IDENTIFIED BY 'SEAFILE_PW';
+  GRANT ALL PRIVILEGES ON ccnet_db.* TO 'seafile'@'%';
+  GRANT ALL PRIVILEGES ON seafile_db.* TO 'seafile'@'%';
+  GRANT ALL PRIVILEGES ON seahub_db.* TO 'seafile'@'%';
+  FLUSH PRIVILEGES;
+"
+```
+
+#### Step 2: Import SQL schemas
+
+The seafile container has no `mysql` client. Use the seafile-mysql container:
+
+```bash
+SQLDIR=/opt/seafile/seafile-server-12.0.14/sql/mysql
+
+# ccnet schema
+docker exec seafile-mysql mysql -uroot -p'ROOT_PW' ccnet_db \
+  < <(docker exec seafile cat $SQLDIR/ccnet.sql)
+
+# seafile schema
+docker exec seafile-mysql mysql -uroot -p'ROOT_PW' seafile_db \
+  < <(docker exec seafile cat $SQLDIR/seafile.sql)
+
+# seahub schema (located in seahub/scripts/upgrade/sql/)
+docker exec seafile-mysql mysql -uroot -p'ROOT_PW' seahub_db \
+  < <(docker exec seafile cat /opt/seafile/seafile-server-12.0.14/seahub/scripts/upgrade/sql/12.0.0/mysql/seahub.sql)
+```
+
+⚠️ **This only recreates empty tables.** Existing repo metadata, users, and
+library mappings are lost. File blocks in `/opt/seafile-data/` remain but are
+orphaned (no DB entries pointing to them). The admin account must be recreated
+via `check_init_admin.py` or the Seafile web UI.
+
+#### Step 3: Restart and run fsck
+
+```bash
+docker restart seafile
+sleep 30
+docker exec seafile bash -lc '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'
+```
+
+### Option C: Reconstruct Repo Registrations from Commit Storage
+
+When DBs are freshly initialized but `/opt/seafile-data/` (file blocks + commits)
+is intact, repos can be reconstructed by scanning the commit storage and
+registering them in the DB. See `references/seafile-repo-reconstruction.md` for
+the full procedure with Python scripts.
+
+**Overview of the technique:**
+
+1. **Scan commit storage**: `storage/commits/<repo_id>/<XX>/<YYYY...>/` — each
+   file is a JSON object with `commit_id`, `root_id`, `parent_id`,
+   `second_parent_id`, `repo_name`, `repo_desc`, `ctime`, `creator_name`.
+2. **Find HEAD commit**: The commit NOT referenced as `parent_id` by any other
+   commit in the same repo. For repos with many commits (>1000), use
+   `find -printf '%T@ %p\n' | sort -rn | head -20` to read only the newest 20.
+3. **Register in DB**: Insert into `Repo`, `Branch`, `RepoOwner`, and `RepoInfo`
+   tables. The `Repo` table only needs `repo_id`; `Branch` needs `(name, repo_id,
+   commit_id)`; `RepoOwner` needs `(repo_id, owner_id)`; `RepoInfo` needs
+   `(repo_id, name, ...)`.
+4. **Run `seaf-fsck.sh --repair`**: After registering repos, fsck verifies
+   integrity and repairs inconsistencies.
+
+⚠️ **Encrypted repo commit IDs may have suffixes** (e.g.,
+`e155972728d6a3205726b94243078ae5f0aaaa3a.FYMWR3`) that exceed the 40-char
+`commit_id` column. Truncate to 40 chars before inserting.
+
+⚠️ **`seaf-fsck.sh --repair` only checks registered repos** — it does NOT discover
+or register new repos. All repos must be registered in the DB first.
+
+⚠️ **`seaf-fsck.sh --export` also only exports registered repos** — it cannot
+recover unregistered repos from the filesystem.
+
+## Key Files in Seafile Docker Container
+
+| Path | Purpose |
+|------|---------|
+| `/opt/seafile/conf/seafile.conf` | Main config — DB host, user, password, fileserver port |
+| `/opt/seafile/seafile-server-VERSION/` | Server installation (scripts, SQL, binaries) |
+| `/opt/seafile/seafile-server-VERSION/seaf-fsck.sh` | Filesystem consistency checker |
+| `/opt/seafile/seafile-server-VERSION/seaf-gc.sh` | Garbage collector for orphaned blocks |
+| `/opt/seafile/seafile-server-VERSION/sql/mysql/` | SQL schema files (ccnet.sql, seafile.sql) |
+| `/opt/seafile/seafile-server-VERSION/seahub/scripts/upgrade/sql/VERSION/mysql/seahub.sql` | Seahub DB schema |
+| `/scripts/start.py` | Container entrypoint — calls `wait_for_mysql()` + `init_seafile_server()` |
+| `/scripts/utils.py` | `wait_for_mysql()` implementation |
+| `/scripts/bootstrap.py` | `init_seafile_server()` — skips if `seafile-data/` exists |
+
+## Docker-Compose Layout (seafile-server.yml)
+
+```yaml
+services:
+  db:
+    image: mariadb:10.11
+    container_name: seafile-mysql
+    volumes:
+      - "${SEAFILE_MYSQL_VOLUME:-/opt/seafile-mysql/db}:/var/lib/mysql"
+    # ...
+  seafile:
+    image: seafileltd/seafile-mc:12.0-latest
+    container_name: seafile
+    environment:
+      - DB_HOST=db
+      - DB_USER=seafile
+      - DB_ROOT_PASSWD=${INIT_SEAFILE_MYSQL_ROOT_PASSWORD}
+      - DB_PASSWORD=${SEAFILE_MYSQL_DB_PASSWORD}
+    volumes:
+      - "${SEAFILE_VOLUME:-/opt/seafile-data}:/shared"
+    depends_on:
+      db:
+        condition: service_healthy
+```
+
+The `seafile` container's `/shared` mount corresponds to `/opt/seafile-data` on the
+host. Inside the container, `/opt/seafile/` is symlinked from `/shared/seafile/`.
+
+## Network Architecture
+
+```
+seafile-net (bridge, 172.18.0.0/16):
+  ├── seafile-mysql    172.18.0.3  (hostname: db)
+  ├── seafile-memcached 172.18.0.2
+  ├── seafile-caddy    172.18.0.4
+  ├── seafile          172.18.0.5
+  └── seadoc           172.18.0.6
+```
+
+`seafile.conf` references `host = db` which resolves via Docker DNS to the
+`seafile-mysql` container.
+
+## Pitfalls
+
+1. **`wait_for_mysql()` uses `seafile` user, not `root`** — Once `seafile.conf`
+   exists, the startup script connects as the `seafile` MySQL user. If that user
+   doesn't exist (volume wipe), the loop runs forever. Container shows "Up" but
+   Seafile never starts.
+
+2. **`init_seafile_server()` skips when `seafile-data/` exists** — Even after
+   recreating the MySQL user and empty databases, the bootstrap won't create
+   tables because it sees existing data. Must import SQL schemas manually or
+   restore the MySQL volume from backup.
+
+3. **`seaf-fsck.sh` exits 0 even on DB failure** — It catches the DB connection
+   error, prints a WARNING, and exits cleanly. Don't rely on exit code; check
+   stderr for `[WARNING]` lines.
+
+4. **No `mysql` client in seafile container** — Use `docker exec seafile-mysql
+   mysql ...` for all MySQL operations. The seafile container only has Python
+   and Seafile binaries.
+
+5. **`seafile-data/` is large and usually intact** — When recovering, don't
+   restore `/opt/seafile-data/` from backup unless corruption is suspected. It
+   contains all file blocks (potentially hundreds of GB). Restoring only the
+   MySQL volume + conf is much faster.
+
+6. **PBS backup listing via `pvesh`** — `pvesh get /nodes/$(hostname)/storage/
+   <storage>/content` outputs a Unicode table, not JSON. Parse with
+   `awk -F'│'` to extract volids. See `proxmox-ve-administration` skill for
+   PBS backup listing details.
+
+7. **MariaDB vs MySQL image** — Seafile Docker uses `mariadb:10.11` but the
+   config says `type = mysql`. MariaDB is compatible. The `MARIADB_AUTO_UPGRADE=1`
+   env var handles schema upgrades on container start.
+
+8. **Image version change can wipe MySQL volume** — Changing the Seafile Docker
+   image (e.g., downgrading `seafile-mc:13.0-latest` → `seafile-mc:12.0-latest`
+   or vice versa) can trigger MySQL volume reinitialization. The container detects
+   a version mismatch and reinitializes the data directory, wiping all databases
+   and users. **Always backup the MySQL volume (`/opt/seafile-mysql/db`) before
+   changing the image version in `.env`.**
+
+9. **Compare `.env` before restoring** — When recovering, always diff the current
+   `.env` against any backup config. Key fields that differ between versions:
+   - `SEAFILE_IMAGE` (e.g., `13.0-latest` vs `12.0-latest`)
+   - `SEAFILE_SERVER_HOSTNAME` (e.g., `cloud.familie-schoen.com` vs
+     `seafile.familie-schoen.com`)
+   - `INIT_SEAFILE_ADMIN_EMAIL`
+   - `JWT_PRIVATE_KEY`
+   - `COMPOSE_FILE` list (e.g., `caddy.yml` may be added/removed)
+
+   Align the config BEFORE starting containers with a restored DB, otherwise
+   Seafile may reinitialize or fail to match the restored data.
+
+10. **Telegram file transfer corrupts large ZIPs** — Sending large binary archives
+    (e.g., a MySQL data directory ZIP) via Telegram can truncate or corrupt the
+    file. The ZIP arrives with valid local file headers but no central directory
+    and invalid size fields (`0xFFFFFFFF`). For transferring server data between
+    PVE nodes, use `scp`, `rsync`, or PBS restore instead of Telegram file
+    attachments. If a received ZIP fails to open, verify with
+    `python3 -c "import zipfile; zipfile.ZipFile('file.zip').namelist()"` — if it
+    raises `BadZipFile`, the transfer corrupted it.
+
+11. **`bootstrap.py` ENV variable substitution bug** — When `bootstrap.py`
+    calls `setup-seafile-mysql.py` via `subprocess.call(cmd, env={...})`,
+    the `env=` dict **replaces** the entire process environment rather than
+    augmenting it. The `env=` dict includes `MYSQL_ROOT_PASSWD` (mapped from
+    `INIT_SEAFILE_MYSQL_ROOT_PASSWORD`), but if the mapping is missing or
+    misnamed, the subprocess gets `MYSQL_ROOT_PASSWD=''` (empty string),
+    producing the error `using password: NO` from MySQL.
+
+    **Symptom**: `setup-seafile-mysql.py` fails with `Access denied for
+    'root'@'localhost' (using password: NO)` even though
+    `INIT_SEAFILE_MYSQL_ROOT_PASSWORD` is correctly set in the container env.
+
+    **Fix**: Run `setup-seafile-mysql.py` manually inside the container with
+    all required ENV vars explicitly exported:
+    ```bash
+    docker exec -e MYSQL_ROOT_PASSWD='<root_pw>' \
+      -e MYSQL_USER='seafile' -e MYSQL_USER_PASSWD='<seafile_pw>' \
+      -e MYSQL_HOST=db -e MYSQL_PORT=3306 \
+      -e CCNET_DB=ccnet_db -e SEAFILE_DB=seafile_db -e SEAHUB_DB=seahub_db \
+      -e SEAFILE_SERVER_HOSTNAME=cloud.example.com \
+      -e INIT_SEAFILE_ADMIN_EMAIL=admin@example.com \
+      -e INIT_SEAFILE_ADMIN_PASSWORD='<admin_pw>' \
+      seafile /opt/seafile/seafile-server-13.0.19/setup-seafile-mysql.sh auto -n seafile
+    ```
+    This bypasses `bootstrap.py`'s broken `env=` substitution and gives the
+    setup script direct access to all required variables.
+
+12. **PBS corrupted chunk blocks ALL file restoration** — A single 0-byte `.bad`
+    chunk file on the PBS server (destroyed by garbage collection) makes the
+    ENTIRE backup unrestorable. PBS aborts on the first file referencing the
+    bad chunk, cleans up, and exits. There is no `--skip-bad-chunks` option.
+    Files alphabetically before the affected file MAY be partially restored,
+    but anything after is lost. The chunk path format is
+    `/mnt/datastore/<ds>/.chunks/<XX>/<hash>.0.bad`.
+
+13. **Multiple backup formats can ALL be unusable** — In this incident, THREE
+    separate backup sources were tried and ALL failed:
+    - PBS backup: corrupted chunk (above)
+    - `seafile-mysql.zip` (Telegram-transferred): BadZipFile, no central directory
+    - `seafile-mysql.tar.zst`: truncated, unexpected EOF after 4 entries
+    - Offsite PBS: no CT111 backup existed at all
+    Lesson: never assume ANY backup is restorable without verifying. Always
+    test-restore before declaring a backup strategy viable.