dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Hermes Agent Skills collection

Browse Source
This commit is contained in:
Debian
2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
devops/seafile-api/SKILL.md
+244
View File
@@ -0,0 +1,244 @@
---
name: seafile-api
description: "Seafile Cloud administration: API interaction (token auth, repos, dirs, file ops) and Docker server administration (fsck, MySQL volume recovery, startup debugging, image-version-change pitfalls). Covers Seafile 12.x–13.x MC Docker edition."
tags: [file-storage, cloud, api, seafile, docker, recovery]
---
# Seafile Cloud Administration
This skill covers two domains:
1. **API Interaction** — token auth, repo/dir/file operations via REST API
2. **Docker Server Administration** — `seaf-fsck`, MySQL volume recovery, startup debugging, selective restore from PBS backups
For Docker server administration, recovery procedures, and fsck troubleshooting, see `references/seafile-docker-recovery.md`.
For backup strategy analysis (offsite/PBS/S3/native), see `references/seafile-immich-backup-strategy.md`.
## API: Authentication
Seafile uses token-based API auth. **Never use cookie/session login** — it doesn't work reliably.
```bash
# Get API token
curl -ks -H 'Content-Type: application/json' \
-d '{"username":"user@example.com","password":"password"}' \
"https://seafile.yourdomain.com/api2/auth-token/"
# → {"token":"abc123..."}
```
## API: List Repositories
```bash
curl -ks -H "Authorization: Token {token}" \
-H "X-Requested-With: XMLHttpRequest" \
"https://seafile.yourdomain.com/api2/repos/"
```
Returns array of repos with type (`"repo"` = personal, `"grepo"` = group-shared).
## API: List Directory Contents
```bash
curl -ks -H "Authorization: Token {token}" \
-H "X-Requested-With: XMLHttpRequest" \
"https://seafile.yourdomain.com/api2/repos/{repo-id}/dir/?p=%2F"
```
Returns array of items: `{type:"dir"/"file", name, size, mtime, id, permission}`.
## API: Download File
**Seafile 13.x (Pro/Plus) — file download via `/api2/repos/.../files/` returns 404.** This endpoint only works on older Seafile versions.
**Workarounds for Seafile 13.x:**
1. Use `requests.Session()` to log in via the web UI (`/accounts/login/`) and navigate to the file page, then extract the download URL from the React SPA's network calls.
2. Use WebDAV: `curl -k -u user:token https://server/dav/path/to/file`
3. Use `seafile-cli` if available.
4. Manually download via browser/CLI and copy to the server.
```bash
# Old endpoint (ONLY works on Seafile < 13.x):
curl -ks -H "Authorization: Token {token}" \
-H "X-Requested-With: XMLHttpRequest" \
"https://seafile.yourdomain.com/api2/repos/{repo-id}/files/?p=%2Fpath%2Fto%2Ffile.txt" \
-o output.txt
```
For login + download via Python `requests`:
```python
import requests, re, urllib.parse
API = "https://seafile.yourdomain.com"
REPO = "repo-id-here"
FILE = "/path/to/file.pdf"
session = requests.Session()
session.headers.update({"User-Agent": "Mozilla/5.0 ..."})
# 1. Get CSRF token
resp = session.get(API, allow_redirects=True, timeout=15)
csrf = re.search(r'name="csrfmiddlewaretoken"[^>]*value="([^"]*)"', resp.text).group(1)
# 2. POST login
session.post(resp.url, data={
"login": "user@example.com", "password": "pw",
"csrfmiddlewaretoken": csrf, "next": "/",
}, allow_redirects=True, timeout=15)
# 3. Access file page — the SPA loads download via JS; inspect page source for download patterns
file_url = f"{API}/library/{REPO}/docs/?p={urllib.parse.quote(FILE, safe='/')}"
resp = session.get(file_url, timeout=15)
# Look for 'download' patterns in resp.text — no static <a> tags exist
```
## API: Create Directory
```bash
curl -ks -X PUT -H "Authorization: Token {token}" \
-H "X-Requested-With: XMLHttpRequest" \
"https://seafile.yourdomain.com/api/v2.1/repos/{repo-id}/directory/" \
--data-urlencode "path=/new_folder"
```
## API: Upload File
**Seafile v2.1 Basic Edition (default) — ALL API upload endpoints return 404.**
The following endpoints are ALL disabled:
- `/api/v2.1/repos/{repo-id}/upload/`
- `/api/v2.0/repos/{repo-id}/upload/`
- `/api/v1/repos/{repo-id}/upload/`
- `/api/repos/{repo-id}/upload/`
- `/api/upload/`
- `/api/v1/upload/`
- `/seafhttp/upload/`
**Fallback: Playwright Web UI Automation**
The React web UI upload button uses selector `[class*="upload-file"]`. It sits in a dropdown, so use JS click + file chooser:
```python
from playwright.sync_api import sync_playwright
import json, subprocess, glob, time
SEAF_URL = "https://cloud.familie-schoen.com"
repo_id = "737eb5b4-731e-4bfe-8a42-8edd62b04c6e" # from /api2/repos/
TARGET_DIR = "/Amazon"
creds = json.loads(subprocess.run(
'op item get "cloud.familie-schoen.com" --vault "Hermes" --format json',
shell=True, capture_output=True, text=True).stdout)
username, password = creds['fields'][0]['value'], creds['fields'][1]['value']
files = sorted(glob.glob("/tmp/amazon-invoices-2025/Amazon_Rechnung_*.pdf"))
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
context = browser.new_context(viewport={"width": 1280, "height": 800}, ignore_https_errors=True)
page = context.new_page()
# Login
page.goto(f"{SEAF_URL}/accounts/login/", wait_until="domcontentloaded", timeout=30000)
page.fill('input[name="login"]', username)
page.fill('input[name="password"]', password)
page.click('button[type="submit"]')
page.wait_for_url(lambda u: "login" not in u.lower(), timeout=10000)
# Navigate to target directory
page.goto(f"{SEAF_URL}/library/{repo_id}/docs/?p=%2F{TARGET_DIR.lstrip('/')}", wait_until="domcontentloaded", timeout=30000)
time.sleep(2)
# Trigger file chooser via JS click (button is in dropdown, not directly clickable)
with page.expect_file_chooser() as fc:
page.evaluate("""() => {
document.querySelector('[class*="upload-file"]')?.click();
}""")
time.sleep(1)
file_chooser = fc.value
file_chooser.set_files(files)
time.sleep(20) # Wait for upload to complete
browser.close()
```
Key details:
- Selector: `[class*="upload-file"]` — matches `.sf3-font-upload-files sf3-font mr-2 dropdown-item-icon`
- Must use JS click (`page.evaluate`) because the button is in a hidden dropdown (visibility: hidden → click timeout)
- Use `page.expect_file_chooser()` context manager to catch the file dialog
- `file_chooser.set_files(files)` accepts a list for batch upload
## API: Delete File
```bash
curl -ks -X DELETE -H "Authorization: Token {token}" \
-H "X-Requested-With: XMLHttpRequest" \
"https://seafile.yourdomain.com/api/v2.1/repos/{repo-id}/files/" \
-d "p=/path/to/file"
```
## Docker Server Administration
### Running seaf-fsck
```bash
# Find the seafile-server version directory
docker exec seafile bash -lc 'ls /opt/seafile/ | grep seafile-server'
# → seafile-server-12.0.14
# Run fsck
docker exec seafile bash -lc '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'
```
The fsck checks consistency between MySQL metadata (Repo table, commits) and the file block storage. If MySQL tables are missing or the `seafile` DB user doesn't exist, fsck will report warnings but exit cleanly without actually checking anything.
### fsck Silent Death (OOM Under Heavy I/O)
**Pitfall**: `seaf-fsck` can die silently under heavy I/O competition
(e.g. concurrent Ceph recovery + RBD copy on the same OSDs). The process
gets OOM-killed or I/O-starved, leaving no error in the fsck output log.
The log simply stops at the last repo being checked, with no "finished"
or error message.
**Detection**: Don't trust the log output alone — check if the process
is still alive:
```bash
# Inside the CT:
ps aux | grep seaf-fsck | grep -v grep | wc -l
# 0 = process died, needs restart
# 3+ = process running (multiple threads)
```
**Restart**: Simply re-run the fsck command. Already-checked repos are
processed quickly (seconds), so progress jumps forward rapidly after
restart. Large photo repos ("Fotos", "Lightroom Bilder") are the usual
stalling points — they have tens of thousands of commits and millions
of file blocks.
**Large repos**: Repos with 24000-34000+ commits take hours per repo
under normal I/O. Under I/O contention they may appear hung for 1+ hour
per repo. The process is alive (state `Dl` = disk I/O wait) but barely
progressing. Serializing I/O (pausing competing operations) dramatically
speeds up fsck.
### Recovery from Lost MySQL Volume
See `references/seafile-docker-recovery.md` for the full debugging procedure covering:
- `wait_for_mysql()` infinite loop diagnosis
- `init_seafile_server()` skipping schema creation
- Selective restore from PBS backup (MySQL volume + conf only, skip seafile-data)
- Manual DB init via `docker run` setup script (Option B)
- Manual DB/User recreation and SQL schema import (Option B2)
- **Repo reconstruction from commit storage** (Option C) — when DBs are fresh but
file blocks are intact, scan commit JSON objects to find HEAD commits and
register repos in `Repo`/`Branch`/`RepoOwner`/`RepoInfo` tables. See
`references/seafile-repo-reconstruction.md` for the full procedure.
## API Pitfalls
- **Wrong API paths** — `/api/v2.1/repos/{repo}/directory/` and `/api/v2.0/` return HTML error pages. Use `/api2/repos/{repo-id}/dir/` for listing.
- **Missing header** — `X-Requested-With: XMLHttpRequest` is required for most API calls, or the server redirects to the login page.
- **Cookie login does NOT work** — The AJAX form POST (`/seahub/ajax/login/`) renders an error page. Always use token auth.
- **Web UI login returns 403 but sets cookie** — The Seafile login POST (`/accounts/login/`) may return HTTP 403 despite successfully setting the session cookie. This is expected on Seafile 13.x; subsequent requests to the same session work.
- **URL encoding** — Paths must be URL-encoded for `p=` parameter (`space` → `%20`, `/` → `%2F`).
- **Seafile 13.x (Pro/Plus)** — The `/api/v2.1/repos/{repo}/files/`, `/seafhttp/`, and WebDAV endpoints all return 404/400. Token auth works for repo listing (`/api2/repos/`) and dir listing (`/api2/repos/{repo}/dir/`), but file download is broken via API. Use web login + requests.Session() workaround or manual download.
- **WebDAV disabled** — Seafile may have WebDAV disabled on newer versions. Check with `curl -k -o /dev/null -w '%{http_code}' -X PROPFIND "https://server/dav/"`.
- **Auth'd 500 vs unauth'd 403** — If authenticated calls (`Authorization: Token ...`) return HTTP 500 (HTML "Page unavailable") while unauthenticated calls return HTTP 403 (valid JSON), the backend is in a broken state. Get a fresh token, and if all endpoints return 500, the server needs a restart.
devops/seafile-api/references/seafile-docker-recovery.md
+498
View File
@@ -0,0 +1,498 @@
# Seafile Docker Recovery: MySQL Volume Loss & Selective PBS Restore
## Scenario
Seafile MC Docker edition on CT 111 (n5pro, 10.0.20.91). The `.env` was changed
(image downgraded from `seafile-mc:13.0-latest` to `seafile-mc:12.0-latest`,
hostname changed from `cloud.familie-schoen.com` to `seafile.familie-schoen.com`),
which triggered MySQL volume reinitialization — wiping all Seafile databases
(`seafile_db`, `ccnet_db`, `seahub_db`) and the `seafile` MySQL user. The file
block storage (`/opt/seafile-data/`) remained intact. A PBS backup of CT 111 from
2026-07-02 existed on `noris_s3` storage. The original `/opt/seafile/` config
directory was preserved as a ZIP by the user.
## Symptoms
### Symptom 1: Container "Up" but not actually serving
```bash
docker ps # → seafile Up 7 hours
docker logs seafile --tail 50 # → "waiting for mysql server to be ready: mysql is not ready" (repeating forever)
```
The container's entrypoint (`/scripts/start.py`) calls `wait_for_mysql()` before
proceeding to `init_seafile_server()`. If MySQL connection fails, it loops forever.
The container stays "Up" (PID 1 alive) but Seafile never starts.
### Symptom 2: seaf-fsck warns about DB connection
```bash
docker exec seafile bash -lc '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'
# → [WARNING] Failed to connect to MySQL: Access denied for user 'seafile'@'172.18.0.5'
# → seaf-fsck run done
# → Done.
```
Fsck exits 0 but didn't actually check anything — the DB connection failed.
### Symptom 3: MySQL has no Seafile databases
```bash
docker exec seafile-mysql mysql -uroot -p'PASSWORD' -e 'SHOW DATABASES;'
# → information_schema, mysql, performance_schema, sys (NO seafile_db, ccnet_db, seahub_db)
```
### Symptom 4: No seafile MySQL user
```bash
docker exec seafile-mysql mysql -uroot -p'PASSWORD' \
-e "SELECT user,host FROM mysql.user WHERE user LIKE '%sea%';"
# → Empty set
```
## Root Cause Analysis
### How `wait_for_mysql()` works
File: `/scripts/utils.py` inside the seafile container:
```python
def wait_for_mysql():
db_host = get_conf('DB_HOST', '127.0.0.1')
db_user = 'root'
db_passwd = get_conf('DB_ROOT_PASSWD', '')
# BUT: if seafile.conf exists, it overrides with the seafile user!
seafile_conf_path = join(topdir, 'conf', 'seafile.conf')
if exists(seafile_conf_path):
cp = ConfigParser()
cp.read(seafile_conf_path)
db_host = cp.get('database', 'host') # → 'db'
db_user = cp.get('database', 'user') # → 'seafile'
db_passwd = cp.get('database', 'password')
db_port = int(cp.get('database', 'port'))
while True:
try:
connection = pymysql.connect(host=db_host, port=db_port,
user=db_user, passwd=db_passwd)
except Exception as e:
# prints "mysql is not ready" and sleeps
```
**Critical insight**: Once `seafile.conf` exists (from a prior successful install),
`wait_for_mysql()` uses the `seafile` user — NOT root. If the MySQL volume was
wiped, the `seafile` user doesn't exist, and this loop runs forever.
### Why `init_seafile_server()` doesn't fix it
Even after manually recreating the `seafile` user and databases, restarting the
container produces:
```
Skip running setup-seafile-mysql.py because there is existing seafile-data folder.
```
The bootstrap script (`/scripts/bootstrap.py`) skips DB initialization when
`seafile-data/` already exists. So the MySQL tables are never created, and fsck
reports `Table 'seafile_db.Repo' doesn't exist`.
## Recovery Procedure
### Option A: Full PBS Restore of MySQL Volume (Preferred)
When a PBS backup exists, restoring the entire MySQL data directory is cleanest
because it includes all users, grants, schemas, and data.
⚠️ **Before attempting PBS restore, verify the backup is restorable** (learned
the hard way 2026-07-06):
1. Check verification state in `pvesh get` output — if `{"state":"failed"}`,
the backup has missing/corrupt chunks and may not restore.
2. Check snapshot directory on PBS server for `.didx` vs `.tmp_didx` —
only `.didx` snapshots are complete and restorable.
3. Check for `.bad` chunk files on PBS server — if any chunk referenced by
the backup is `.bad` (0 bytes), restore aborts immediately with no option
to skip. Files dependent on that chunk are permanently lost.
See `proxmox-ve-administration` skill → `references/pbs-lxc-setup-2026-07.md`
pitfalls #13–16 for PBS credential lookup, verification checking, corrupted
chunk diagnosis, and incomplete backup detection.
If PBS restore fails due to corrupted chunks, fall back to Option B (manual
recreation) and use `seaf-fsck.sh --repair` to rebuild repo metadata from
the intact `/opt/seafile-data/` file blocks.
#### Step 1: Locate the PBS backup
```bash
# On the PVE node hosting the CT (n5pro = 10.0.20.91):
pvesh get /nodes/$(hostname)/storage/noris_s3/content --content-type backup 2>&1 \
| awk -F'│' '{print $3}' \
| grep 'ct/111'
# → noris_s3:backup/ct/111/2026-07-02T21:00:00Z
```
**Note**: `pvesh get` outputs a Unicode box-drawing table, not JSON. Parse with
`awk -F'│'` (Unicode pipe character U+2502). Column 3 contains the volid.
#### Step 2: Stop Seafile containers
```bash
# On the PVE node:
pct exec 111 -- docker stop seafile seafile-mysql seafile-memcached
```
#### Step 3: Rename broken MySQL volume
```bash
pct exec 111 -- mv /opt/seafile-mysql/db /opt/seafile-mysql/db.broken-$(date +%Y%m%d)
```
#### Step 4: Restore from PBS
Use `pct restore` with `--storage` pointing to the CT's root storage, then
selectively extract only `/opt/seafile-mysql/db/` from the restored backup.
Alternatively, use `proxmox-backup-client` to restore specific files:
```bash
# Map the PBS backup to a temp location
# PBS restore maps the CT filesystem
pct exec 111 -- bash -c '
mkdir -p /tmp/restore
# Use pb-client to restore just the mysql db path
# (requires PBS credentials)
'
```
#### Step 5: Verify MySQL data
```bash
pct exec 111 -- docker start seafile-mysql
sleep 10
pct exec 111 -- docker exec seafile-mysql mysql -uroot -p'PASSWORD' \
-e "SELECT user,host FROM mysql.user WHERE user='seafile'; SHOW DATABASES;"
# Should show: seafile@%, seafile_db, ccnet_db, seahub_db
```
#### Step 6: Start Seafile
```bash
pct exec 111 -- docker start seafile
sleep 30
pct exec 111 -- docker logs seafile --tail 20
# Should show: "Seafile server started", "Seahub is started"
```
#### Step 7: Run fsck
```bash
pct exec 111 -- docker exec seafile bash -lc \
'/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'
```
With restored DB, fsck should connect and check all repos.
### Option B: Manual DB Init via `docker run` Setup Script (Preferred over SQL import)
When PBS restore fails and you need fresh DBs, the cleanest approach is running
the official `setup-seafile-mysql.sh` in a throwaway container with proper env
vars. This creates all schemas, users, and config files correctly.
**Critical**: The `bootstrap.py` in the main container skips `init_seafile_server()`
when `/shared/seafile/seafile-data/` exists. To force fresh DB init:
1. Stop the seafile container
2. Rename `seafile-data` temporarily: `mv .../seafile-data .../seafile-data.bak`
3. Clear `conf/`: `rm -rf .../conf/*`
4. Drop any half-created databases: `DROP DATABASE ccnet_db, seafile_db, seahub_db`
5. Run setup in a throwaway container:
```bash
docker run --rm --name seafile-setup --network seafile-net \
-e SERVER_NAME=seafile \
-e SERVER_IP=cloud.familie-schoen.com \
-e MYSQL_USER=seafile \
-e MYSQL_USER_PASSWD='SEAFILE_PW' \
-e MYSQL_USER_HOST=% \
-e MYSQL_HOST=db \
-e MYSQL_PORT=3306 \
-e MYSQL_ROOT_PASSWD='ROOT_PW' \
-e CCNET_DB=ccnet_db \
-e SEAFILE_DB=seafile_db \
-e SEAHUB_DB=seahub_db \
-e SEAFILE_SERVER_HOSTNAME=cloud.familie-schoen.com \
-e SEAFILE_SERVER_PROTOCOL=https \
-e INIT_SEAFILE_ADMIN_EMAIL=admin@example.com \
-e INIT_SEAFILE_ADMIN_PASSWORD=password \
-e TIME_ZONE=Europe/Berlin \
-v /opt/seafile-data:/shared \
seafileltd/seafile-mc:13.0-latest \
/opt/seafile/seafile-server-13.0.19/setup-seafile-mysql.sh auto -n seafile
```
⚠️ The env var names matter: `MYSQL_ROOT_PASSWD` (not `MYSQL_ROOT_PASSWORD`),
`MYSQL_USER_PASSWD` (not `MYSQL_USER_PASSWORD`). The setup script reads
`get_param_val(args.mysql_root_passwd, 'MYSQL_ROOT_PASSWD')` from `os.environ`.
6. Restore original `seafile-data` and `conf`:
```bash
rm -rf .../seafile-data && mv .../seafile-data.bak .../seafile-data
# Restore conf from backup (contains correct DB credentials, SERVICE_URL, etc.)
cp -a .../conf.bak/* .../conf/
```
7. Update `SERVICE_URL` in `seahub_settings.py` if hostname changed:
```bash
sed -i 's|old.domain.com|new.domain.com|g' .../conf/seahub_settings.py
```
8. Start the container normally — it will run upgrade scripts (12.0→13.0 etc.)
and start Seafile.
### Option B2: Manual SQL Schema Import (Legacy Approach)
If `docker run` setup doesn't work, manually recreate databases and import schemas.
#### Step 1: Create databases and user
```bash
docker exec seafile-mysql mysql -uroot -p'ROOT_PW' -e "
CREATE DATABASE IF NOT EXISTS ccnet_db CHARACTER SET utf8 COLLATE utf8_general_ci;
CREATE DATABASE IF NOT EXISTS seafile_db CHARACTER SET utf8 COLLATE utf8_general_ci;
CREATE DATABASE IF NOT EXISTS seahub_db CHARACTER SET utf8 COLLATE utf8_general_ci;
CREATE USER 'seafile'@'%' IDENTIFIED BY 'SEAFILE_PW';
GRANT ALL PRIVILEGES ON ccnet_db.* TO 'seafile'@'%';
GRANT ALL PRIVILEGES ON seafile_db.* TO 'seafile'@'%';
GRANT ALL PRIVILEGES ON seahub_db.* TO 'seafile'@'%';
FLUSH PRIVILEGES;
"
```
#### Step 2: Import SQL schemas
The seafile container has no `mysql` client. Use the seafile-mysql container:
```bash
SQLDIR=/opt/seafile/seafile-server-12.0.14/sql/mysql
# ccnet schema
docker exec seafile-mysql mysql -uroot -p'ROOT_PW' ccnet_db \
< <(docker exec seafile cat $SQLDIR/ccnet.sql)
# seafile schema
docker exec seafile-mysql mysql -uroot -p'ROOT_PW' seafile_db \
< <(docker exec seafile cat $SQLDIR/seafile.sql)
# seahub schema (located in seahub/scripts/upgrade/sql/)
docker exec seafile-mysql mysql -uroot -p'ROOT_PW' seahub_db \
< <(docker exec seafile cat /opt/seafile/seafile-server-12.0.14/seahub/scripts/upgrade/sql/12.0.0/mysql/seahub.sql)
```
⚠️ **This only recreates empty tables.** Existing repo metadata, users, and
library mappings are lost. File blocks in `/opt/seafile-data/` remain but are
orphaned (no DB entries pointing to them). The admin account must be recreated
via `check_init_admin.py` or the Seafile web UI.
#### Step 3: Restart and run fsck
```bash
docker restart seafile
sleep 30
docker exec seafile bash -lc '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'
```
### Option C: Reconstruct Repo Registrations from Commit Storage
When DBs are freshly initialized but `/opt/seafile-data/` (file blocks + commits)
is intact, repos can be reconstructed by scanning the commit storage and
registering them in the DB. See `references/seafile-repo-reconstruction.md` for
the full procedure with Python scripts.
**Overview of the technique:**
1. **Scan commit storage**: `storage/commits/<repo_id>/<XX>/<YYYY...>/` — each
file is a JSON object with `commit_id`, `root_id`, `parent_id`,
`second_parent_id`, `repo_name`, `repo_desc`, `ctime`, `creator_name`.
2. **Find HEAD commit**: The commit NOT referenced as `parent_id` by any other
commit in the same repo. For repos with many commits (>1000), use
`find -printf '%T@ %p\n' | sort -rn | head -20` to read only the newest 20.
3. **Register in DB**: Insert into `Repo`, `Branch`, `RepoOwner`, and `RepoInfo`
tables. The `Repo` table only needs `repo_id`; `Branch` needs `(name, repo_id,
commit_id)`; `RepoOwner` needs `(repo_id, owner_id)`; `RepoInfo` needs
`(repo_id, name, ...)`.
4. **Run `seaf-fsck.sh --repair`**: After registering repos, fsck verifies
integrity and repairs inconsistencies.
⚠️ **Encrypted repo commit IDs may have suffixes** (e.g.,
`e155972728d6a3205726b94243078ae5f0aaaa3a.FYMWR3`) that exceed the 40-char
`commit_id` column. Truncate to 40 chars before inserting.
⚠️ **`seaf-fsck.sh --repair` only checks registered repos** — it does NOT discover
or register new repos. All repos must be registered in the DB first.
⚠️ **`seaf-fsck.sh --export` also only exports registered repos** — it cannot
recover unregistered repos from the filesystem.
## Key Files in Seafile Docker Container
| Path | Purpose |
|------|---------|
| `/opt/seafile/conf/seafile.conf` | Main config — DB host, user, password, fileserver port |
| `/opt/seafile/seafile-server-VERSION/` | Server installation (scripts, SQL, binaries) |
| `/opt/seafile/seafile-server-VERSION/seaf-fsck.sh` | Filesystem consistency checker |
| `/opt/seafile/seafile-server-VERSION/seaf-gc.sh` | Garbage collector for orphaned blocks |
| `/opt/seafile/seafile-server-VERSION/sql/mysql/` | SQL schema files (ccnet.sql, seafile.sql) |
| `/opt/seafile/seafile-server-VERSION/seahub/scripts/upgrade/sql/VERSION/mysql/seahub.sql` | Seahub DB schema |
| `/scripts/start.py` | Container entrypoint — calls `wait_for_mysql()` + `init_seafile_server()` |
| `/scripts/utils.py` | `wait_for_mysql()` implementation |
| `/scripts/bootstrap.py` | `init_seafile_server()` — skips if `seafile-data/` exists |
## Docker-Compose Layout (seafile-server.yml)
```yaml
services:
db:
image: mariadb:10.11
container_name: seafile-mysql
volumes:
- "${SEAFILE_MYSQL_VOLUME:-/opt/seafile-mysql/db}:/var/lib/mysql"
# ...
seafile:
image: seafileltd/seafile-mc:12.0-latest
container_name: seafile
environment:
- DB_HOST=db
- DB_USER=seafile
- DB_ROOT_PASSWD=${INIT_SEAFILE_MYSQL_ROOT_PASSWORD}
- DB_PASSWORD=${SEAFILE_MYSQL_DB_PASSWORD}
volumes:
- "${SEAFILE_VOLUME:-/opt/seafile-data}:/shared"
depends_on:
db:
condition: service_healthy
```
The `seafile` container's `/shared` mount corresponds to `/opt/seafile-data` on the
host. Inside the container, `/opt/seafile/` is symlinked from `/shared/seafile/`.
## Network Architecture
```
seafile-net (bridge, 172.18.0.0/16):
├── seafile-mysql 172.18.0.3 (hostname: db)
├── seafile-memcached 172.18.0.2
├── seafile-caddy 172.18.0.4
├── seafile 172.18.0.5
└── seadoc 172.18.0.6
```
`seafile.conf` references `host = db` which resolves via Docker DNS to the
`seafile-mysql` container.
## Pitfalls
1. **`wait_for_mysql()` uses `seafile` user, not `root`** — Once `seafile.conf`
exists, the startup script connects as the `seafile` MySQL user. If that user
doesn't exist (volume wipe), the loop runs forever. Container shows "Up" but
Seafile never starts.
2. **`init_seafile_server()` skips when `seafile-data/` exists** — Even after
recreating the MySQL user and empty databases, the bootstrap won't create
tables because it sees existing data. Must import SQL schemas manually or
restore the MySQL volume from backup.
3. **`seaf-fsck.sh` exits 0 even on DB failure** — It catches the DB connection
error, prints a WARNING, and exits cleanly. Don't rely on exit code; check
stderr for `[WARNING]` lines.
4. **No `mysql` client in seafile container** — Use `docker exec seafile-mysql
mysql ...` for all MySQL operations. The seafile container only has Python
and Seafile binaries.
5. **`seafile-data/` is large and usually intact** — When recovering, don't
restore `/opt/seafile-data/` from backup unless corruption is suspected. It
contains all file blocks (potentially hundreds of GB). Restoring only the
MySQL volume + conf is much faster.
6. **PBS backup listing via `pvesh`** — `pvesh get /nodes/$(hostname)/storage/
<storage>/content` outputs a Unicode table, not JSON. Parse with
`awk -F'│'` to extract volids. See `proxmox-ve-administration` skill for
PBS backup listing details.
7. **MariaDB vs MySQL image** — Seafile Docker uses `mariadb:10.11` but the
config says `type = mysql`. MariaDB is compatible. The `MARIADB_AUTO_UPGRADE=1`
env var handles schema upgrades on container start.
8. **Image version change can wipe MySQL volume** — Changing the Seafile Docker
image (e.g., downgrading `seafile-mc:13.0-latest` → `seafile-mc:12.0-latest`
or vice versa) can trigger MySQL volume reinitialization. The container detects
a version mismatch and reinitializes the data directory, wiping all databases
and users. **Always backup the MySQL volume (`/opt/seafile-mysql/db`) before
changing the image version in `.env`.**
9. **Compare `.env` before restoring** — When recovering, always diff the current
`.env` against any backup config. Key fields that differ between versions:
- `SEAFILE_IMAGE` (e.g., `13.0-latest` vs `12.0-latest`)
- `SEAFILE_SERVER_HOSTNAME` (e.g., `cloud.familie-schoen.com` vs
`seafile.familie-schoen.com`)
- `INIT_SEAFILE_ADMIN_EMAIL`
- `JWT_PRIVATE_KEY`
- `COMPOSE_FILE` list (e.g., `caddy.yml` may be added/removed)
Align the config BEFORE starting containers with a restored DB, otherwise
Seafile may reinitialize or fail to match the restored data.
10. **Telegram file transfer corrupts large ZIPs** — Sending large binary archives
(e.g., a MySQL data directory ZIP) via Telegram can truncate or corrupt the
file. The ZIP arrives with valid local file headers but no central directory
and invalid size fields (`0xFFFFFFFF`). For transferring server data between
PVE nodes, use `scp`, `rsync`, or PBS restore instead of Telegram file
attachments. If a received ZIP fails to open, verify with
`python3 -c "import zipfile; zipfile.ZipFile('file.zip').namelist()"` — if it
raises `BadZipFile`, the transfer corrupted it.
11. **`bootstrap.py` ENV variable substitution bug** — When `bootstrap.py`
calls `setup-seafile-mysql.py` via `subprocess.call(cmd, env={...})`,
the `env=` dict **replaces** the entire process environment rather than
augmenting it. The `env=` dict includes `MYSQL_ROOT_PASSWD` (mapped from
`INIT_SEAFILE_MYSQL_ROOT_PASSWORD`), but if the mapping is missing or
misnamed, the subprocess gets `MYSQL_ROOT_PASSWD=''` (empty string),
producing the error `using password: NO` from MySQL.
**Symptom**: `setup-seafile-mysql.py` fails with `Access denied for
'root'@'localhost' (using password: NO)` even though
`INIT_SEAFILE_MYSQL_ROOT_PASSWORD` is correctly set in the container env.
**Fix**: Run `setup-seafile-mysql.py` manually inside the container with
all required ENV vars explicitly exported:
```bash
docker exec -e MYSQL_ROOT_PASSWD='<root_pw>' \
-e MYSQL_USER='seafile' -e MYSQL_USER_PASSWD='<seafile_pw>' \
-e MYSQL_HOST=db -e MYSQL_PORT=3306 \
-e CCNET_DB=ccnet_db -e SEAFILE_DB=seafile_db -e SEAHUB_DB=seahub_db \
-e SEAFILE_SERVER_HOSTNAME=cloud.example.com \
-e INIT_SEAFILE_ADMIN_EMAIL=admin@example.com \
-e INIT_SEAFILE_ADMIN_PASSWORD='<admin_pw>' \
seafile /opt/seafile/seafile-server-13.0.19/setup-seafile-mysql.sh auto -n seafile
```
This bypasses `bootstrap.py`'s broken `env=` substitution and gives the
setup script direct access to all required variables.
12. **PBS corrupted chunk blocks ALL file restoration** — A single 0-byte `.bad`
chunk file on the PBS server (destroyed by garbage collection) makes the
ENTIRE backup unrestorable. PBS aborts on the first file referencing the
bad chunk, cleans up, and exits. There is no `--skip-bad-chunks` option.
Files alphabetically before the affected file MAY be partially restored,
but anything after is lost. The chunk path format is
`/mnt/datastore/<ds>/.chunks/<XX>/<hash>.0.bad`.
13. **Multiple backup formats can ALL be unusable** — In this incident, THREE
separate backup sources were tried and ALL failed:
- PBS backup: corrupted chunk (above)
- `seafile-mysql.zip` (Telegram-transferred): BadZipFile, no central directory
- `seafile-mysql.tar.zst`: truncated, unexpected EOF after 4 entries
- Offsite PBS: no CT111 backup existed at all
Lesson: never assume ANY backup is restorable without verifying. Always
test-restore before declaring a backup strategy viable.
devops/seafile-api/references/seafile-immich-backup-strategy.md
+202
View File
@@ -0,0 +1,202 @@
# Seafile & Immich Backup Strategy Analysis
## When to Use
When planning offsite/backup strategy for Seafile (558 GB) or Immich
on bandwidth-limited remote PBS, and deciding what to back up vs.
what to leave on Ceph-only redundancy.
## Seafile Backup Strategies (6 Options)
### Context
- Seafile Pro 13.0.19 MC Docker edition on CT111
- File blocks: `/opt/seafile-data/seafile/seafile-data/storage/blocks/` (intrinsically deduplicated, content-addressable)
- Config: `/opt/seafile/.env`, `seafile.conf`, `seahub_settings.py`
- DB: MySQL (`seafile_db`, `ccnet_db`, `seahub_db`) — currently in `seafile-mysql` container, migration to Galera planned
- Total size: ~558 GB (mostly file blocks)
- Ceph redundancy: blocks stored on Ceph EC4+1 (5 HDD OSDs, 1-failure tolerance)
### Option Matrix
| # | Strategy | Offsite Volume | Pros | Cons |
|---|----------|---------------|------|------|
| 1 | **PBS Full (current)** | 558 GB init, then delta | Simple, everything included | Huge for offsite, slow restore |
| 2 | **Config+DB only** | <1 GB | Fast daily offsite, trivial restore | Blocks only in Ceph — total Ceph loss = all files gone |
| 3 | **SeafGC + PBS exclude blocks** | ~2 GB | Clean, regularly garbage-collected | Blocks only in Ceph, needs GC discipline |
| 4 | **Seafile Native Backup** | Variable | Official way, incremental | Needs second Seafile server (not S3) |
| 5 | **RBD Snapshot + PBS** | 558 GB init, delta like #1 | Point-in-time consistent | Ceph pool must support snapshots |
| 6 | **Blocks on S3 (Pro feature)** | Rootfs ~2 GB offsite, blocks on S3 | Clean separation, Ceph relieved | Migration effort, S3 costs |
### Recommendation
- **Short-term**: Option 2 (Config+DB offsite) + regular `seaf-gc.sh`. Blocks
are redundant in Ceph EC4+1. For DR against total Ceph loss: Option 4 as
second pillar.
- **Long-term**: Option 6 (S3 storage backend) — moves blocks out of Ceph
entirely, making PBS offsite trivial (~2 GB) and Ceph only hosts config+DB.
## Seafile Pro S3 Storage Backend
Seafile Pro supports S3 as **primary storage backend** (not backup target).
All file blocks are stored directly on S3 instead of local disk/Ceph.
### Configuration (in `seafile.conf`)
```ini
[storage_backend]
name = s3
bucket = seafile-data
key_id = XXX
key = XXX
region = eu-central-1
endpoint = https://s3.example.com
```
### Key Distinctions
- **Primary storage** — replaces local/Ceph block storage entirely
- **NOT a backup target** — S3 is where live data lives, not a backup copy
- **Migration required** — existing blocks must be migrated to S3
- **Ceph relief** — once on S3, Ceph only holds config+DB (~2 GB), making
PBS offsite backup trivial and fast
### When to Choose S3 Backend
- Ceph capacity is constrained (OSDs near full)
- Offsite backup of 558 GB blocks is impractical (bandwidth-limited)
- S3 storage is cheaper than maintaining Ceph OSDs
- Want clean separation: config+DB on Ceph (small, fast), blocks on S3 (large, cheap)
## Seafile Native Backup (`seaf-backup-cmd`)
### How It Works
- Uses Seafile's internal RPC API to synchronize libraries to a **second
Seafile server**
- Incremental — only changed commits/blocks are transferred
- The second server must be a full Seafile server (same version)
- Commands: `seaf-backup-cmd status`, `seaf-backup-cmd sync <repo-id>`
### Limitations
- **Needs a second Seafile server** — not a simple file/rsync target
- **Not S3-compatible** — target must be Seafile server with storage backend
- **Setup complexity** — must configure backup server in `seafile.conf`
- **Per-library sync** — no global "backup everything" by default
### Location
```bash
# In the seafile container:
docker exec seafile /opt/seafile/seafile-server-latest/seahub/scripts/seaf-backup-cmd.sh
# Error without proper SEAFILE_DATA_DIR env — needs server running
```
### When to Choose
- Need true DR replica (second running Seafile instance)
- Have infrastructure for a second server
- Want incremental, deduplicated replication (not full copies)
## Immich Backup Considerations
### Architecture
- CT115 (immich): stopped, 20 GB rootfs — upload directory location unclear
- CT117 (immich-postgresql): running, 914 MB DB on proxmox6
- Photos/videos are NOT deduplicated (unlike Seafile blocks)
### Strategy Options
| # | Strategy | Offsite Volume | Notes |
|---|----------|---------------|-------|
| 1 | PBS Full both CTs | ~21 GB | Simple but may grow |
| 2 | DB dump + upload-dir rsync | DB ~1 GB + uploads | Selective |
| 3 | DB only offsite | <1 GB | Photos only in Ceph |
| 4 | Uploads on Ceph, DB on Galera | DB offsite <1 GB | Clean separation |
### Key Question
Where are the actual photo/video uploads stored? CT115 is stopped —
uploads may be on a mount point, external volume, or within the
PostgreSQL CT. Must identify before choosing a strategy.
## Bandwidth-Aware Offsite Backup Prioritization
When adding services to a bandwidth-limited offsite PBS:
### Tier 1: Small + Critical Infrastructure (add first)
- Traefik (~3 GB) — reverse proxy config, all routes
- Gitea (~1.2 GB) — code repositories
- Authelia, DNS — tiny but critical
- Paperless (~16 GB) — documents (medium size, high value)
### Tier 2: Medium + Important
- Litellm (~3.4 GB) — API config
- Seafile config+DB (~2 GB) — without blocks
### Tier 3: Evaluate Separately
- Seafile blocks (~558 GB) — consider S3 backend or native backup
- Immich uploads — identify storage location first
### Skip
- Frigate (~40 GB) — rewritable video, low value
- High-churn data that's easily regenerated
## PBS Offsite Job Configuration
### Adding CTs to Offsite Job
```bash
# Edit /etc/pve/jobs.cfg from any PVE node (pmxcfs replicates)
# Current: vmid 106
# Target: vmid 106,108,104,99999
sed -i '/offsite-ha-daily/,/^$/{s/vmid 106/vmid 106,108,104,99999/}' /etc/pve/jobs.cfg
```
### Initial Full Backup (Manual)
Must run `vzdump` on the node where each CT actually runs (see
`proxmox-ve-administration` → `references/pbs-lxc-setup-2026-07.md`
pitfall #17 for the silent-skip issue):
```bash
# Determine node for each CT first
pvesh get /cluster/resources --type vm --output-format json | python3 -c "
import json,sys
for v in json.load(sys.stdin):
if v.get('vmid') in [108,99999,104]:
print(f\"CT{v['vmid']:5d} node={v['node']}\")"
# Run on correct nodes
ssh root@10.0.20.60 "vzdump 108 --storage noris_offsite --mode snapshot --compress zstd"
ssh root@10.0.20.70 "vzdump 99999 --storage noris_offsite --mode snapshot --compress zstd"
ssh root@10.0.20.91 "vzdump 104 --storage noris_offsite --mode snapshot --compress zstd"
```
### Verification
```bash
# Check backups arrived at PBS (may need content cache refresh)
pvesh get /nodes/proxmox1/storage/noris_offsite/content --output-format json | \
python3 -c "
import json,sys,datetime
for b in sorted(json.load(sys.stdin), key=lambda x: x.get('ctime',0), reverse=True)[:10]:
print(f\"{b['volid']:55s} {b.get('size',0)//1024//1024:>8} MB {datetime.datetime.fromtimestamp(b.get('ctime',0)).strftime('%Y-%m-%d %H:%M')}\")"
```
## Session Context (2026-07-07)
- Offsite job expanded: `vmid 106` → `vmid 106,108,104,99999`
- Initial full backups triggered manually on correct nodes
- CT111 (Seafile) backup showed 0 MB on noris_s3 — empty/failed backup
(previous valid: 558 GB on 2026-07-06)
- Frigate excluded from all backups per user decision
- Seafile backup strategy discussion: user exploring alternatives to
558 GB full backup, considering S3 storage backend (Pro feature)
- Immich: CT115 stopped, upload location unidentified
devops/seafile-api/references/seafile-repo-reconstruction.md
+247
View File
@@ -0,0 +1,247 @@
# Seafile Repo Reconstruction from Commit Storage
## When to Use
After MySQL databases have been wiped/reinitialized but `/opt/seafile-data/`
(file blocks + commit objects) is intact. This procedure registers all repos
found in the commit storage back into the fresh MySQL DBs so they appear in the
web UI and API.
## Commit Storage Layout
```
/opt/seafile-data/seafile/seafile-data/storage/commits/
<repo_id>/
<first2chars_of_commit_sha>/
<remaining38chars_of_commit_sha> ← JSON file, commit object
```
Example: commit `5fb48bca960bf13a15df0b84dbee73ef1d8f98d6` lives at:
`storage/commits/<repo_id>/5f/b48bca960bf13a15df0b84dbee73ef1d8f98d6`
## Commit Object Format (JSON)
Each commit file is a JSON object:
```json
{
"commit_id": "5fb48bca960bf13a15df0b84dbee73ef1d8f98d6",
"root_id": "0000000000000000000000000000000000000000",
"repo_id": "3428f3b9-d6c0-4d23-8a03-f34ce8b888e1",
"creator_name": "dominik@familie-schoen.com",
"creator": "0000000000000000000000000000000000000000",
"description": "Created library",
"ctime": 1755169060,
"parent_id": null,
"second_parent_id": null,
"repo_name": "Arztrechnungen",
"repo_desc": "Arztrechnungen",
"repo_category": null,
"no_local_history": 1,
"version": 1
}
```
## Finding the HEAD Commit
The HEAD commit is the one NOT referenced as `parent_id` (or `second_parent_id`)
by any other commit in the same repo. Algorithm:
1. Read all (or top N by mtime) commit JSON objects for a repo
2. Collect all `parent_id` and `second_parent_id` values into a set
3. HEAD = the commit ID(s) NOT in that parent set
4. If multiple HEADs exist (branch tips), pick the one with the latest `ctime`
### Performance for Large Repos
Some repos have tens of thousands of commits (34000+). Reading all JSON files
is too slow (60s+ timeout). Optimization:
```bash
# Get only the 20 newest commit files by filesystem mtime
find "$repo_dir" -type f -printf '%T@ %p\n' | sort -rn | head -20
```
Then parse only those 20 files. The HEAD is very likely among the newest commits.
⚠️ Filesystem mtime reflects when PBS restored the files, NOT the commit creation
time. Don't rely on mtime alone for determining HEAD — use the parent-reference
algorithm on the sampled commits.
## DB Tables to Populate
| Table | DB | Columns | Notes |
|-------|-----|---------|-------|
| `Repo` | `seafile_db` | `id` (auto), `repo_id` (unique) | Just insert the repo_id |
| `Branch` | `seafile_db` | `id` (auto), `name`, `repo_id`, `commit_id` | name='master', commit_id=HEAD |
| `RepoOwner` | `seafile_db` | `id` (auto), `repo_id` (unique), `owner_id` | owner_id = email address |
| `RepoInfo` | `seafile_db` | `id` (auto), `repo_id` (unique), `name`, `update_time`, `version`, `is_encrypted`, `last_modifier`, `status` | name from commit JSON `repo_name` |
Without `RepoOwner`, repos won't appear in the API (`/api2/repos/`) even if
registered in `Repo` and `Branch`. Without `RepoInfo`, repos show generic names.
## Step-by-Step Procedure
### Step 1: Generate SQL for Repo + Branch registration
```python
#!/usr/bin/env python3
import os, json, subprocess
COMMITS_DIR = "/opt/seafile-data/seafile/seafile-data/storage/commits"
OWNER = "dominik@example.com" # admin email
sql_lines = ["-- Register repos"]
for repo_id in sorted(os.listdir(COMMITS_DIR)):
repo_dir = os.path.join(COMMITS_DIR, repo_id)
if not os.path.isdir(repo_dir):
continue
# Get newest 20 commits by mtime
result = subprocess.run(
["find", repo_dir, "-type", "f", "-printf", "%T@ %p\n"],
capture_output=True, text=True, timeout=10)
lines = result.stdout.strip().split('\n') if result.stdout.strip() else []
lines.sort(key=lambda x: float(x.split()[0]) if x.strip() else 0, reverse=True)
top_files = [l.split(None, 1)[1] for l in lines[:20] if ' ' in l]
commits = {}
all_parents = set()
repo_name = None
latest_ctime = 0
latest_cid = None
for fpath in top_files:
parts = fpath.rstrip('/').split('/')
commit_id = parts[-2] + parts[-1] # subdir + filename = full SHA
try:
with open(fpath, 'r') as f:
data = json.load(f)
commits[commit_id] = data
if data.get('parent_id'):
all_parents.add(data['parent_id'])
if data.get('second_parent_id'):
all_parents.add(data['second_parent_id'])
if data.get('ctime', 0) > latest_ctime:
latest_ctime = data['ctime']
latest_cid = commit_id
rn = data.get('repo_name')
if rn:
repo_name = rn
except:
pass
head_candidates = [c for c in commits if c not in all_parents]
head_id = head_candidates[0] if len(head_candidates) == 1 else \
max(head_candidates, key=lambda c: commits[c].get('ctime', 0)) if head_candidates else latest_cid
if not head_id:
continue
if head_id in commits:
rn = commits[head_id].get('repo_name')
if rn:
repo_name = rn
if not repo_name:
repo_name = f"Repo-{repo_id[:8]}"
safe_name = repo_name.replace("'", "\\'")
# Truncate encrypted commit IDs to 40 chars
head_id_safe = head_id[:40]
sql_lines.append(f"INSERT IGNORE INTO seafile_db.Repo (repo_id) VALUES ('{repo_id}');")
sql_lines.append(f"INSERT IGNORE INTO seafile_db.Branch (name, repo_id, commit_id) VALUES ('master', '{repo_id}', '{head_id_safe}');")
sql_lines.append(f"INSERT IGNORE INTO seafile_db.RepoOwner (repo_id, owner_id) VALUES ('{repo_id}', '{OWNER}');")
sql_lines.append(f"INSERT IGNORE INTO seafile_db.RepoInfo (repo_id, name, update_time, version, is_encrypted, last_modifier, status) VALUES ('{repo_id}', '{safe_name}', UNIX_TIMESTAMP(), 1, 0, '{OWNER}', 0);")
with open("/tmp/register_repos.sql", "w") as f:
f.write("\n".join(sql_lines) + "\n")
print(f"Generated {len(sql_lines)-1} statements")
```
### Step 2: Execute the SQL
```bash
docker exec -i seafile-mysql mysql -uroot -p'ROOT_PW' < /tmp/register_repos.sql
```
### Step 3: Run seaf-fsck repair
```bash
docker exec seafile /opt/seafile/seafile-server-13.0.19/seaf-fsck.sh --repair
```
For large repos, run in background:
```bash
nohup docker exec seafile /opt/seafile/seafile-server-13.0.19/seaf-fsck.sh --repair \
> /tmp/fsck-output.log 2>&1 &
```
Monitor progress:
```bash
grep -c "Fsck finished" /tmp/fsck-output.log
tail -5 /tmp/fsck-output.log
```
### Step 4: Verify via API
```bash
# Get auth token
TOKEN=$(curl -s -X POST http://localhost:80/api2/auth-token/ \
-d 'username=admin@example.com&password=PASSWORD' | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
# List repos
curl -s -H "Authorization: Token $TOKEN" http://localhost:80/api2/repos/ \
| python3 -c "import sys,json; data=json.load(sys.stdin); print(f'{len(data)} repos visible')"
# Check a specific repo's contents
curl -s -H "Authorization: Token $TOKEN" \
"http://localhost:80/api2/repos/<repo_id>/dir/" \
| python3 -c "import sys,json; data=json.load(sys.stdin); print(f'{len(data)} items')"
```
## Pitfalls
1. **Encrypted repo commit IDs have suffixes** — Encrypted repos append a suffix
like `.FYMWR3` to the 40-char SHA1, producing a 46-char string that exceeds
the `commit_id CHAR(41)` column. Error: `Data too long for column 'commit_id'`.
Fix: truncate to 40 chars with `head_id[:40]`.
2. **`seaf-fsck.sh --repair` only checks REGISTERED repos** — It does NOT
discover or register new repos from the filesystem. All repos must be inserted
into the `Repo` table first. Same applies to `seaf-fsck.sh --export`.
3. **Repos invisible without `RepoOwner`** — Even with `Repo` + `Branch` entries,
repos won't appear in `/api2/repos/` without a `RepoOwner` entry mapping the
repo to a user's email.
4. **Repos show generic names without `RepoInfo`** — Without `RepoInfo`, repos
appear as unnamed entries. The `name` column comes from the commit JSON's
`repo_name` field.
5. **Repo sizes show 0 after reconstruction** — Size calculation happens during
`seaf-fsck` or background indexing. Sizes will populate after fsck completes.
6. **Shares, groups, and non-admin users are lost** — This procedure restores
repo ownership and file access, but share permissions, group memberships,
and user accounts (except the admin) must be recreated manually.
7. **Large repos stall fsck** — Repos with 30000+ commits can take 10+ minutes
during fsck. Run in background and monitor via `grep -c "Fsck finished"`.
8. **CRITICAL: Never run multiple fsck processes simultaneously** — If
`seaf-fsck.sh --repair` is already running (check with
`ps aux | grep seaf-fsck | grep -v grep`), do NOT start a second one — even
via a different invocation method (e.g., one via `nohup` on the host and
another via `docker exec`). Two concurrent `--repair` processes on the same
storage can corrupt commit/block data. Always `kill -9` the duplicate before
continuing. Only one `seaf-fsck` process should ever be running at a time.
9. **fsck can die silently (OOM-kill)** — `seaf-fsck.sh --repair` may be
OOM-killed by the kernel with no trace in the log file. The log simply
stops at whatever repo was being processed, and `ps aux | grep seaf-fsck`
returns 0 processes. Always check process liveness, not just log tail.
If killed, restart fsck — it resumes from where it left off (repos
already checked are quickly re-verified).
10. **Large photo repos stall fsck indefinitely** — Repos with many large
image files (e.g., "Fotos", "Lightroom Bilder") can take 30+ minutes
each during fsck. Combined with concurrent Ceph I/O (recovery, RBD
copies), fsck throughput drops to near-zero. If fsck appears stuck,
check whether heavy I/O is competing and consider pausing other
operations.