dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
hermes-skills/devops/seafile-api/references/seafile-docker-recovery.md
T

20 KiB
Raw Permalink Blame History

Seafile Docker Recovery: MySQL Volume Loss & Selective PBS Restore

Scenario

Seafile MC Docker edition on CT 111 (n5pro, 10.0.20.91). The .env was changed (image downgraded from seafile-mc:13.0-latest to seafile-mc:12.0-latest, hostname changed from cloud.familie-schoen.com to seafile.familie-schoen.com), which triggered MySQL volume reinitialization — wiping all Seafile databases (seafile_db, ccnet_db, seahub_db) and the seafile MySQL user. The file block storage (/opt/seafile-data/) remained intact. A PBS backup of CT 111 from 2026-07-02 existed on noris_s3 storage. The original /opt/seafile/ config directory was preserved as a ZIP by the user.

Symptoms

Symptom 1: Container "Up" but not actually serving

docker ps                    # → seafile Up 7 hours
docker logs seafile --tail 50 # → "waiting for mysql server to be ready: mysql is not ready" (repeating forever)

The container's entrypoint (/scripts/start.py) calls wait_for_mysql() before proceeding to init_seafile_server(). If MySQL connection fails, it loops forever. The container stays "Up" (PID 1 alive) but Seafile never starts.

Symptom 2: seaf-fsck warns about DB connection

docker exec seafile bash -lc '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'
# → [WARNING] Failed to connect to MySQL: Access denied for user 'seafile'@'172.18.0.5'
# → seaf-fsck run done
# → Done.

Fsck exits 0 but didn't actually check anything — the DB connection failed.

Symptom 3: MySQL has no Seafile databases

docker exec seafile-mysql mysql -uroot -p'PASSWORD' -e 'SHOW DATABASES;'
# → information_schema, mysql, performance_schema, sys  (NO seafile_db, ccnet_db, seahub_db)

Symptom 4: No seafile MySQL user

docker exec seafile-mysql mysql -uroot -p'PASSWORD' \
  -e "SELECT user,host FROM mysql.user WHERE user LIKE '%sea%';"
# → Empty set

Root Cause Analysis

How wait_for_mysql() works

File: /scripts/utils.py inside the seafile container:

def wait_for_mysql():
    db_host = get_conf('DB_HOST', '127.0.0.1')
    db_user = 'root'
    db_passwd = get_conf('DB_ROOT_PASSWD', '')

    # BUT: if seafile.conf exists, it overrides with the seafile user!
    seafile_conf_path = join(topdir, 'conf', 'seafile.conf')
    if exists(seafile_conf_path):
        cp = ConfigParser()
        cp.read(seafile_conf_path)
        db_host = cp.get('database', 'host')    # → 'db'
        db_user = cp.get('database', 'user')    # → 'seafile'
        db_passwd = cp.get('database', 'password')
        db_port = int(cp.get('database', 'port'))

    while True:
        try:
            connection = pymysql.connect(host=db_host, port=db_port,
                                         user=db_user, passwd=db_passwd)
        except Exception as e:
            # prints "mysql is not ready" and sleeps

Critical insight: Once seafile.conf exists (from a prior successful install), wait_for_mysql() uses the seafile user — NOT root. If the MySQL volume was wiped, the seafile user doesn't exist, and this loop runs forever.

Why init_seafile_server() doesn't fix it

Even after manually recreating the seafile user and databases, restarting the container produces:

Skip running setup-seafile-mysql.py because there is existing seafile-data folder.

The bootstrap script (/scripts/bootstrap.py) skips DB initialization when seafile-data/ already exists. So the MySQL tables are never created, and fsck reports Table 'seafile_db.Repo' doesn't exist.

Recovery Procedure

Option A: Full PBS Restore of MySQL Volume (Preferred)

When a PBS backup exists, restoring the entire MySQL data directory is cleanest because it includes all users, grants, schemas, and data.

⚠️ Before attempting PBS restore, verify the backup is restorable (learned the hard way 2026-07-06):

  1. Check verification state in pvesh get output — if {"state":"failed"}, the backup has missing/corrupt chunks and may not restore.
  2. Check snapshot directory on PBS server for .didx vs .tmp_didx — only .didx snapshots are complete and restorable.
  3. Check for .bad chunk files on PBS server — if any chunk referenced by the backup is .bad (0 bytes), restore aborts immediately with no option to skip. Files dependent on that chunk are permanently lost.

See proxmox-ve-administration skill → references/pbs-lxc-setup-2026-07.md pitfalls #13–16 for PBS credential lookup, verification checking, corrupted chunk diagnosis, and incomplete backup detection.

If PBS restore fails due to corrupted chunks, fall back to Option B (manual recreation) and use seaf-fsck.sh --repair to rebuild repo metadata from the intact /opt/seafile-data/ file blocks.

Step 1: Locate the PBS backup

# On the PVE node hosting the CT (n5pro = 10.0.20.91):
pvesh get /nodes/$(hostname)/storage/noris_s3/content --content-type backup 2>&1 \
  | awk -F'│' '{print $3}' \
  | grep 'ct/111'
# → noris_s3:backup/ct/111/2026-07-02T21:00:00Z

Note: pvesh get outputs a Unicode box-drawing table, not JSON. Parse with awk -F'│' (Unicode pipe character U+2502). Column 3 contains the volid.

Step 2: Stop Seafile containers

# On the PVE node:
pct exec 111 -- docker stop seafile seafile-mysql seafile-memcached

Step 3: Rename broken MySQL volume

pct exec 111 -- mv /opt/seafile-mysql/db /opt/seafile-mysql/db.broken-$(date +%Y%m%d)

Step 4: Restore from PBS

Use pct restore with --storage pointing to the CT's root storage, then selectively extract only /opt/seafile-mysql/db/ from the restored backup.

Alternatively, use proxmox-backup-client to restore specific files:

# Map the PBS backup to a temp location
# PBS restore maps the CT filesystem
pct exec 111 -- bash -c '
  mkdir -p /tmp/restore
  # Use pb-client to restore just the mysql db path
  # (requires PBS credentials)
'

Step 5: Verify MySQL data

pct exec 111 -- docker start seafile-mysql
sleep 10
pct exec 111 -- docker exec seafile-mysql mysql -uroot -p'PASSWORD' \
  -e "SELECT user,host FROM mysql.user WHERE user='seafile'; SHOW DATABASES;"
# Should show: seafile@%, seafile_db, ccnet_db, seahub_db

Step 6: Start Seafile

pct exec 111 -- docker start seafile
sleep 30
pct exec 111 -- docker logs seafile --tail 20
# Should show: "Seafile server started", "Seahub is started"

Step 7: Run fsck

pct exec 111 -- docker exec seafile bash -lc \
  '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'

With restored DB, fsck should connect and check all repos.

Option B: Manual DB Init via docker run Setup Script (Preferred over SQL import)

When PBS restore fails and you need fresh DBs, the cleanest approach is running the official setup-seafile-mysql.sh in a throwaway container with proper env vars. This creates all schemas, users, and config files correctly.

Critical: The bootstrap.py in the main container skips init_seafile_server() when /shared/seafile/seafile-data/ exists. To force fresh DB init:

  1. Stop the seafile container
  2. Rename seafile-data temporarily: mv .../seafile-data .../seafile-data.bak
  3. Clear conf/: rm -rf .../conf/*
  4. Drop any half-created databases: DROP DATABASE ccnet_db, seafile_db, seahub_db
  5. Run setup in a throwaway container:
docker run --rm --name seafile-setup --network seafile-net \
  -e SERVER_NAME=seafile \
  -e SERVER_IP=cloud.familie-schoen.com \
  -e MYSQL_USER=seafile \
  -e MYSQL_USER_PASSWD='SEAFILE_PW' \
  -e MYSQL_USER_HOST=% \
  -e MYSQL_HOST=db \
  -e MYSQL_PORT=3306 \
  -e MYSQL_ROOT_PASSWD='ROOT_PW' \
  -e CCNET_DB=ccnet_db \
  -e SEAFILE_DB=seafile_db \
  -e SEAHUB_DB=seahub_db \
  -e SEAFILE_SERVER_HOSTNAME=cloud.familie-schoen.com \
  -e SEAFILE_SERVER_PROTOCOL=https \
  -e INIT_SEAFILE_ADMIN_EMAIL=admin@example.com \
  -e INIT_SEAFILE_ADMIN_PASSWORD=password \
  -e TIME_ZONE=Europe/Berlin \
  -v /opt/seafile-data:/shared \
  seafileltd/seafile-mc:13.0-latest \
  /opt/seafile/seafile-server-13.0.19/setup-seafile-mysql.sh auto -n seafile

⚠️ The env var names matter: MYSQL_ROOT_PASSWD (not MYSQL_ROOT_PASSWORD), MYSQL_USER_PASSWD (not MYSQL_USER_PASSWORD). The setup script reads get_param_val(args.mysql_root_passwd, 'MYSQL_ROOT_PASSWD') from os.environ.

  1. Restore original seafile-data and conf:
rm -rf .../seafile-data && mv .../seafile-data.bak .../seafile-data
# Restore conf from backup (contains correct DB credentials, SERVICE_URL, etc.)
cp -a .../conf.bak/* .../conf/
  1. Update SERVICE_URL in seahub_settings.py if hostname changed:
sed -i 's|old.domain.com|new.domain.com|g' .../conf/seahub_settings.py
  1. Start the container normally — it will run upgrade scripts (12.0→13.0 etc.) and start Seafile.

Option B2: Manual SQL Schema Import (Legacy Approach)

If docker run setup doesn't work, manually recreate databases and import schemas.

Step 1: Create databases and user

docker exec seafile-mysql mysql -uroot -p'ROOT_PW' -e "
  CREATE DATABASE IF NOT EXISTS ccnet_db CHARACTER SET utf8 COLLATE utf8_general_ci;
  CREATE DATABASE IF NOT EXISTS seafile_db CHARACTER SET utf8 COLLATE utf8_general_ci;
  CREATE DATABASE IF NOT EXISTS seahub_db CHARACTER SET utf8 COLLATE utf8_general_ci;
  CREATE USER 'seafile'@'%' IDENTIFIED BY 'SEAFILE_PW';
  GRANT ALL PRIVILEGES ON ccnet_db.* TO 'seafile'@'%';
  GRANT ALL PRIVILEGES ON seafile_db.* TO 'seafile'@'%';
  GRANT ALL PRIVILEGES ON seahub_db.* TO 'seafile'@'%';
  FLUSH PRIVILEGES;
"

Step 2: Import SQL schemas

The seafile container has no mysql client. Use the seafile-mysql container:

SQLDIR=/opt/seafile/seafile-server-12.0.14/sql/mysql

# ccnet schema
docker exec seafile-mysql mysql -uroot -p'ROOT_PW' ccnet_db \
  < <(docker exec seafile cat $SQLDIR/ccnet.sql)

# seafile schema
docker exec seafile-mysql mysql -uroot -p'ROOT_PW' seafile_db \
  < <(docker exec seafile cat $SQLDIR/seafile.sql)

# seahub schema (located in seahub/scripts/upgrade/sql/)
docker exec seafile-mysql mysql -uroot -p'ROOT_PW' seahub_db \
  < <(docker exec seafile cat /opt/seafile/seafile-server-12.0.14/seahub/scripts/upgrade/sql/12.0.0/mysql/seahub.sql)

⚠️ This only recreates empty tables. Existing repo metadata, users, and library mappings are lost. File blocks in /opt/seafile-data/ remain but are orphaned (no DB entries pointing to them). The admin account must be recreated via check_init_admin.py or the Seafile web UI.

Step 3: Restart and run fsck

docker restart seafile
sleep 30
docker exec seafile bash -lc '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'

Option C: Reconstruct Repo Registrations from Commit Storage

When DBs are freshly initialized but /opt/seafile-data/ (file blocks + commits) is intact, repos can be reconstructed by scanning the commit storage and registering them in the DB. See references/seafile-repo-reconstruction.md for the full procedure with Python scripts.

Overview of the technique:

  1. Scan commit storage: storage/commits/<repo_id>/<XX>/<YYYY...>/ — each file is a JSON object with commit_id, root_id, parent_id, second_parent_id, repo_name, repo_desc, ctime, creator_name.
  2. Find HEAD commit: The commit NOT referenced as parent_id by any other commit in the same repo. For repos with many commits (>1000), use find -printf '%T@ %p\n' | sort -rn | head -20 to read only the newest 20.
  3. Register in DB: Insert into Repo, Branch, RepoOwner, and RepoInfo tables. The Repo table only needs repo_id; Branch needs (name, repo_id, commit_id); RepoOwner needs (repo_id, owner_id); RepoInfo needs (repo_id, name, ...).
  4. Run seaf-fsck.sh --repair: After registering repos, fsck verifies integrity and repairs inconsistencies.

⚠️ Encrypted repo commit IDs may have suffixes (e.g., e155972728d6a3205726b94243078ae5f0aaaa3a.FYMWR3) that exceed the 40-char commit_id column. Truncate to 40 chars before inserting.

⚠️ seaf-fsck.sh --repair only checks registered repos — it does NOT discover or register new repos. All repos must be registered in the DB first.

⚠️ seaf-fsck.sh --export also only exports registered repos — it cannot recover unregistered repos from the filesystem.

Key Files in Seafile Docker Container

Path Purpose
/opt/seafile/conf/seafile.conf Main config — DB host, user, password, fileserver port
/opt/seafile/seafile-server-VERSION/ Server installation (scripts, SQL, binaries)
/opt/seafile/seafile-server-VERSION/seaf-fsck.sh Filesystem consistency checker
/opt/seafile/seafile-server-VERSION/seaf-gc.sh Garbage collector for orphaned blocks
/opt/seafile/seafile-server-VERSION/sql/mysql/ SQL schema files (ccnet.sql, seafile.sql)
/opt/seafile/seafile-server-VERSION/seahub/scripts/upgrade/sql/VERSION/mysql/seahub.sql Seahub DB schema
/scripts/start.py Container entrypoint — calls wait_for_mysql() + init_seafile_server()
/scripts/utils.py wait_for_mysql() implementation
/scripts/bootstrap.py init_seafile_server() — skips if seafile-data/ exists

Docker-Compose Layout (seafile-server.yml)

services:
  db:
    image: mariadb:10.11
    container_name: seafile-mysql
    volumes:
      - "${SEAFILE_MYSQL_VOLUME:-/opt/seafile-mysql/db}:/var/lib/mysql"
    # ...
  seafile:
    image: seafileltd/seafile-mc:12.0-latest
    container_name: seafile
    environment:
      - DB_HOST=db
      - DB_USER=seafile
      - DB_ROOT_PASSWD=${INIT_SEAFILE_MYSQL_ROOT_PASSWORD}
      - DB_PASSWORD=${SEAFILE_MYSQL_DB_PASSWORD}
    volumes:
      - "${SEAFILE_VOLUME:-/opt/seafile-data}:/shared"
    depends_on:
      db:
        condition: service_healthy

The seafile container's /shared mount corresponds to /opt/seafile-data on the host. Inside the container, /opt/seafile/ is symlinked from /shared/seafile/.

Network Architecture

seafile-net (bridge, 172.18.0.0/16):
  ├── seafile-mysql    172.18.0.3  (hostname: db)
  ├── seafile-memcached 172.18.0.2
  ├── seafile-caddy    172.18.0.4
  ├── seafile          172.18.0.5
  └── seadoc           172.18.0.6

seafile.conf references host = db which resolves via Docker DNS to the seafile-mysql container.

Pitfalls

  1. wait_for_mysql() uses seafile user, not root — Once seafile.conf exists, the startup script connects as the seafile MySQL user. If that user doesn't exist (volume wipe), the loop runs forever. Container shows "Up" but Seafile never starts.

  2. init_seafile_server() skips when seafile-data/ exists — Even after recreating the MySQL user and empty databases, the bootstrap won't create tables because it sees existing data. Must import SQL schemas manually or restore the MySQL volume from backup.

  3. seaf-fsck.sh exits 0 even on DB failure — It catches the DB connection error, prints a WARNING, and exits cleanly. Don't rely on exit code; check stderr for [WARNING] lines.

  4. No mysql client in seafile container — Use docker exec seafile-mysql mysql ... for all MySQL operations. The seafile container only has Python and Seafile binaries.

  5. seafile-data/ is large and usually intact — When recovering, don't restore /opt/seafile-data/ from backup unless corruption is suspected. It contains all file blocks (potentially hundreds of GB). Restoring only the MySQL volume + conf is much faster.

  6. PBS backup listing via pvesh — pvesh get /nodes/$(hostname)/storage/ <storage>/content outputs a Unicode table, not JSON. Parse with awk -F'│' to extract volids. See proxmox-ve-administration skill for PBS backup listing details.

  7. MariaDB vs MySQL image — Seafile Docker uses mariadb:10.11 but the config says type = mysql. MariaDB is compatible. The MARIADB_AUTO_UPGRADE=1 env var handles schema upgrades on container start.

  8. Image version change can wipe MySQL volume — Changing the Seafile Docker image (e.g., downgrading seafile-mc:13.0-latest → seafile-mc:12.0-latest or vice versa) can trigger MySQL volume reinitialization. The container detects a version mismatch and reinitializes the data directory, wiping all databases and users. Always backup the MySQL volume (/opt/seafile-mysql/db) before changing the image version in .env.

  9. Compare .env before restoring — When recovering, always diff the current .env against any backup config. Key fields that differ between versions:

    • SEAFILE_IMAGE (e.g., 13.0-latest vs 12.0-latest)
    • SEAFILE_SERVER_HOSTNAME (e.g., cloud.familie-schoen.com vs seafile.familie-schoen.com)
    • INIT_SEAFILE_ADMIN_EMAIL
    • JWT_PRIVATE_KEY
    • COMPOSE_FILE list (e.g., caddy.yml may be added/removed)

    Align the config BEFORE starting containers with a restored DB, otherwise Seafile may reinitialize or fail to match the restored data.

  10. Telegram file transfer corrupts large ZIPs — Sending large binary archives (e.g., a MySQL data directory ZIP) via Telegram can truncate or corrupt the file. The ZIP arrives with valid local file headers but no central directory and invalid size fields (0xFFFFFFFF). For transferring server data between PVE nodes, use scp, rsync, or PBS restore instead of Telegram file attachments. If a received ZIP fails to open, verify with python3 -c "import zipfile; zipfile.ZipFile('file.zip').namelist()" — if it raises BadZipFile, the transfer corrupted it.

  11. bootstrap.py ENV variable substitution bug — When bootstrap.py calls setup-seafile-mysql.py via subprocess.call(cmd, env={...}), the env= dict replaces the entire process environment rather than augmenting it. The env= dict includes MYSQL_ROOT_PASSWD (mapped from INIT_SEAFILE_MYSQL_ROOT_PASSWORD), but if the mapping is missing or misnamed, the subprocess gets MYSQL_ROOT_PASSWD='' (empty string), producing the error using password: NO from MySQL.

    Symptom: setup-seafile-mysql.py fails with Access denied for 'root'@'localhost' (using password: NO) even though INIT_SEAFILE_MYSQL_ROOT_PASSWORD is correctly set in the container env.

    Fix: Run setup-seafile-mysql.py manually inside the container with all required ENV vars explicitly exported:

    docker exec -e MYSQL_ROOT_PASSWD='<root_pw>' \
  -e MYSQL_USER='seafile' -e MYSQL_USER_PASSWD='<seafile_pw>' \
  -e MYSQL_HOST=db -e MYSQL_PORT=3306 \
  -e CCNET_DB=ccnet_db -e SEAFILE_DB=seafile_db -e SEAHUB_DB=seahub_db \
  -e SEAFILE_SERVER_HOSTNAME=cloud.example.com \
  -e INIT_SEAFILE_ADMIN_EMAIL=admin@example.com \
  -e INIT_SEAFILE_ADMIN_PASSWORD='<admin_pw>' \
  seafile /opt/seafile/seafile-server-13.0.19/setup-seafile-mysql.sh auto -n seafile

    This bypasses bootstrap.py's broken env= substitution and gives the setup script direct access to all required variables.

  12. PBS corrupted chunk blocks ALL file restoration — A single 0-byte .bad chunk file on the PBS server (destroyed by garbage collection) makes the ENTIRE backup unrestorable. PBS aborts on the first file referencing the bad chunk, cleans up, and exits. There is no --skip-bad-chunks option. Files alphabetically before the affected file MAY be partially restored, but anything after is lost. The chunk path format is /mnt/datastore/<ds>/.chunks/<XX>/<hash>.0.bad.

  13. Multiple backup formats can ALL be unusable — In this incident, THREE separate backup sources were tried and ALL failed:

    • PBS backup: corrupted chunk (above)
    • seafile-mysql.zip (Telegram-transferred): BadZipFile, no central directory
    • seafile-mysql.tar.zst: truncated, unexpected EOF after 4 entries
    • Offsite PBS: no CT111 backup existed at all Lesson: never assume ANY backup is restorable without verifying. Always test-restore before declaring a backup strategy viable.
Reference in New Issue View Git Blame Copy Permalink