hermes-skills/devops/seafile-api/references/seafile-docker-recovery.md

# Seafile Docker Recovery: MySQL Volume Loss & Selective PBS Restore

## Scenario

Seafile MC Docker edition on CT 111 (n5pro, 10.0.20.91). The `.env` was changed
(image downgraded from `seafile-mc:13.0-latest` to `seafile-mc:12.0-latest`,
hostname changed from `cloud.familie-schoen.com` to `seafile.familie-schoen.com`),
which triggered MySQL volume reinitialization — wiping all Seafile databases
(`seafile_db`, `ccnet_db`, `seahub_db`) and the `seafile` MySQL user. The file
block storage (`/opt/seafile-data/`) remained intact. A PBS backup of CT 111 from
2026-07-02 existed on `noris_s3` storage. The original `/opt/seafile/` config
directory was preserved as a ZIP by the user.

## Symptoms

### Symptom 1: Container "Up" but not actually serving

```bash
docker ps                    # → seafile Up 7 hours
docker logs seafile --tail 50 # → "waiting for mysql server to be ready: mysql is not ready" (repeating forever)
```

The container's entrypoint (`/scripts/start.py`) calls `wait_for_mysql()` before
proceeding to `init_seafile_server()`. If MySQL connection fails, it loops forever.
The container stays "Up" (PID 1 alive) but Seafile never starts.

### Symptom 2: seaf-fsck warns about DB connection

```bash
docker exec seafile bash -lc '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'
# → [WARNING] Failed to connect to MySQL: Access denied for user 'seafile'@'172.18.0.5'
# → seaf-fsck run done
# → Done.
```

Fsck exits 0 but didn't actually check anything — the DB connection failed.

### Symptom 3: MySQL has no Seafile databases

```bash
docker exec seafile-mysql mysql -uroot -p'PASSWORD' -e 'SHOW DATABASES;'
# → information_schema, mysql, performance_schema, sys  (NO seafile_db, ccnet_db, seahub_db)
```

### Symptom 4: No seafile MySQL user

```bash
docker exec seafile-mysql mysql -uroot -p'PASSWORD' \
  -e "SELECT user,host FROM mysql.user WHERE user LIKE '%sea%';"
# → Empty set
```

## Root Cause Analysis

### How `wait_for_mysql()` works

File: `/scripts/utils.py` inside the seafile container:

```python
def wait_for_mysql():
    db_host = get_conf('DB_HOST', '127.0.0.1')
    db_user = 'root'
    db_passwd = get_conf('DB_ROOT_PASSWD', '')

    # BUT: if seafile.conf exists, it overrides with the seafile user!
    seafile_conf_path = join(topdir, 'conf', 'seafile.conf')
    if exists(seafile_conf_path):
        cp = ConfigParser()
        cp.read(seafile_conf_path)
        db_host = cp.get('database', 'host')    # → 'db'
        db_user = cp.get('database', 'user')    # → 'seafile'
        db_passwd = cp.get('database', 'password')
        db_port = int(cp.get('database', 'port'))

    while True:
        try:
            connection = pymysql.connect(host=db_host, port=db_port,
                                         user=db_user, passwd=db_passwd)
        except Exception as e:
            # prints "mysql is not ready" and sleeps
```

**Critical insight**: Once `seafile.conf` exists (from a prior successful install),
`wait_for_mysql()` uses the `seafile` user — NOT root. If the MySQL volume was
wiped, the `seafile` user doesn't exist, and this loop runs forever.

### Why `init_seafile_server()` doesn't fix it

Even after manually recreating the `seafile` user and databases, restarting the
container produces:

```
Skip running setup-seafile-mysql.py because there is existing seafile-data folder.
```

The bootstrap script (`/scripts/bootstrap.py`) skips DB initialization when
`seafile-data/` already exists. So the MySQL tables are never created, and fsck
reports `Table 'seafile_db.Repo' doesn't exist`.

## Recovery Procedure

### Option A: Full PBS Restore of MySQL Volume (Preferred)

When a PBS backup exists, restoring the entire MySQL data directory is cleanest
because it includes all users, grants, schemas, and data.

⚠️ **Before attempting PBS restore, verify the backup is restorable** (learned
the hard way 2026-07-06):

1. Check verification state in `pvesh get` output — if `{"state":"failed"}`,
   the backup has missing/corrupt chunks and may not restore.
2. Check snapshot directory on PBS server for `.didx` vs `.tmp_didx` —
   only `.didx` snapshots are complete and restorable.
3. Check for `.bad` chunk files on PBS server — if any chunk referenced by
   the backup is `.bad` (0 bytes), restore aborts immediately with no option
   to skip. Files dependent on that chunk are permanently lost.

See `proxmox-ve-administration` skill → `references/pbs-lxc-setup-2026-07.md`
pitfalls #13–16 for PBS credential lookup, verification checking, corrupted
chunk diagnosis, and incomplete backup detection.

If PBS restore fails due to corrupted chunks, fall back to Option B (manual
recreation) and use `seaf-fsck.sh --repair` to rebuild repo metadata from
the intact `/opt/seafile-data/` file blocks.

#### Step 1: Locate the PBS backup

```bash
# On the PVE node hosting the CT (n5pro = 10.0.20.91):
pvesh get /nodes/$(hostname)/storage/noris_s3/content --content-type backup 2>&1 \
  | awk -F'│' '{print $3}' \
  | grep 'ct/111'
# → noris_s3:backup/ct/111/2026-07-02T21:00:00Z
```

**Note**: `pvesh get` outputs a Unicode box-drawing table, not JSON. Parse with
`awk -F'│'` (Unicode pipe character U+2502). Column 3 contains the volid.

#### Step 2: Stop Seafile containers

```bash
# On the PVE node:
pct exec 111 -- docker stop seafile seafile-mysql seafile-memcached
```

#### Step 3: Rename broken MySQL volume

```bash
pct exec 111 -- mv /opt/seafile-mysql/db /opt/seafile-mysql/db.broken-$(date +%Y%m%d)
```

#### Step 4: Restore from PBS

Use `pct restore` with `--storage` pointing to the CT's root storage, then
selectively extract only `/opt/seafile-mysql/db/` from the restored backup.

Alternatively, use `proxmox-backup-client` to restore specific files:

```bash
# Map the PBS backup to a temp location
# PBS restore maps the CT filesystem
pct exec 111 -- bash -c '
  mkdir -p /tmp/restore
  # Use pb-client to restore just the mysql db path
  # (requires PBS credentials)
'
```

#### Step 5: Verify MySQL data

```bash
pct exec 111 -- docker start seafile-mysql
sleep 10
pct exec 111 -- docker exec seafile-mysql mysql -uroot -p'PASSWORD' \
  -e "SELECT user,host FROM mysql.user WHERE user='seafile'; SHOW DATABASES;"
# Should show: seafile@%, seafile_db, ccnet_db, seahub_db
```

#### Step 6: Start Seafile

```bash
pct exec 111 -- docker start seafile
sleep 30
pct exec 111 -- docker logs seafile --tail 20
# Should show: "Seafile server started", "Seahub is started"
```

#### Step 7: Run fsck

```bash
pct exec 111 -- docker exec seafile bash -lc \
  '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'
```

With restored DB, fsck should connect and check all repos.

### Option B: Manual DB Init via `docker run` Setup Script (Preferred over SQL import)

When PBS restore fails and you need fresh DBs, the cleanest approach is running
the official `setup-seafile-mysql.sh` in a throwaway container with proper env
vars. This creates all schemas, users, and config files correctly.

**Critical**: The `bootstrap.py` in the main container skips `init_seafile_server()`
when `/shared/seafile/seafile-data/` exists. To force fresh DB init:

1. Stop the seafile container
2. Rename `seafile-data` temporarily: `mv .../seafile-data .../seafile-data.bak`
3. Clear `conf/`: `rm -rf .../conf/*`
4. Drop any half-created databases: `DROP DATABASE ccnet_db, seafile_db, seahub_db`
5. Run setup in a throwaway container:

```bash
docker run --rm --name seafile-setup --network seafile-net \
  -e SERVER_NAME=seafile \
  -e SERVER_IP=cloud.familie-schoen.com \
  -e MYSQL_USER=seafile \
  -e MYSQL_USER_PASSWD='SEAFILE_PW' \
  -e MYSQL_USER_HOST=% \
  -e MYSQL_HOST=db \
  -e MYSQL_PORT=3306 \
  -e MYSQL_ROOT_PASSWD='ROOT_PW' \
  -e CCNET_DB=ccnet_db \
  -e SEAFILE_DB=seafile_db \
  -e SEAHUB_DB=seahub_db \
  -e SEAFILE_SERVER_HOSTNAME=cloud.familie-schoen.com \
  -e SEAFILE_SERVER_PROTOCOL=https \
  -e INIT_SEAFILE_ADMIN_EMAIL=admin@example.com \
  -e INIT_SEAFILE_ADMIN_PASSWORD=password \
  -e TIME_ZONE=Europe/Berlin \
  -v /opt/seafile-data:/shared \
  seafileltd/seafile-mc:13.0-latest \
  /opt/seafile/seafile-server-13.0.19/setup-seafile-mysql.sh auto -n seafile
```

⚠️ The env var names matter: `MYSQL_ROOT_PASSWD` (not `MYSQL_ROOT_PASSWORD`),
`MYSQL_USER_PASSWD` (not `MYSQL_USER_PASSWORD`). The setup script reads
`get_param_val(args.mysql_root_passwd, 'MYSQL_ROOT_PASSWD')` from `os.environ`.

6. Restore original `seafile-data` and `conf`:
```bash
rm -rf .../seafile-data && mv .../seafile-data.bak .../seafile-data
# Restore conf from backup (contains correct DB credentials, SERVICE_URL, etc.)
cp -a .../conf.bak/* .../conf/
```

7. Update `SERVICE_URL` in `seahub_settings.py` if hostname changed:
```bash
sed -i 's|old.domain.com|new.domain.com|g' .../conf/seahub_settings.py
```

8. Start the container normally — it will run upgrade scripts (12.0→13.0 etc.)
   and start Seafile.

### Option B2: Manual SQL Schema Import (Legacy Approach)

If `docker run` setup doesn't work, manually recreate databases and import schemas.

#### Step 1: Create databases and user

```bash
docker exec seafile-mysql mysql -uroot -p'ROOT_PW' -e "
  CREATE DATABASE IF NOT EXISTS ccnet_db CHARACTER SET utf8 COLLATE utf8_general_ci;
  CREATE DATABASE IF NOT EXISTS seafile_db CHARACTER SET utf8 COLLATE utf8_general_ci;
  CREATE DATABASE IF NOT EXISTS seahub_db CHARACTER SET utf8 COLLATE utf8_general_ci;
  CREATE USER 'seafile'@'%' IDENTIFIED BY 'SEAFILE_PW';
  GRANT ALL PRIVILEGES ON ccnet_db.* TO 'seafile'@'%';
  GRANT ALL PRIVILEGES ON seafile_db.* TO 'seafile'@'%';
  GRANT ALL PRIVILEGES ON seahub_db.* TO 'seafile'@'%';
  FLUSH PRIVILEGES;
"
```

#### Step 2: Import SQL schemas

The seafile container has no `mysql` client. Use the seafile-mysql container:

```bash
SQLDIR=/opt/seafile/seafile-server-12.0.14/sql/mysql

# ccnet schema
docker exec seafile-mysql mysql -uroot -p'ROOT_PW' ccnet_db \
  < <(docker exec seafile cat $SQLDIR/ccnet.sql)

# seafile schema
docker exec seafile-mysql mysql -uroot -p'ROOT_PW' seafile_db \
  < <(docker exec seafile cat $SQLDIR/seafile.sql)

# seahub schema (located in seahub/scripts/upgrade/sql/)
docker exec seafile-mysql mysql -uroot -p'ROOT_PW' seahub_db \
  < <(docker exec seafile cat /opt/seafile/seafile-server-12.0.14/seahub/scripts/upgrade/sql/12.0.0/mysql/seahub.sql)
```

⚠️ **This only recreates empty tables.** Existing repo metadata, users, and
library mappings are lost. File blocks in `/opt/seafile-data/` remain but are
orphaned (no DB entries pointing to them). The admin account must be recreated
via `check_init_admin.py` or the Seafile web UI.

#### Step 3: Restart and run fsck

```bash
docker restart seafile
sleep 30
docker exec seafile bash -lc '/opt/seafile/seafile-server-12.0.14/seaf-fsck.sh'
```

### Option C: Reconstruct Repo Registrations from Commit Storage

When DBs are freshly initialized but `/opt/seafile-data/` (file blocks + commits)
is intact, repos can be reconstructed by scanning the commit storage and
registering them in the DB. See `references/seafile-repo-reconstruction.md` for
the full procedure with Python scripts.

**Overview of the technique:**

1. **Scan commit storage**: `storage/commits/<repo_id>/<XX>/<YYYY...>/` — each
   file is a JSON object with `commit_id`, `root_id`, `parent_id`,
   `second_parent_id`, `repo_name`, `repo_desc`, `ctime`, `creator_name`.
2. **Find HEAD commit**: The commit NOT referenced as `parent_id` by any other
   commit in the same repo. For repos with many commits (>1000), use
   `find -printf '%T@ %p\n' | sort -rn | head -20` to read only the newest 20.
3. **Register in DB**: Insert into `Repo`, `Branch`, `RepoOwner`, and `RepoInfo`
   tables. The `Repo` table only needs `repo_id`; `Branch` needs `(name, repo_id,
   commit_id)`; `RepoOwner` needs `(repo_id, owner_id)`; `RepoInfo` needs
   `(repo_id, name, ...)`.
4. **Run `seaf-fsck.sh --repair`**: After registering repos, fsck verifies
   integrity and repairs inconsistencies.

⚠️ **Encrypted repo commit IDs may have suffixes** (e.g.,
`e155972728d6a3205726b94243078ae5f0aaaa3a.FYMWR3`) that exceed the 40-char
`commit_id` column. Truncate to 40 chars before inserting.

⚠️ **`seaf-fsck.sh --repair` only checks registered repos** — it does NOT discover
or register new repos. All repos must be registered in the DB first.

⚠️ **`seaf-fsck.sh --export` also only exports registered repos** — it cannot
recover unregistered repos from the filesystem.

## Key Files in Seafile Docker Container

| Path | Purpose |
|------|---------|
| `/opt/seafile/conf/seafile.conf` | Main config — DB host, user, password, fileserver port |
| `/opt/seafile/seafile-server-VERSION/` | Server installation (scripts, SQL, binaries) |
| `/opt/seafile/seafile-server-VERSION/seaf-fsck.sh` | Filesystem consistency checker |
| `/opt/seafile/seafile-server-VERSION/seaf-gc.sh` | Garbage collector for orphaned blocks |
| `/opt/seafile/seafile-server-VERSION/sql/mysql/` | SQL schema files (ccnet.sql, seafile.sql) |
| `/opt/seafile/seafile-server-VERSION/seahub/scripts/upgrade/sql/VERSION/mysql/seahub.sql` | Seahub DB schema |
| `/scripts/start.py` | Container entrypoint — calls `wait_for_mysql()` + `init_seafile_server()` |
| `/scripts/utils.py` | `wait_for_mysql()` implementation |
| `/scripts/bootstrap.py` | `init_seafile_server()` — skips if `seafile-data/` exists |

## Docker-Compose Layout (seafile-server.yml)

```yaml
services:
  db:
    image: mariadb:10.11
    container_name: seafile-mysql
    volumes:
      - "${SEAFILE_MYSQL_VOLUME:-/opt/seafile-mysql/db}:/var/lib/mysql"
    # ...
  seafile:
    image: seafileltd/seafile-mc:12.0-latest
    container_name: seafile
    environment:
      - DB_HOST=db
      - DB_USER=seafile
      - DB_ROOT_PASSWD=${INIT_SEAFILE_MYSQL_ROOT_PASSWORD}
      - DB_PASSWORD=${SEAFILE_MYSQL_DB_PASSWORD}
    volumes:
      - "${SEAFILE_VOLUME:-/opt/seafile-data}:/shared"
    depends_on:
      db:
        condition: service_healthy
```

The `seafile` container's `/shared` mount corresponds to `/opt/seafile-data` on the
host. Inside the container, `/opt/seafile/` is symlinked from `/shared/seafile/`.

## Network Architecture

```
seafile-net (bridge, 172.18.0.0/16):
  ├── seafile-mysql    172.18.0.3  (hostname: db)
  ├── seafile-memcached 172.18.0.2
  ├── seafile-caddy    172.18.0.4
  ├── seafile          172.18.0.5
  └── seadoc           172.18.0.6
```

`seafile.conf` references `host = db` which resolves via Docker DNS to the
`seafile-mysql` container.

## Pitfalls

1. **`wait_for_mysql()` uses `seafile` user, not `root`** — Once `seafile.conf`
   exists, the startup script connects as the `seafile` MySQL user. If that user
   doesn't exist (volume wipe), the loop runs forever. Container shows "Up" but
   Seafile never starts.

2. **`init_seafile_server()` skips when `seafile-data/` exists** — Even after
   recreating the MySQL user and empty databases, the bootstrap won't create
   tables because it sees existing data. Must import SQL schemas manually or
   restore the MySQL volume from backup.

3. **`seaf-fsck.sh` exits 0 even on DB failure** — It catches the DB connection
   error, prints a WARNING, and exits cleanly. Don't rely on exit code; check
   stderr for `[WARNING]` lines.

4. **No `mysql` client in seafile container** — Use `docker exec seafile-mysql
   mysql ...` for all MySQL operations. The seafile container only has Python
   and Seafile binaries.

5. **`seafile-data/` is large and usually intact** — When recovering, don't
   restore `/opt/seafile-data/` from backup unless corruption is suspected. It
   contains all file blocks (potentially hundreds of GB). Restoring only the
   MySQL volume + conf is much faster.

6. **PBS backup listing via `pvesh`** — `pvesh get /nodes/$(hostname)/storage/
   <storage>/content` outputs a Unicode table, not JSON. Parse with
   `awk -F'│'` to extract volids. See `proxmox-ve-administration` skill for
   PBS backup listing details.

7. **MariaDB vs MySQL image** — Seafile Docker uses `mariadb:10.11` but the
   config says `type = mysql`. MariaDB is compatible. The `MARIADB_AUTO_UPGRADE=1`
   env var handles schema upgrades on container start.

8. **Image version change can wipe MySQL volume** — Changing the Seafile Docker
   image (e.g., downgrading `seafile-mc:13.0-latest` → `seafile-mc:12.0-latest`
   or vice versa) can trigger MySQL volume reinitialization. The container detects
   a version mismatch and reinitializes the data directory, wiping all databases
   and users. **Always backup the MySQL volume (`/opt/seafile-mysql/db`) before
   changing the image version in `.env`.**

9. **Compare `.env` before restoring** — When recovering, always diff the current
   `.env` against any backup config. Key fields that differ between versions:
   - `SEAFILE_IMAGE` (e.g., `13.0-latest` vs `12.0-latest`)
   - `SEAFILE_SERVER_HOSTNAME` (e.g., `cloud.familie-schoen.com` vs
     `seafile.familie-schoen.com`)
   - `INIT_SEAFILE_ADMIN_EMAIL`
   - `JWT_PRIVATE_KEY`
   - `COMPOSE_FILE` list (e.g., `caddy.yml` may be added/removed)

   Align the config BEFORE starting containers with a restored DB, otherwise
   Seafile may reinitialize or fail to match the restored data.

10. **Telegram file transfer corrupts large ZIPs** — Sending large binary archives
    (e.g., a MySQL data directory ZIP) via Telegram can truncate or corrupt the
    file. The ZIP arrives with valid local file headers but no central directory
    and invalid size fields (`0xFFFFFFFF`). For transferring server data between
    PVE nodes, use `scp`, `rsync`, or PBS restore instead of Telegram file
    attachments. If a received ZIP fails to open, verify with
    `python3 -c "import zipfile; zipfile.ZipFile('file.zip').namelist()"` — if it
    raises `BadZipFile`, the transfer corrupted it.

11. **`bootstrap.py` ENV variable substitution bug** — When `bootstrap.py`
    calls `setup-seafile-mysql.py` via `subprocess.call(cmd, env={...})`,
    the `env=` dict **replaces** the entire process environment rather than
    augmenting it. The `env=` dict includes `MYSQL_ROOT_PASSWD` (mapped from
    `INIT_SEAFILE_MYSQL_ROOT_PASSWORD`), but if the mapping is missing or
    misnamed, the subprocess gets `MYSQL_ROOT_PASSWD=''` (empty string),
    producing the error `using password: NO` from MySQL.

    **Symptom**: `setup-seafile-mysql.py` fails with `Access denied for
    'root'@'localhost' (using password: NO)` even though
    `INIT_SEAFILE_MYSQL_ROOT_PASSWORD` is correctly set in the container env.

    **Fix**: Run `setup-seafile-mysql.py` manually inside the container with
    all required ENV vars explicitly exported:
    ```bash
    docker exec -e MYSQL_ROOT_PASSWD='<root_pw>' \
      -e MYSQL_USER='seafile' -e MYSQL_USER_PASSWD='<seafile_pw>' \
      -e MYSQL_HOST=db -e MYSQL_PORT=3306 \
      -e CCNET_DB=ccnet_db -e SEAFILE_DB=seafile_db -e SEAHUB_DB=seahub_db \
      -e SEAFILE_SERVER_HOSTNAME=cloud.example.com \
      -e INIT_SEAFILE_ADMIN_EMAIL=admin@example.com \
      -e INIT_SEAFILE_ADMIN_PASSWORD='<admin_pw>' \
      seafile /opt/seafile/seafile-server-13.0.19/setup-seafile-mysql.sh auto -n seafile
    ```
    This bypasses `bootstrap.py`'s broken `env=` substitution and gives the
    setup script direct access to all required variables.

12. **PBS corrupted chunk blocks ALL file restoration** — A single 0-byte `.bad`
    chunk file on the PBS server (destroyed by garbage collection) makes the
    ENTIRE backup unrestorable. PBS aborts on the first file referencing the
    bad chunk, cleans up, and exits. There is no `--skip-bad-chunks` option.
    Files alphabetically before the affected file MAY be partially restored,
    but anything after is lost. The chunk path format is
    `/mnt/datastore/<ds>/.chunks/<XX>/<hash>.0.bad`.

13. **Multiple backup formats can ALL be unusable** — In this incident, THREE
    separate backup sources were tried and ALL failed:
    - PBS backup: corrupted chunk (above)
    - `seafile-mysql.zip` (Telegram-transferred): BadZipFile, no central directory
    - `seafile-mysql.tar.zst`: truncated, unexpected EOF after 4 entries
    - Offsite PBS: no CT111 backup existed at all
    Lesson: never assume ANY backup is restorable without verifying. Always
    test-restore before declaring a backup strategy viable.