Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
@@ -0,0 +1,207 @@
|
||||
# Docker Analysis: 10.0.30.100 (2026-07-04)
|
||||
|
||||
## Host Info
|
||||
- **Host**: ubuntu (10.0.30.100)
|
||||
- **Storage Driver**: ZFS on `pool01_n2_redundant/docker`
|
||||
- **Docker Root**: `/var/lib/docker`
|
||||
- **Dataset USED**: 811 GB (includes snapshot overhead)
|
||||
- **Dataset REFER**: 491 GB (live data)
|
||||
- **Snapshot Overhead**: ~320 GB (deleted layers held by ZFS snapshots)
|
||||
- **ZFS Subdatasets**: 735 (was 744 before 2 paperless tags removed)
|
||||
- **Images**: 80 (before partial cleanup)
|
||||
- **Containers**: 5 running (47 config dirs total — many dead/stopped)
|
||||
- **Volumes**: 117 named + anonymous
|
||||
|
||||
## Daemon State: PARTIALLY RECOVERED
|
||||
|
||||
Docker daemon was `active` but effectively non-functional:
|
||||
- `docker system df` crashed: `error getting build cache usage: failed to get usage for HASH: layer does not exist`
|
||||
- `docker ps` sometimes worked (intermittently), sometimes timed out
|
||||
- `docker images --format` returned empty output with exit 0 (silent failure)
|
||||
- `docker system prune` failed: `a prune operation is already running`
|
||||
- `docker rmi` for some images worked (2 paperless tags removed), most timed out
|
||||
- `journalctl -u docker` showed: `error creating zfs mount: mount pool/docker/HASH: no such file or directory` and `Error unmounting /var/lib/docker/zfs/graph/HASH: invalid argument`
|
||||
|
||||
## Recovery Attempt & Outcome
|
||||
|
||||
1. `systemctl restart docker` — hung in `deactivating` for 60s, then `stop-sigterm` timeout, SIGKILL
|
||||
2. `dockerd` became zombie (`<defunct>`)
|
||||
3. Killed leftover `containerd-shim-runc-v2` process
|
||||
4. `systemctl restart containerd` — succeeded
|
||||
5. `systemctl start docker` — entered `activating` state
|
||||
6. After 2-3 minutes, daemon reached `active` (ZFS graphdriver scanned 735 subdatasets)
|
||||
7. After restart: `docker ps` worked (5 containers up), but `docker images` returned empty and `docker system df` timed out
|
||||
8. Triggered `docker image prune -a` — spawned `zfs destroy -r` for each layer, each entering D-state due to I/O contention with concurrent rsync
|
||||
9. `zfs destroy` processes hung in D-state (uninterruptible) — blocked all subsequent Docker operations until each completed sequentially
|
||||
10. Daemon remained `active` throughout — `docker ps` continued to work, but `docker images` stayed empty and `docker system df` timed out
|
||||
|
||||
**Outcome**: Daemon IS up and serving containers, but graphdriver is degraded. `docker ps` works; `docker images` returns empty; `docker system df` times out. The `zfs destroy` cascade from `image prune -a` is still running sequentially in the background.
|
||||
|
||||
## `zfs destroy` D-State Under I/O Contention
|
||||
|
||||
When Docker's ZFS graphdriver cleans up layers, it spawns `zfs destroy -r pool/docker/<hash>` for each removed layer. Under heavy pool I/O (e.g. concurrent rsync reading 300+ GB from the same ZFS pool), each `zfs destroy` enters **D-state (uninterruptible sleep)** and can take minutes per layer.
|
||||
|
||||
**Symptoms:**
|
||||
- `ps aux | grep "zfs destroy"` shows process in `D` state
|
||||
- Docker CLI commands (`docker images`, `docker system df`) time out while destroy runs
|
||||
- `zpool iostat` shows high read throughput (rsync) but near-zero writes (destroy blocked)
|
||||
- The dataset being destroyed may already report "does not exist" in `zfs list` — the destroy is stuck in kernel metadata flush, not actual data deletion
|
||||
|
||||
**Key insight:** This is NOT a permanent hang — `zfs destroy` eventually completes, but each one is serialized and I/O-starved. Docker appears broken during this window but is actually just waiting for ZFS to finish cleaning up.
|
||||
|
||||
**Recovery:** Patience. Don't kill the `zfs destroy` process (D-state is immune to SIGKILL anyway). Wait for I/O contention to ease (e.g. let rsync finish), then Docker operations resume. If urgent, reduce concurrent I/O on the pool.
|
||||
|
||||
## Root Cause
|
||||
|
||||
735 ZFS subdatasets for Docker layers, several with corrupt ZFS mounts (`no such file or directory`). The ZFS graphdriver tries to mount each subdataset during daemon startup; corrupt ones cause indefinite hangs. The daemon cannot complete initialization.
|
||||
|
||||
## Complete Volume Size Inventory
|
||||
|
||||
### Named Volumes (all significant sizes)
|
||||
|
||||
| Volume | Size | Purpose |
|
||||
|--------|------|---------|
|
||||
| embyconfig | 860 MB | Emby Media Server config |
|
||||
| jellyfin_jellyfin_config | 314 MB | Jellyfin config |
|
||||
| monitoring_prometheus_data | 146 MB | Prometheus metrics |
|
||||
| matomo | 134 MB | Matomo analytics |
|
||||
| monitoring_grafana_db | 61 MB | Grafana dashboards |
|
||||
| ghost-imkerei | 43 MB | Ghost blog (Imkerei) |
|
||||
| jellyfin_jellyfin_cache | 18 MB | Jellyfin cache |
|
||||
| grafana | 7.2 MB | Grafana (legacy?) |
|
||||
| handbrake_config | 3 MB | Handbrake config |
|
||||
| openproject_openproject_assets | 912 KB | OpenProject |
|
||||
| monitoring_alertmanager | 14 KB | Alertmanager config |
|
||||
| ethereum | 14 KB | Ethereum node |
|
||||
| monitoring_prometheus_alerts | 13 KB | Prometheus alerts |
|
||||
| monitoring_promtail_config | 7 KB | Promtail config |
|
||||
| monitoring_loki_config | 7 KB | Loki config |
|
||||
| bitmonero | 180 KB | Monero miner |
|
||||
| bienen_proxy | 59 KB | nginx proxy (Imkerei) |
|
||||
| ghost_proxy | 59 KB | nginx proxy (Ghost) |
|
||||
|
||||
**Total named volumes: ~1.5 GB.** Volumes are NOT the space problem.
|
||||
|
||||
### Anonymous Volumes (hash names)
|
||||
~100 volumes, all < 1 MB each. Total < 5 MB. Negligible.
|
||||
|
||||
### Container Config Dirs
|
||||
47 directories à ~100-200 KB (logs/metadata). Total ~5 MB.
|
||||
|
||||
### Where the 811 GB Actually Lives
|
||||
|
||||
| Area | Size | % of total |
|
||||
|------|------|------------|
|
||||
| ZFS Image Layers (735 subdatasets) | ~491 GB | 60% |
|
||||
| Snapshot/Clone Overhead | ~320 GB | 40% |
|
||||
| Named Volumes | ~1.5 GB | <1% |
|
||||
| Anonymous Volumes | ~5 MB | ~0% |
|
||||
| Container Configs | ~5 MB | ~0% |
|
||||
|
||||
## Seafile Stack Inventory
|
||||
|
||||
### Docker Volumes (11 total!)
|
||||
|
||||
| Volume | Content | Notes |
|
||||
|--------|---------|-------|
|
||||
| seafile_data | logs, nginx, seafile core | Main Seafile data |
|
||||
| seafile_custom_data | commits, fs, storage, thumbnails, avatars | **Actual file storage** |
|
||||
| seafile-conf | Seafile config | |
|
||||
| seafile_custom_conf | Custom config | |
|
||||
| seafile_custom_proxy_conf | Proxy config | |
|
||||
| seafile_custom_mariadb | MariaDB data (22 entries) | Active DB? |
|
||||
| seafile-mysql | MySQL data (28 entries) | Active DB? |
|
||||
| seafile_mysql | MySQL data (24 entries) | Another DB instance |
|
||||
| seafile-mysql.crashed | Crashed MySQL (50 entries) | ⚠️ Old crash, never cleaned |
|
||||
| seafile_seafile-mysql-2023 | 2023 backup MySQL (16 entries) | Old backup |
|
||||
|
||||
**3 MySQL/MariaDB instances** + 1 crashed + 1 old 2023 backup = 5 database volumes for one Seafile installation.
|
||||
|
||||
### Legacy Seafile (non-Docker)
|
||||
- `/pool01_n2_redundant/homes/dominik/seafile-server/` — Original 2014 Seafile install (src + seahub only, no data)
|
||||
|
||||
### Seafile Container Status
|
||||
NOT running. No Seafile containers in `docker ps`.
|
||||
|
||||
## Mailserver Stack Inventory
|
||||
|
||||
### Docker Volume: `mailserver`
|
||||
| Subdir | Content | Notes |
|
||||
|--------|---------|-------|
|
||||
| mail/ | Postfächer (UID 1024, since 2018) | Actual mail data |
|
||||
| mysql/ | Mailserver MySQL DB | |
|
||||
| mysql_bak/ | MySQL backup | |
|
||||
| rainloop/ | Rainloop webmail | UID 991 |
|
||||
| redis/ | Redis cache | |
|
||||
| traefik/ | Traefik config | |
|
||||
|
||||
### ZFS Maildirs
|
||||
| Path | Size | Owner |
|
||||
|------|------|-------|
|
||||
| `/pool01_n2_redundant/homes/dominik/.Maildir` | 540 MB | Dominik |
|
||||
| `/pool01_n2_redundant/homes/sarah/.Maildir` | 2 GB | Sarah |
|
||||
|
||||
### Mailserver Container Status
|
||||
NOT running. No mailserver container in `docker ps`.
|
||||
|
||||
## ZFS Homes Overview
|
||||
|
||||
| Dataset | USED | REFER |
|
||||
|---------|------|-------|
|
||||
| pool01_n2_redundant/homes | 456 GB | 455 GB |
|
||||
|
||||
Homes contain: dominik (72 entries), sarah (31 entries). Includes .Maildir, seafile-server legacy, and many personal directories.
|
||||
|
||||
## Running Containers (after recovery)
|
||||
|
||||
| Name | Image | Status |
|
||||
|------|-------|--------|
|
||||
| Traefik | traefik:latest | Up |
|
||||
| website-imkerei-bienen-proxy-1 | nginx:alpine | Up |
|
||||
| website-imkerei-proxy-1 | nginx:alpine | Up |
|
||||
| website-imkerei-website-1 | ghost:5.3 | Up |
|
||||
| portainer | portainer/portainer-ee:lts | Up |
|
||||
|
||||
spoolman-spoolman-1 was stuck in "Removal In Progress" — cleared after daemon restart.
|
||||
|
||||
## Top Images by Size (before crash)
|
||||
|
||||
| Size | Image |
|
||||
|------|-------|
|
||||
| 4.08 GB | exadel/compreface-core:1.2.0 |
|
||||
| 2.2 GB | koush/scrypted:latest |
|
||||
| 1.98 GB | home-assistant:stable |
|
||||
| 1.67 GB | gotenberg/gotenberg:8.9 |
|
||||
| 1.55 GB | thecodingmachine/gotenberg:latest |
|
||||
| 1.54 GB | homegear/homegear:stable |
|
||||
| 1.38 GB | seafile-mc:11.0-latest |
|
||||
| 1.37 GB | seafile-mc:latest |
|
||||
| 1.37 GB | paperless-ngx:2.17.1 |
|
||||
| 1.36 GB | paperless-ngx:2.15.1 |
|
||||
| 1.29 GB | immich-machine-learning:openvino |
|
||||
| 1.26 GB | iobroker:latest |
|
||||
| 1.09 GB | spoolman:latest |
|
||||
| 1.07 GB | double-take:latest |
|
||||
| 1.04 GB | jellyfin:latest |
|
||||
|
||||
## Duplicate Images Identified
|
||||
|
||||
| Image | Versions | Est. Waste |
|
||||
|-------|----------|------------|
|
||||
| paperless-ngx | 2.9.0, 2.12, 2.12.0, 2.15.1, 2.17.1 + 2× ghcr | ~6 GB |
|
||||
| seafile-mc | 11.0-latest, latest | ~2.7 GB |
|
||||
| gotenberg | 8.9, latest | ~3.2 GB |
|
||||
| portainer-ee | lts, 2.21.5, `<none>` | ~0.7 GB |
|
||||
| memcached | 1.6, latest, `<none>` | ~0.25 GB |
|
||||
| postgres | 13, 16 | ~0.85 GB |
|
||||
| Dangling | 3× `<none>` | ~0.3 GB |
|
||||
|
||||
## Recommendation
|
||||
|
||||
Docker ZFS storage driver is irreparably corrupt. Options:
|
||||
1. **Rebuild (recommended)**: Stop docker, rename `/var/lib/docker`, restart with `overlay2` on a separate ext4/xfs volume, redeploy containers. Volumes on ZFS survive.
|
||||
2. **Manual ZFS cleanup**: Find and `zfs destroy` corrupt subdatasets. Risky, time-consuming, may not fix all issues.
|
||||
|
||||
## All Unique Repos (60)
|
||||
|
||||
alpine, apache/tika, bitnami/mongodb, bitnami/redis, buanet/iobroker, containrrr/watchtower, dschoen/jenkins-jekyll-docker, eclipse-mosquitto, emby/embyserver, exadel/compreface-{admin,api,core,fe}, flaresolverr/flaresolverr, gcr.io/cadvisor/cadvisor, ghcr.io/donkie/spoolman, ghcr.io/home-assistant/home-assistant, ghcr.io/immich-app/immich-machine-learning, ghcr.io/paperless-ngx/paperless-ngx, ghost, giansalex/monero-miner, gotenberg/gotenberg, grafana/grafana, grafana/loki, grafana/promtail, homegear/homegear, hon95/prometheus-nut-exporter, influxdb, jakowenko/double-take, jellyfin/jellyfin, jlesage/handbrake, koush/scrypted, lanrat/hass-screenshot, lscr.io/linuxserver/bookstack, lscr.io/linuxserver/mariadb, mailserver2/mailserver, mailserver2/postfixadmin, mailserver2/rainloop, mariadb, matomo, memcached, mitmproxy/mitmproxy, mysql, nginx, openhab/openhab, oznu/homebridge, paperlessngx/paperless-ngx, portainer/portainer-ee, postgres, prom/alertmanager, prom/node-exporter, prom/prometheus, redis, registry, schoendominik/sma-emeter, seafileltd/seafile-mc, telegraf, thecodingmachine/gotenberg, traefik, vxcontrol/rancher-letsencrypt, willfarrell/autoheal
|
||||
Reference in New Issue
Block a user