hermes-skills/devops/docker-host-administration/references/10.0.30.100-docker-analysis-2026-07.md

# Docker Analysis: 10.0.30.100 (2026-07-04)

## Host Info
- **Host**: ubuntu (10.0.30.100)
- **Storage Driver**: ZFS on `pool01_n2_redundant/docker`
- **Docker Root**: `/var/lib/docker`
- **Dataset USED**: 811 GB (includes snapshot overhead)
- **Dataset REFER**: 491 GB (live data)
- **Snapshot Overhead**: ~320 GB (deleted layers held by ZFS snapshots)
- **ZFS Subdatasets**: 735 (was 744 before 2 paperless tags removed)
- **Images**: 80 (before partial cleanup)
- **Containers**: 5 running (47 config dirs total — many dead/stopped)
- **Volumes**: 117 named + anonymous

## Daemon State: PARTIALLY RECOVERED

Docker daemon was `active` but effectively non-functional:
- `docker system df` crashed: `error getting build cache usage: failed to get usage for HASH: layer does not exist`
- `docker ps` sometimes worked (intermittently), sometimes timed out
- `docker images --format` returned empty output with exit 0 (silent failure)
- `docker system prune` failed: `a prune operation is already running`
- `docker rmi` for some images worked (2 paperless tags removed), most timed out
- `journalctl -u docker` showed: `error creating zfs mount: mount pool/docker/HASH: no such file or directory` and `Error unmounting /var/lib/docker/zfs/graph/HASH: invalid argument`

## Recovery Attempt & Outcome

1. `systemctl restart docker` — hung in `deactivating` for 60s, then `stop-sigterm` timeout, SIGKILL
2. `dockerd` became zombie (`<defunct>`)
3. Killed leftover `containerd-shim-runc-v2` process
4. `systemctl restart containerd` — succeeded
5. `systemctl start docker` — entered `activating` state
6. After 2-3 minutes, daemon reached `active` (ZFS graphdriver scanned 735 subdatasets)
7. After restart: `docker ps` worked (5 containers up), but `docker images` returned empty and `docker system df` timed out
8. Triggered `docker image prune -a` — spawned `zfs destroy -r` for each layer, each entering D-state due to I/O contention with concurrent rsync
9. `zfs destroy` processes hung in D-state (uninterruptible) — blocked all subsequent Docker operations until each completed sequentially
10. Daemon remained `active` throughout — `docker ps` continued to work, but `docker images` stayed empty and `docker system df` timed out

**Outcome**: Daemon IS up and serving containers, but graphdriver is degraded. `docker ps` works; `docker images` returns empty; `docker system df` times out. The `zfs destroy` cascade from `image prune -a` is still running sequentially in the background.

## `zfs destroy` D-State Under I/O Contention

When Docker's ZFS graphdriver cleans up layers, it spawns `zfs destroy -r pool/docker/<hash>` for each removed layer. Under heavy pool I/O (e.g. concurrent rsync reading 300+ GB from the same ZFS pool), each `zfs destroy` enters **D-state (uninterruptible sleep)** and can take minutes per layer.

**Symptoms:**
- `ps aux | grep "zfs destroy"` shows process in `D` state
- Docker CLI commands (`docker images`, `docker system df`) time out while destroy runs
- `zpool iostat` shows high read throughput (rsync) but near-zero writes (destroy blocked)
- The dataset being destroyed may already report "does not exist" in `zfs list` — the destroy is stuck in kernel metadata flush, not actual data deletion

**Key insight:** This is NOT a permanent hang — `zfs destroy` eventually completes, but each one is serialized and I/O-starved. Docker appears broken during this window but is actually just waiting for ZFS to finish cleaning up.

**Recovery:** Patience. Don't kill the `zfs destroy` process (D-state is immune to SIGKILL anyway). Wait for I/O contention to ease (e.g. let rsync finish), then Docker operations resume. If urgent, reduce concurrent I/O on the pool.

## Root Cause

735 ZFS subdatasets for Docker layers, several with corrupt ZFS mounts (`no such file or directory`). The ZFS graphdriver tries to mount each subdataset during daemon startup; corrupt ones cause indefinite hangs. The daemon cannot complete initialization.

## Complete Volume Size Inventory

### Named Volumes (all significant sizes)

| Volume | Size | Purpose |
|--------|------|---------|
| embyconfig | 860 MB | Emby Media Server config |
| jellyfin_jellyfin_config | 314 MB | Jellyfin config |
| monitoring_prometheus_data | 146 MB | Prometheus metrics |
| matomo | 134 MB | Matomo analytics |
| monitoring_grafana_db | 61 MB | Grafana dashboards |
| ghost-imkerei | 43 MB | Ghost blog (Imkerei) |
| jellyfin_jellyfin_cache | 18 MB | Jellyfin cache |
| grafana | 7.2 MB | Grafana (legacy?) |
| handbrake_config | 3 MB | Handbrake config |
| openproject_openproject_assets | 912 KB | OpenProject |
| monitoring_alertmanager | 14 KB | Alertmanager config |
| ethereum | 14 KB | Ethereum node |
| monitoring_prometheus_alerts | 13 KB | Prometheus alerts |
| monitoring_promtail_config | 7 KB | Promtail config |
| monitoring_loki_config | 7 KB | Loki config |
| bitmonero | 180 KB | Monero miner |
| bienen_proxy | 59 KB | nginx proxy (Imkerei) |
| ghost_proxy | 59 KB | nginx proxy (Ghost) |

**Total named volumes: ~1.5 GB.** Volumes are NOT the space problem.

### Anonymous Volumes (hash names)
~100 volumes, all < 1 MB each. Total < 5 MB. Negligible.

### Container Config Dirs
47 directories à ~100-200 KB (logs/metadata). Total ~5 MB.

### Where the 811 GB Actually Lives

| Area | Size | % of total |
|------|------|------------|
| ZFS Image Layers (735 subdatasets) | ~491 GB | 60% |
| Snapshot/Clone Overhead | ~320 GB | 40% |
| Named Volumes | ~1.5 GB | <1% |
| Anonymous Volumes | ~5 MB | ~0% |
| Container Configs | ~5 MB | ~0% |

## Seafile Stack Inventory

### Docker Volumes (11 total!)

| Volume | Content | Notes |
|--------|---------|-------|
| seafile_data | logs, nginx, seafile core | Main Seafile data |
| seafile_custom_data | commits, fs, storage, thumbnails, avatars | **Actual file storage** |
| seafile-conf | Seafile config | |
| seafile_custom_conf | Custom config | |
| seafile_custom_proxy_conf | Proxy config | |
| seafile_custom_mariadb | MariaDB data (22 entries) | Active DB? |
| seafile-mysql | MySQL data (28 entries) | Active DB? |
| seafile_mysql | MySQL data (24 entries) | Another DB instance |
| seafile-mysql.crashed | Crashed MySQL (50 entries) | ⚠️ Old crash, never cleaned |
| seafile_seafile-mysql-2023 | 2023 backup MySQL (16 entries) | Old backup |

**3 MySQL/MariaDB instances** + 1 crashed + 1 old 2023 backup = 5 database volumes for one Seafile installation.

### Legacy Seafile (non-Docker)
- `/pool01_n2_redundant/homes/dominik/seafile-server/` — Original 2014 Seafile install (src + seahub only, no data)

### Seafile Container Status
NOT running. No Seafile containers in `docker ps`.

## Mailserver Stack Inventory

### Docker Volume: `mailserver`
| Subdir | Content | Notes |
|--------|---------|-------|
| mail/ | Postfächer (UID 1024, since 2018) | Actual mail data |
| mysql/ | Mailserver MySQL DB | |
| mysql_bak/ | MySQL backup | |
| rainloop/ | Rainloop webmail | UID 991 |
| redis/ | Redis cache | |
| traefik/ | Traefik config | |

### ZFS Maildirs
| Path | Size | Owner |
|------|------|-------|
| `/pool01_n2_redundant/homes/dominik/.Maildir` | 540 MB | Dominik |
| `/pool01_n2_redundant/homes/sarah/.Maildir` | 2 GB | Sarah |

### Mailserver Container Status
NOT running. No mailserver container in `docker ps`.

## ZFS Homes Overview

| Dataset | USED | REFER |
|---------|------|-------|
| pool01_n2_redundant/homes | 456 GB | 455 GB |

Homes contain: dominik (72 entries), sarah (31 entries). Includes .Maildir, seafile-server legacy, and many personal directories.

## Running Containers (after recovery)

| Name | Image | Status |
|------|-------|--------|
| Traefik | traefik:latest | Up |
| website-imkerei-bienen-proxy-1 | nginx:alpine | Up |
| website-imkerei-proxy-1 | nginx:alpine | Up |
| website-imkerei-website-1 | ghost:5.3 | Up |
| portainer | portainer/portainer-ee:lts | Up |

spoolman-spoolman-1 was stuck in "Removal In Progress" — cleared after daemon restart.

## Top Images by Size (before crash)

| Size | Image |
|------|-------|
| 4.08 GB | exadel/compreface-core:1.2.0 |
| 2.2 GB | koush/scrypted:latest |
| 1.98 GB | home-assistant:stable |
| 1.67 GB | gotenberg/gotenberg:8.9 |
| 1.55 GB | thecodingmachine/gotenberg:latest |
| 1.54 GB | homegear/homegear:stable |
| 1.38 GB | seafile-mc:11.0-latest |
| 1.37 GB | seafile-mc:latest |
| 1.37 GB | paperless-ngx:2.17.1 |
| 1.36 GB | paperless-ngx:2.15.1 |
| 1.29 GB | immich-machine-learning:openvino |
| 1.26 GB | iobroker:latest |
| 1.09 GB | spoolman:latest |
| 1.07 GB | double-take:latest |
| 1.04 GB | jellyfin:latest |

## Duplicate Images Identified

| Image | Versions | Est. Waste |
|-------|----------|------------|
| paperless-ngx | 2.9.0, 2.12, 2.12.0, 2.15.1, 2.17.1 + 2× ghcr | ~6 GB |
| seafile-mc | 11.0-latest, latest | ~2.7 GB |
| gotenberg | 8.9, latest | ~3.2 GB |
| portainer-ee | lts, 2.21.5, `<none>` | ~0.7 GB |
| memcached | 1.6, latest, `<none>` | ~0.25 GB |
| postgres | 13, 16 | ~0.85 GB |
| Dangling | 3× `<none>` | ~0.3 GB |

## Recommendation

Docker ZFS storage driver is irreparably corrupt. Options:
1. **Rebuild (recommended)**: Stop docker, rename `/var/lib/docker`, restart with `overlay2` on a separate ext4/xfs volume, redeploy containers. Volumes on ZFS survive.
2. **Manual ZFS cleanup**: Find and `zfs destroy` corrupt subdatasets. Risky, time-consuming, may not fix all issues.

## All Unique Repos (60)

alpine, apache/tika, bitnami/mongodb, bitnami/redis, buanet/iobroker, containrrr/watchtower, dschoen/jenkins-jekyll-docker, eclipse-mosquitto, emby/embyserver, exadel/compreface-{admin,api,core,fe}, flaresolverr/flaresolverr, gcr.io/cadvisor/cadvisor, ghcr.io/donkie/spoolman, ghcr.io/home-assistant/home-assistant, ghcr.io/immich-app/immich-machine-learning, ghcr.io/paperless-ngx/paperless-ngx, ghost, giansalex/monero-miner, gotenberg/gotenberg, grafana/grafana, grafana/loki, grafana/promtail, homegear/homegear, hon95/prometheus-nut-exporter, influxdb, jakowenko/double-take, jellyfin/jellyfin, jlesage/handbrake, koush/scrypted, lanrat/hass-screenshot, lscr.io/linuxserver/bookstack, lscr.io/linuxserver/mariadb, mailserver2/mailserver, mailserver2/postfixadmin, mailserver2/rainloop, mariadb, matomo, memcached, mitmproxy/mitmproxy, mysql, nginx, openhab/openhab, oznu/homebridge, paperlessngx/paperless-ngx, portainer/portainer-ee, postgres, prom/alertmanager, prom/node-exporter, prom/prometheus, redis, registry, schoendominik/sma-emeter, seafileltd/seafile-mc, telegraf, thecodingmachine/gotenberg, traefik, vxcontrol/rancher-letsencrypt, willfarrell/autoheal