dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
hermes-skills/devops/docker-host-administration/references/10.0.30.100-docker-analysis-2026-07.md
T

10 KiB
Raw Permalink Blame History

Docker Analysis: 10.0.30.100 (2026-07-04)

Host Info

  • Host: ubuntu (10.0.30.100)
  • Storage Driver: ZFS on pool01_n2_redundant/docker
  • Docker Root: /var/lib/docker
  • Dataset USED: 811 GB (includes snapshot overhead)
  • Dataset REFER: 491 GB (live data)
  • Snapshot Overhead: ~320 GB (deleted layers held by ZFS snapshots)
  • ZFS Subdatasets: 735 (was 744 before 2 paperless tags removed)
  • Images: 80 (before partial cleanup)
  • Containers: 5 running (47 config dirs total — many dead/stopped)
  • Volumes: 117 named + anonymous

Daemon State: PARTIALLY RECOVERED

Docker daemon was active but effectively non-functional:

  • docker system df crashed: error getting build cache usage: failed to get usage for HASH: layer does not exist
  • docker ps sometimes worked (intermittently), sometimes timed out
  • docker images --format returned empty output with exit 0 (silent failure)
  • docker system prune failed: a prune operation is already running
  • docker rmi for some images worked (2 paperless tags removed), most timed out
  • journalctl -u docker showed: error creating zfs mount: mount pool/docker/HASH: no such file or directory and Error unmounting /var/lib/docker/zfs/graph/HASH: invalid argument

Recovery Attempt & Outcome

  1. systemctl restart docker — hung in deactivating for 60s, then stop-sigterm timeout, SIGKILL
  2. dockerd became zombie (<defunct>)
  3. Killed leftover containerd-shim-runc-v2 process
  4. systemctl restart containerd — succeeded
  5. systemctl start docker — entered activating state
  6. After 2-3 minutes, daemon reached active (ZFS graphdriver scanned 735 subdatasets)
  7. After restart: docker ps worked (5 containers up), but docker images returned empty and docker system df timed out
  8. Triggered docker image prune -a — spawned zfs destroy -r for each layer, each entering D-state due to I/O contention with concurrent rsync
  9. zfs destroy processes hung in D-state (uninterruptible) — blocked all subsequent Docker operations until each completed sequentially
  10. Daemon remained active throughout — docker ps continued to work, but docker images stayed empty and docker system df timed out

Outcome: Daemon IS up and serving containers, but graphdriver is degraded. docker ps works; docker images returns empty; docker system df times out. The zfs destroy cascade from image prune -a is still running sequentially in the background.

zfs destroy D-State Under I/O Contention

When Docker's ZFS graphdriver cleans up layers, it spawns zfs destroy -r pool/docker/<hash> for each removed layer. Under heavy pool I/O (e.g. concurrent rsync reading 300+ GB from the same ZFS pool), each zfs destroy enters D-state (uninterruptible sleep) and can take minutes per layer.

Symptoms:

  • ps aux | grep "zfs destroy" shows process in D state
  • Docker CLI commands (docker images, docker system df) time out while destroy runs
  • zpool iostat shows high read throughput (rsync) but near-zero writes (destroy blocked)
  • The dataset being destroyed may already report "does not exist" in zfs list — the destroy is stuck in kernel metadata flush, not actual data deletion

Key insight: This is NOT a permanent hang — zfs destroy eventually completes, but each one is serialized and I/O-starved. Docker appears broken during this window but is actually just waiting for ZFS to finish cleaning up.

Recovery: Patience. Don't kill the zfs destroy process (D-state is immune to SIGKILL anyway). Wait for I/O contention to ease (e.g. let rsync finish), then Docker operations resume. If urgent, reduce concurrent I/O on the pool.

Root Cause

735 ZFS subdatasets for Docker layers, several with corrupt ZFS mounts (no such file or directory). The ZFS graphdriver tries to mount each subdataset during daemon startup; corrupt ones cause indefinite hangs. The daemon cannot complete initialization.

Complete Volume Size Inventory

Named Volumes (all significant sizes)

Volume Size Purpose
embyconfig 860 MB Emby Media Server config
jellyfin_jellyfin_config 314 MB Jellyfin config
monitoring_prometheus_data 146 MB Prometheus metrics
matomo 134 MB Matomo analytics
monitoring_grafana_db 61 MB Grafana dashboards
ghost-imkerei 43 MB Ghost blog (Imkerei)
jellyfin_jellyfin_cache 18 MB Jellyfin cache
grafana 7.2 MB Grafana (legacy?)
handbrake_config 3 MB Handbrake config
openproject_openproject_assets 912 KB OpenProject
monitoring_alertmanager 14 KB Alertmanager config
ethereum 14 KB Ethereum node
monitoring_prometheus_alerts 13 KB Prometheus alerts
monitoring_promtail_config 7 KB Promtail config
monitoring_loki_config 7 KB Loki config
bitmonero 180 KB Monero miner
bienen_proxy 59 KB nginx proxy (Imkerei)
ghost_proxy 59 KB nginx proxy (Ghost)

Total named volumes: ~1.5 GB. Volumes are NOT the space problem.

Anonymous Volumes (hash names)

~100 volumes, all < 1 MB each. Total < 5 MB. Negligible.

Container Config Dirs

47 directories à ~100-200 KB (logs/metadata). Total ~5 MB.

Where the 811 GB Actually Lives

Area Size % of total
ZFS Image Layers (735 subdatasets) ~491 GB 60%
Snapshot/Clone Overhead ~320 GB 40%
Named Volumes ~1.5 GB <1%
Anonymous Volumes ~5 MB ~0%
Container Configs ~5 MB ~0%

Seafile Stack Inventory

Docker Volumes (11 total!)

Volume Content Notes
seafile_data logs, nginx, seafile core Main Seafile data
seafile_custom_data commits, fs, storage, thumbnails, avatars Actual file storage
seafile-conf Seafile config
seafile_custom_conf Custom config
seafile_custom_proxy_conf Proxy config
seafile_custom_mariadb MariaDB data (22 entries) Active DB?
seafile-mysql MySQL data (28 entries) Active DB?
seafile_mysql MySQL data (24 entries) Another DB instance
seafile-mysql.crashed Crashed MySQL (50 entries) ⚠️ Old crash, never cleaned
seafile_seafile-mysql-2023 2023 backup MySQL (16 entries) Old backup

3 MySQL/MariaDB instances + 1 crashed + 1 old 2023 backup = 5 database volumes for one Seafile installation.

Legacy Seafile (non-Docker)

  • /pool01_n2_redundant/homes/dominik/seafile-server/ — Original 2014 Seafile install (src + seahub only, no data)

Seafile Container Status

NOT running. No Seafile containers in docker ps.

Mailserver Stack Inventory

Docker Volume: mailserver

Subdir Content Notes
mail/ Postfächer (UID 1024, since 2018) Actual mail data
mysql/ Mailserver MySQL DB
mysql_bak/ MySQL backup
rainloop/ Rainloop webmail UID 991
redis/ Redis cache
traefik/ Traefik config

ZFS Maildirs

Path Size Owner
/pool01_n2_redundant/homes/dominik/.Maildir 540 MB Dominik
/pool01_n2_redundant/homes/sarah/.Maildir 2 GB Sarah

Mailserver Container Status

NOT running. No mailserver container in docker ps.

ZFS Homes Overview

Dataset USED REFER
pool01_n2_redundant/homes 456 GB 455 GB

Homes contain: dominik (72 entries), sarah (31 entries). Includes .Maildir, seafile-server legacy, and many personal directories.

Running Containers (after recovery)

Name Image Status
Traefik traefik:latest Up
website-imkerei-bienen-proxy-1 nginx:alpine Up
website-imkerei-proxy-1 nginx:alpine Up
website-imkerei-website-1 ghost:5.3 Up
portainer portainer/portainer-ee:lts Up

spoolman-spoolman-1 was stuck in "Removal In Progress" — cleared after daemon restart.

Top Images by Size (before crash)

Size Image
4.08 GB exadel/compreface-core:1.2.0
2.2 GB koush/scrypted:latest
1.98 GB home-assistant:stable
1.67 GB gotenberg/gotenberg:8.9
1.55 GB thecodingmachine/gotenberg:latest
1.54 GB homegear/homegear:stable
1.38 GB seafile-mc:11.0-latest
1.37 GB seafile-mc:latest
1.37 GB paperless-ngx:2.17.1
1.36 GB paperless-ngx:2.15.1
1.29 GB immich-machine-learning:openvino
1.26 GB iobroker:latest
1.09 GB spoolman:latest
1.07 GB double-take:latest
1.04 GB jellyfin:latest

Duplicate Images Identified

Image Versions Est. Waste
paperless-ngx 2.9.0, 2.12, 2.12.0, 2.15.1, 2.17.1 + 2× ghcr ~6 GB
seafile-mc 11.0-latest, latest ~2.7 GB
gotenberg 8.9, latest ~3.2 GB
portainer-ee lts, 2.21.5, <none> ~0.7 GB
memcached 1.6, latest, <none> ~0.25 GB
postgres 13, 16 ~0.85 GB
Dangling 3× <none> ~0.3 GB

Recommendation

Docker ZFS storage driver is irreparably corrupt. Options:

  1. Rebuild (recommended): Stop docker, rename /var/lib/docker, restart with overlay2 on a separate ext4/xfs volume, redeploy containers. Volumes on ZFS survive.
  2. Manual ZFS cleanup: Find and zfs destroy corrupt subdatasets. Risky, time-consuming, may not fix all issues.

All Unique Repos (60)

alpine, apache/tika, bitnami/mongodb, bitnami/redis, buanet/iobroker, containrrr/watchtower, dschoen/jenkins-jekyll-docker, eclipse-mosquitto, emby/embyserver, exadel/compreface-{admin,api,core,fe}, flaresolverr/flaresolverr, gcr.io/cadvisor/cadvisor, ghcr.io/donkie/spoolman, ghcr.io/home-assistant/home-assistant, ghcr.io/immich-app/immich-machine-learning, ghcr.io/paperless-ngx/paperless-ngx, ghost, giansalex/monero-miner, gotenberg/gotenberg, grafana/grafana, grafana/loki, grafana/promtail, homegear/homegear, hon95/prometheus-nut-exporter, influxdb, jakowenko/double-take, jellyfin/jellyfin, jlesage/handbrake, koush/scrypted, lanrat/hass-screenshot, lscr.io/linuxserver/bookstack, lscr.io/linuxserver/mariadb, mailserver2/mailserver, mailserver2/postfixadmin, mailserver2/rainloop, mariadb, matomo, memcached, mitmproxy/mitmproxy, mysql, nginx, openhab/openhab, oznu/homebridge, paperlessngx/paperless-ngx, portainer/portainer-ee, postgres, prom/alertmanager, prom/node-exporter, prom/prometheus, redis, registry, schoendominik/sma-emeter, seafileltd/seafile-mc, telegraf, thecodingmachine/gotenberg, traefik, vxcontrol/rancher-letsencrypt, willfarrell/autoheal

Reference in New Issue View Git Blame Copy Permalink