hermes-skills/devops/docker-host-administration/references/docker-volume-forensics-2026-07.md

Docker Volume Forensics: Tracing Lost Data — 2026-07-14

Context

User believed years of HomeAssistant InfluxDB data existed on Docker host 10.0.30.100. Initial scan found only a 15 KB skeleton volume. A deep forensic search was needed to locate the actual data.

Search Methodology

When looking for "lost" Docker container data, check ALL of these — not just named volumes:

1. Named Docker Volumes

# List all volumes (anonymous ones have hash names)
docker volume ls -q | wc -l  # 121 volumes found

# Search all volume mountpoints for specific data signatures
for d in /var/lib/docker/volumes/*/; do
  if ls "${d}_data/" 2>/dev/null | grep -q "^data$\|^meta$\|^wal$\|^influxd.bolt$\|^engine$"; then
    echo "FOUND: $d"
    ls -la "${d}_data/"
  fi
done

2. Bind Mounts (Host Filesystem)

Containers may use bind mounts instead of named volumes. Check:

# Directories on root partition
ls -la /var/lib/influxdb/ /var/lib/influxdb2/ 2>/dev/null

# ZFS pools (if Docker is on ZFS)
find /pool01_n2_redundant -maxdepth 5 -name "*.tsm" 2>/dev/null  # InfluxDB v1
find /pool01_n2_redundant -maxdepth 5 -name "influxd.bolt" 2>/dev/null  # InfluxDB v2

# Home directories (common bind mount source)
ls /pool01_n2_redundant/homes/dominik/ | grep -i influx

3. Docker Config Files Outside Volumes

# InfluxDB config was at /var/lib/docker/influx_cfg/influxdb.conf
# (NOT inside a volume — a bind-mounted config directory)
find /var/lib/docker -maxdepth 3 -name "*.conf" -o -name "*.yml" 2>/dev/null | grep -i influx

4. Portainer Stack Definitions

Portainer stores compose stack definitions in TWO locations:

Location 1: /data/compose/ (sparse, older stacks — may have empty dirs) Location 2: /var/lib/docker/volumes/portainer_data/_data/compose/ (canonical)

Each stack directory contains versioned subdirectories (v1/, v2/, ...) with docker-compose.yml. To read the latest version of every stack:

for d in /var/lib/docker/volumes/portainer_data/_data/compose/*/; do
  sid=$(basename $d)
  latest=$(ls -d ${d}v*/ 2>/dev/null | sort -V | tail -1)
  if [ -n "$latest" ]; then
    echo "=== STACK $sid ($(basename $latest)) ==="
    cat "${latest}"* 2>/dev/null
    echo ""
  fi
done

Portainer DB (BoltDB) is at: /var/lib/docker/volumes/portainer_data/_data/portainer.db

Can also extract stack defs via strings:

strings /var/lib/docker/volumes/portainer_data/_data/portainer.db | grep -i influx

Portainer backups at /var/lib/docker/volumes/portainer_data/_data/backup/ contain Portainer config + compose files but NOT Docker volume data.

4b. Real-World Portainer Stack Discovery (10.0.30.100, 2026-07-14)

Found 9 stacks in canonical location: 1 (Traefik), 7 (Seafile), 10 (Mailserver), 13 (Seafile-old), 14 (Monitoring: InfluxDB+Grafana+Loki), 18 (Imkerei/Ghost blog), 19 (GitLab), 26 (Mosquitto+HomeAssistant), 56 (Spoolman).

Stack 14 contained the InfluxDB service definition:

• InfluxSrv container with influxdb:latest
• Data volume 506f355... mounted at /var/lib/influxdb2 — deleted (not in docker volume ls, not on disk)
• Config volume c2f27e78... mounted at /etc/influxdb2 — exists but empty
• Bind mounts: /var/lib/docker/influx_cfg/influxdb.conf (v1 config from März 2017) and types.db (collectd types)
• Grafana env showed INFLUXDB_NAME: cadvisor — infrastructure monitoring, NOT Home Assistant data

Conclusion: The InfluxDB v1 data was deleted with the container. Only the Docker image (376 MB) and config artifacts survived.

5. Shell History (Last Resort)

Previous Docker commands may reveal the container's mount config:

# Check shell history files on ZFS home directories
cat /pool01_n2_redundant/homes/dominik/.influx_history
cat /pool01_n2_redundant/homes/dominik/.docker_history 2>/dev/null
cat /pool01_n2_redundant/homes/dominik/.bash_history 2>/dev/null | grep influx

6. Docker Image Inventory

Old images indicate what services ran, even if containers are gone:

docker images --format "{{.Repository}}:{{.Tag}}\t{{.Size}}" | grep -i influx
# influxdb:latest   376MB   (2 years ago)
# telegraf:latest   374MB

Key Findings (10.0.30.100, 2026-07-14)

What	Where	Size	Status
InfluxDB v1 data	Anonymous volume d735775...	15 KB	Skeleton only — influxd.bolt + influxd.sqlite from Jan 2023
InfluxDB v1 config	/var/lib/docker/influx_cfg/influxdb.conf	—	v1-style config (meta/data/wal dirs)
Telegraf config	Named volume telegraf	246 KB	Config only, no data
Shell history	/pool01_n2_redundant/homes/dominik/.influx_history	1 KB	Shows collectd + cadvisor databases (2017-era)
Docker image	influxdb:latest	376 MB	Cached, 2 years old
.tsm files	Nowhere	—	No InfluxDB v1 data files found on any filesystem

Conclusion: The InfluxDB v1 container was removed years ago. Its data volume contained only the bolt/sqlite skeleton (metadata without actual time-series data). The real v1 data (*.tsm files) was deleted with the container. Only the Docker image and config artifacts remained.

Pitfall: "Data Must Be Here" vs. "Data IS Gone"

When a user insists data existed on a host, systematically check all storage locations. BUT — if after exhaustive search no data files are found (no .tsm, no data/ directory with shards, no large volumes), the data was likely deleted when the container was removed. Docker container removal (docker rm) does NOT delete volumes by default, but docker rm -v or docker run --rm DOES delete anonymous volumes.

Pitfall: du Hangs on ZFS-Backed Docker Volumes

When checking Docker volume sizes on a ZFS-backed Docker installation, du -sh /var/lib/docker/volumes/*/_data can hang indefinitely (D-state) due to ZFS metadata I/O. Use:

# Quick ls-based check (doesn't read file contents)
ls -la /var/lib/docker/volumes/VOLNAME/_data/

# ZFS-native size (instant, no file traversal)
zfs list -o refer pool/docker/VOLNAME 2>/dev/null

# Per-volume with timeout (5s max per volume)
for vol in $(docker volume ls -q); do
  timeout 5 du -sh /var/lib/docker/volumes/$vol/_data 2>/dev/null || echo "$vol: TIMEOUT"
done