hermes-skills/devops/docker-host-administration/references/docker-volume-forensics-2026-07.md

# Docker Volume Forensics: Tracing Lost Data — 2026-07-14

## Context

User believed years of HomeAssistant InfluxDB data existed on Docker
host 10.0.30.100. Initial scan found only a 15 KB skeleton volume. A
deep forensic search was needed to locate the actual data.

## Search Methodology

When looking for "lost" Docker container data, check ALL of these —
not just named volumes:

### 1. Named Docker Volumes

```bash
# List all volumes (anonymous ones have hash names)
docker volume ls -q | wc -l  # 121 volumes found

# Search all volume mountpoints for specific data signatures
for d in /var/lib/docker/volumes/*/; do
  if ls "${d}_data/" 2>/dev/null | grep -q "^data$\|^meta$\|^wal$\|^influxd.bolt$\|^engine$"; then
    echo "FOUND: $d"
    ls -la "${d}_data/"
  fi
done
```

### 2. Bind Mounts (Host Filesystem)

Containers may use bind mounts instead of named volumes. Check:
```bash
# Directories on root partition
ls -la /var/lib/influxdb/ /var/lib/influxdb2/ 2>/dev/null

# ZFS pools (if Docker is on ZFS)
find /pool01_n2_redundant -maxdepth 5 -name "*.tsm" 2>/dev/null  # InfluxDB v1
find /pool01_n2_redundant -maxdepth 5 -name "influxd.bolt" 2>/dev/null  # InfluxDB v2

# Home directories (common bind mount source)
ls /pool01_n2_redundant/homes/dominik/ | grep -i influx
```

### 3. Docker Config Files Outside Volumes

```bash
# InfluxDB config was at /var/lib/docker/influx_cfg/influxdb.conf
# (NOT inside a volume — a bind-mounted config directory)
find /var/lib/docker -maxdepth 3 -name "*.conf" -o -name "*.yml" 2>/dev/null | grep -i influx
```

### 4. Portainer Stack Definitions

Portainer stores compose stack definitions in TWO locations:

**Location 1:** `/data/compose/` (sparse, older stacks — may have empty dirs)
**Location 2:** `/var/lib/docker/volumes/portainer_data/_data/compose/` (canonical)

Each stack directory contains versioned subdirectories (`v1/`, `v2/`, ...)
with `docker-compose.yml`. To read the latest version of every stack:

```bash
for d in /var/lib/docker/volumes/portainer_data/_data/compose/*/; do
  sid=$(basename $d)
  latest=$(ls -d ${d}v*/ 2>/dev/null | sort -V | tail -1)
  if [ -n "$latest" ]; then
    echo "=== STACK $sid ($(basename $latest)) ==="
    cat "${latest}"* 2>/dev/null
    echo ""
  fi
done
```

Portainer DB (BoltDB) is at:
`/var/lib/docker/volumes/portainer_data/_data/portainer.db`

Can also extract stack defs via `strings`:
```bash
strings /var/lib/docker/volumes/portainer_data/_data/portainer.db | grep -i influx
```

Portainer backups at `/var/lib/docker/volumes/portainer_data/_data/backup/`
contain Portainer config + compose files but NOT Docker volume data.

### 4b. Real-World Portainer Stack Discovery (10.0.30.100, 2026-07-14)

Found 9 stacks in canonical location: 1 (Traefik), 7 (Seafile), 10
(Mailserver), 13 (Seafile-old), 14 (Monitoring: InfluxDB+Grafana+Loki),
18 (Imkerei/Ghost blog), 19 (GitLab), 26 (Mosquitto+HomeAssistant),
56 (Spoolman).

Stack 14 contained the InfluxDB service definition:
- `InfluxSrv` container with `influxdb:latest`
- Data volume `506f355...` mounted at `/var/lib/influxdb2` — **deleted**
  (not in `docker volume ls`, not on disk)
- Config volume `c2f27e78...` mounted at `/etc/influxdb2` — exists but empty
- Bind mounts: `/var/lib/docker/influx_cfg/influxdb.conf` (v1 config from
  März 2017) and `types.db` (collectd types)
- Grafana env showed `INFLUXDB_NAME: cadvisor` — infrastructure monitoring,
  NOT Home Assistant data

Conclusion: The InfluxDB v1 data was deleted with the container. Only the
Docker image (376 MB) and config artifacts survived.

### 5. Shell History (Last Resort)

Previous Docker commands may reveal the container's mount config:
```bash
# Check shell history files on ZFS home directories
cat /pool01_n2_redundant/homes/dominik/.influx_history
cat /pool01_n2_redundant/homes/dominik/.docker_history 2>/dev/null
cat /pool01_n2_redundant/homes/dominik/.bash_history 2>/dev/null | grep influx
```

### 6. Docker Image Inventory

Old images indicate what services ran, even if containers are gone:
```bash
docker images --format "{{.Repository}}:{{.Tag}}\t{{.Size}}" | grep -i influx
# influxdb:latest   376MB   (2 years ago)
# telegraf:latest   374MB
```

## Key Findings (10.0.30.100, 2026-07-14)

| What | Where | Size | Status |
|------|-------|------|--------|
| InfluxDB v1 data | Anonymous volume `d735775...` | 15 KB | Skeleton only — `influxd.bolt` + `influxd.sqlite` from Jan 2023 |
| InfluxDB v1 config | `/var/lib/docker/influx_cfg/influxdb.conf` | — | v1-style config (meta/data/wal dirs) |
| Telegraf config | Named volume `telegraf` | 246 KB | Config only, no data |
| Shell history | `/pool01_n2_redundant/homes/dominik/.influx_history` | 1 KB | Shows `collectd` + `cadvisor` databases (2017-era) |
| Docker image | `influxdb:latest` | 376 MB | Cached, 2 years old |
| `.tsm` files | Nowhere | — | No InfluxDB v1 data files found on any filesystem |

**Conclusion:** The InfluxDB v1 container was removed years ago. Its
data volume contained only the bolt/sqlite skeleton (metadata without
actual time-series data). The real v1 data (`*.tsm` files) was deleted
with the container. Only the Docker image and config artifacts remained.

## Pitfall: "Data Must Be Here" vs. "Data IS Gone"

When a user insists data existed on a host, systematically check all
storage locations. BUT — if after exhaustive search no data files are
found (no `.tsm`, no `data/` directory with shards, no large volumes),
the data was likely deleted when the container was removed. Docker
container removal (`docker rm`) does NOT delete volumes by default, but
`docker rm -v` or `docker run --rm` DOES delete anonymous volumes.

## Pitfall: `du` Hangs on ZFS-Backed Docker Volumes

When checking Docker volume sizes on a ZFS-backed Docker installation,
`du -sh /var/lib/docker/volumes/*/_data` can hang indefinitely (D-state)
due to ZFS metadata I/O. Use:

```bash
# Quick ls-based check (doesn't read file contents)
ls -la /var/lib/docker/volumes/VOLNAME/_data/

# ZFS-native size (instant, no file traversal)
zfs list -o refer pool/docker/VOLNAME 2>/dev/null

# Per-volume with timeout (5s max per volume)
for vol in $(docker volume ls -q); do
  timeout 5 du -sh /var/lib/docker/volumes/$vol/_data 2>/dev/null || echo "$vol: TIMEOUT"
done
```