feat: add home-assistant-dashboard-conventions skill + update multiple skills
- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
This commit is contained in:
Debian
@@ -0,0 +1,166 @@
|
||||
# Docker Volume Forensics: Tracing Lost Data — 2026-07-14
|
||||
|
||||
## Context
|
||||
|
||||
User believed years of HomeAssistant InfluxDB data existed on Docker
|
||||
host 10.0.30.100. Initial scan found only a 15 KB skeleton volume. A
|
||||
deep forensic search was needed to locate the actual data.
|
||||
|
||||
## Search Methodology
|
||||
|
||||
When looking for "lost" Docker container data, check ALL of these —
|
||||
not just named volumes:
|
||||
|
||||
### 1. Named Docker Volumes
|
||||
|
||||
```bash
|
||||
# List all volumes (anonymous ones have hash names)
|
||||
docker volume ls -q | wc -l # 121 volumes found
|
||||
|
||||
# Search all volume mountpoints for specific data signatures
|
||||
for d in /var/lib/docker/volumes/*/; do
|
||||
if ls "${d}_data/" 2>/dev/null | grep -q "^data$\|^meta$\|^wal$\|^influxd.bolt$\|^engine$"; then
|
||||
echo "FOUND: $d"
|
||||
ls -la "${d}_data/"
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
### 2. Bind Mounts (Host Filesystem)
|
||||
|
||||
Containers may use bind mounts instead of named volumes. Check:
|
||||
```bash
|
||||
# Directories on root partition
|
||||
ls -la /var/lib/influxdb/ /var/lib/influxdb2/ 2>/dev/null
|
||||
|
||||
# ZFS pools (if Docker is on ZFS)
|
||||
find /pool01_n2_redundant -maxdepth 5 -name "*.tsm" 2>/dev/null # InfluxDB v1
|
||||
find /pool01_n2_redundant -maxdepth 5 -name "influxd.bolt" 2>/dev/null # InfluxDB v2
|
||||
|
||||
# Home directories (common bind mount source)
|
||||
ls /pool01_n2_redundant/homes/dominik/ | grep -i influx
|
||||
```
|
||||
|
||||
### 3. Docker Config Files Outside Volumes
|
||||
|
||||
```bash
|
||||
# InfluxDB config was at /var/lib/docker/influx_cfg/influxdb.conf
|
||||
# (NOT inside a volume — a bind-mounted config directory)
|
||||
find /var/lib/docker -maxdepth 3 -name "*.conf" -o -name "*.yml" 2>/dev/null | grep -i influx
|
||||
```
|
||||
|
||||
### 4. Portainer Stack Definitions
|
||||
|
||||
Portainer stores compose stack definitions in TWO locations:
|
||||
|
||||
**Location 1:** `/data/compose/` (sparse, older stacks — may have empty dirs)
|
||||
**Location 2:** `/var/lib/docker/volumes/portainer_data/_data/compose/` (canonical)
|
||||
|
||||
Each stack directory contains versioned subdirectories (`v1/`, `v2/`, ...)
|
||||
with `docker-compose.yml`. To read the latest version of every stack:
|
||||
|
||||
```bash
|
||||
for d in /var/lib/docker/volumes/portainer_data/_data/compose/*/; do
|
||||
sid=$(basename $d)
|
||||
latest=$(ls -d ${d}v*/ 2>/dev/null | sort -V | tail -1)
|
||||
if [ -n "$latest" ]; then
|
||||
echo "=== STACK $sid ($(basename $latest)) ==="
|
||||
cat "${latest}"* 2>/dev/null
|
||||
echo ""
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
Portainer DB (BoltDB) is at:
|
||||
`/var/lib/docker/volumes/portainer_data/_data/portainer.db`
|
||||
|
||||
Can also extract stack defs via `strings`:
|
||||
```bash
|
||||
strings /var/lib/docker/volumes/portainer_data/_data/portainer.db | grep -i influx
|
||||
```
|
||||
|
||||
Portainer backups at `/var/lib/docker/volumes/portainer_data/_data/backup/`
|
||||
contain Portainer config + compose files but NOT Docker volume data.
|
||||
|
||||
### 4b. Real-World Portainer Stack Discovery (10.0.30.100, 2026-07-14)
|
||||
|
||||
Found 9 stacks in canonical location: 1 (Traefik), 7 (Seafile), 10
|
||||
(Mailserver), 13 (Seafile-old), 14 (Monitoring: InfluxDB+Grafana+Loki),
|
||||
18 (Imkerei/Ghost blog), 19 (GitLab), 26 (Mosquitto+HomeAssistant),
|
||||
56 (Spoolman).
|
||||
|
||||
Stack 14 contained the InfluxDB service definition:
|
||||
- `InfluxSrv` container with `influxdb:latest`
|
||||
- Data volume `506f355...` mounted at `/var/lib/influxdb2` — **deleted**
|
||||
(not in `docker volume ls`, not on disk)
|
||||
- Config volume `c2f27e78...` mounted at `/etc/influxdb2` — exists but empty
|
||||
- Bind mounts: `/var/lib/docker/influx_cfg/influxdb.conf` (v1 config from
|
||||
März 2017) and `types.db` (collectd types)
|
||||
- Grafana env showed `INFLUXDB_NAME: cadvisor` — infrastructure monitoring,
|
||||
NOT Home Assistant data
|
||||
|
||||
Conclusion: The InfluxDB v1 data was deleted with the container. Only the
|
||||
Docker image (376 MB) and config artifacts survived.
|
||||
|
||||
### 5. Shell History (Last Resort)
|
||||
|
||||
Previous Docker commands may reveal the container's mount config:
|
||||
```bash
|
||||
# Check shell history files on ZFS home directories
|
||||
cat /pool01_n2_redundant/homes/dominik/.influx_history
|
||||
cat /pool01_n2_redundant/homes/dominik/.docker_history 2>/dev/null
|
||||
cat /pool01_n2_redundant/homes/dominik/.bash_history 2>/dev/null | grep influx
|
||||
```
|
||||
|
||||
### 6. Docker Image Inventory
|
||||
|
||||
Old images indicate what services ran, even if containers are gone:
|
||||
```bash
|
||||
docker images --format "{{.Repository}}:{{.Tag}}\t{{.Size}}" | grep -i influx
|
||||
# influxdb:latest 376MB (2 years ago)
|
||||
# telegraf:latest 374MB
|
||||
```
|
||||
|
||||
## Key Findings (10.0.30.100, 2026-07-14)
|
||||
|
||||
| What | Where | Size | Status |
|
||||
|------|-------|------|--------|
|
||||
| InfluxDB v1 data | Anonymous volume `d735775...` | 15 KB | Skeleton only — `influxd.bolt` + `influxd.sqlite` from Jan 2023 |
|
||||
| InfluxDB v1 config | `/var/lib/docker/influx_cfg/influxdb.conf` | — | v1-style config (meta/data/wal dirs) |
|
||||
| Telegraf config | Named volume `telegraf` | 246 KB | Config only, no data |
|
||||
| Shell history | `/pool01_n2_redundant/homes/dominik/.influx_history` | 1 KB | Shows `collectd` + `cadvisor` databases (2017-era) |
|
||||
| Docker image | `influxdb:latest` | 376 MB | Cached, 2 years old |
|
||||
| `.tsm` files | Nowhere | — | No InfluxDB v1 data files found on any filesystem |
|
||||
|
||||
**Conclusion:** The InfluxDB v1 container was removed years ago. Its
|
||||
data volume contained only the bolt/sqlite skeleton (metadata without
|
||||
actual time-series data). The real v1 data (`*.tsm` files) was deleted
|
||||
with the container. Only the Docker image and config artifacts remained.
|
||||
|
||||
## Pitfall: "Data Must Be Here" vs. "Data IS Gone"
|
||||
|
||||
When a user insists data existed on a host, systematically check all
|
||||
storage locations. BUT — if after exhaustive search no data files are
|
||||
found (no `.tsm`, no `data/` directory with shards, no large volumes),
|
||||
the data was likely deleted when the container was removed. Docker
|
||||
container removal (`docker rm`) does NOT delete volumes by default, but
|
||||
`docker rm -v` or `docker run --rm` DOES delete anonymous volumes.
|
||||
|
||||
## Pitfall: `du` Hangs on ZFS-Backed Docker Volumes
|
||||
|
||||
When checking Docker volume sizes on a ZFS-backed Docker installation,
|
||||
`du -sh /var/lib/docker/volumes/*/_data` can hang indefinitely (D-state)
|
||||
due to ZFS metadata I/O. Use:
|
||||
|
||||
```bash
|
||||
# Quick ls-based check (doesn't read file contents)
|
||||
ls -la /var/lib/docker/volumes/VOLNAME/_data/
|
||||
|
||||
# ZFS-native size (instant, no file traversal)
|
||||
zfs list -o refer pool/docker/VOLNAME 2>/dev/null
|
||||
|
||||
# Per-volume with timeout (5s max per volume)
|
||||
for vol in $(docker volume ls -q); do
|
||||
timeout 5 du -sh /var/lib/docker/volumes/$vol/_data 2>/dev/null || echo "$vol: TIMEOUT"
|
||||
done
|
||||
```
|
||||
Reference in New Issue
Block a user