127 lines
5.2 KiB
Markdown
127 lines
5.2 KiB
Markdown
# 10.0.30.100 Decommission — Session Detail (2026-06-30)
|
||
|
||
## Host Info
|
||
- **Host**: 10.0.30.100 (Ubuntu, hostname `ubuntu`)
|
||
- **User**: dominik (password auth initially, SSH key deployed)
|
||
- **Docker**: 28.1.1, ZFS backend (`pool01_n2_redundant/docker` dataset, 2.7TB)
|
||
- **SSH Key**: `~/.ssh/hermes_pve_agent` deployed to dominik's authorized_keys
|
||
|
||
## SUDO_ASKPASS Setup
|
||
- Password: provided in chat (do not persist)
|
||
- Askpass script deployed to `/tmp/askpass.sh` on remote host
|
||
- Pattern: `SUDO_ASKPASS=/tmp/askpass.sh sudo -A COMMAND`
|
||
|
||
## Services Removed (10 initial services)
|
||
|
||
### Containers
|
||
None were running — only images and volumes remained. All 10 services had been stopped previously.
|
||
|
||
### Images Removed (~28 GB)
|
||
| Service | Images | Approx Size |
|
||
|---------|--------|-------------|
|
||
| Frigate | 0.16.0-beta2, beta4, stable | ~8.5 GB |
|
||
| Immich | server + ML + ML-openvino | ~3.0 GB |
|
||
| Paperless | 6 versions (2.9.0–2.17.1) | ~8.3 GB |
|
||
| Scrypted | latest | 2.2 GB |
|
||
| Homegear | stable | 1.5 GB |
|
||
| ioBroker | latest | 1.3 GB |
|
||
| Homebridge | ubuntu-no-avahi | 0.9 GB |
|
||
| openHAB | 3.4.5 | 0.7 GB |
|
||
| Node-RED | latest | 0.6 GB |
|
||
| Jenkins | jekyll-docker | 0.5 GB |
|
||
|
||
### Volumes Removed (20 volumes)
|
||
frigate_config, frigate_storage, frigate_postgres-data, frigate_double-take, homebridge, homegear-data-etc, homegear-data-lib, homegear-data-log, immich_model-cache, immich_pgdata, iobrokerdata, jenkins_home, nodered, openhab4_openhab4_addons, openhab4_openhab4_conf, openhab4_openhab4_userdata, paperless_data, paperless_media, paperless_redisdata, scrypted
|
||
|
||
### Bind Mounts Cleaned
|
||
- `/pool01_n2_redundant/HomeAssistant/frigate` — deleted (clips, recordings)
|
||
- `/pool01_n2_redundant/homes/dominik/.homebridge` — deleted (config, accessories)
|
||
- `/pool01_n2_redundant/Frigate/` — emptied (ZFS dataset left, "dataset is busy")
|
||
|
||
## GitLab + Runner Removal (additional)
|
||
|
||
### Containers Removed
|
||
- `development-gitlab-1` (GitLab CE 15.8.1)
|
||
- `gitlab-runner` (gitlab-runner:alpine)
|
||
- `development-register-runner-1` (registration container, already exited)
|
||
- `development-dind-1` (Docker-in-Docker, **was causing restart-loop**)
|
||
|
||
### Images Removed
|
||
- gitlab/gitlab-ce:15.8.1-ce.0
|
||
- gitlab/gitlab-runner:alpine
|
||
- docker:20-dind
|
||
- docker:dind
|
||
|
||
### Volumes Removed
|
||
- gitlab_config, gitlab_data, gitlab_logs
|
||
- development_gitlab-runner
|
||
- 2× runner cache volumes (runner-xtxmd-6p-project-11-concurrent-0-cache-*)
|
||
|
||
## InfluxDB + Home Assistant Volume Removal
|
||
|
||
### Volumes Deleted (~273 GB freed)
|
||
| Volume | Size | Content | Data Range |
|
||
|--------|------|---------|------------|
|
||
| `506f355d...` (anonymous) | 270 GB | InfluxDB engine (data + wal) | Oct 2022 – Aug 2025 |
|
||
| `homeassistant_home_assistant_config` | 2.7 GB | HA config (duplicate, HA runs on 10.0.30.10) | — |
|
||
|
||
## Remaining Volumes on Host (analysis done, not yet acted on)
|
||
|
||
### Large Volumes
|
||
| Volume | Size | Content | Data Range |
|
||
|--------|------|---------|------------|
|
||
| seafile_data | 246 GB | Seafile (logs, nginx, seafile-conf) | — |
|
||
| seafile_custom_data | 226 GB | Seafile storage (blocks, commits, fs) | Mar 2015 – May 2025 |
|
||
| registry_lib | 1.4 GB | Docker Registry | — |
|
||
|
||
### Medium Volumes
|
||
| Volume | Size | Content |
|
||
|--------|------|---------|
|
||
| embyconfig | 860 MB | Emby Media Server |
|
||
| trivy_cache | 433 MB | Trivy Security Scanner |
|
||
| jellyfin_jellyfin_config | 314 MB | Jellyfin |
|
||
| matomo | 134 MB | Matomo Analytics |
|
||
| monitoring_prometheus_data | 146 MB | Prometheus |
|
||
| monitoring_grafana_db | 61 MB | Grafana |
|
||
| portainer_data | 63 MB | Portainer |
|
||
| pgdata | 46 MB | PostgreSQL |
|
||
| ghost-imkerei | 43 MB | Ghost Blog |
|
||
| openproject_postgresql_data | 25 MB | OpenProject |
|
||
|
||
### Small Volumes (<10 MB)
|
||
grafana, vicare, handbrake_config, openproject_assets, spoolman, proxy_conf, traefik_acme, telegraf, sabnzbd, bitmonero, ethereum, bienen_proxy, ghost_proxy, xenia_*, ~60 anonymous hash volumes (mostly 7 KB each, one 943 KB)
|
||
|
||
## Docker Daemon Instability
|
||
|
||
### Root Cause
|
||
`development-dind-1` container had `restart: always` and crashed on boot, creating a restart-loop. This made the Docker daemon extremely slow/unresponsive:
|
||
- `docker images` hung or returned empty (false verification!)
|
||
- `docker rmi` timed out
|
||
- `docker volume rm` timed out
|
||
- `docker ps` worked but slowly
|
||
|
||
### Resolution
|
||
1. `kill -9 $(pgrep dockerd)` — force kill daemon
|
||
2. `systemctl start docker` — restart
|
||
3. Wait 30-60s for ZFS backend to initialize
|
||
4. Quickly `docker rm -f development-dind-1` before it restarts
|
||
5. After removal, daemon became stable
|
||
|
||
### False Verification Incident
|
||
Initial verification showed `docker images | grep -i frigate` returning empty → concluded "all images deleted." After daemon restart, images were visible again. Lesson: always verify daemon stability before trusting `docker images` output.
|
||
|
||
## Remaining Containers After Cleanup
|
||
- portainer (Up)
|
||
- Traefik (Up)
|
||
- spoolman-spoolman-1 (Exited)
|
||
- website-imkerei-bienen-proxy-1 (Up)
|
||
- website-imkerei-proxy-1 (Up)
|
||
- website-imkerei-website-1 (Up)
|
||
|
||
## Model ID Drift (noris.ai)
|
||
- Old model ID: `vllm/gemma-4-31b-it-dynamo` (no longer available)
|
||
- New model ID: `vllm/gemma-4-31b-it` (verified via `/v1/models` endpoint)
|
||
- Also available: `vllm/prod/gemma-4-31b-it`
|
||
- Fixed in: `recipe_scraper_v8_intl.py` line 108
|
||
- Updated: cronjob prompt, memory entry
|