hermes-skills/devops/infrastructure-recon/references/host-10.0.30.100-services.md

# Host 10.0.30.100 (hostname: ubuntu) — Full Service Inventory & Decommissioning Plan

Last updated: 2026-06-30 (session: Docker service cleanup — 10 services removed)

## Access

- **SSH**: Key auth with `~/.ssh/hermes_pve_agent` (deployed 2026-06-30). Previously password-only (`sshpass -p '<password>' ssh dominik@10.0.30.100`).
- **sudo**: `SUDO_ASKPASS=/tmp/askpass.sh sudo -A <cmd>` — see SKILL.md "Remote sudo Without Interactive Password" section
- **1Password**: "NUT UPS Monitor (10.0.30.100)" — contains NUT service creds, NOT SSH creds

## Hardware

- 4 CPU, Ubuntu 24.04.4 LTS
- **1× 256 GB SSD** (sda) — system disk (ext4 + swap)
- **9× 2.7 TB HDD** (sdb–sdi):
  - sdb, sdc, sdg, sdh, sdi → ZFS pool members
  - sde, sdf → Ceph BlueStore OSDs (LVM)
  - sdd → ZFS pool member

## ZFS Pool: pool01_n2_redundant

| Property | Value |
|----------|-------|
| Size | 10.9 TB |
| Allocated | 6.80 TB (62%) |
| Free | 4.10 TB |
| Fragmentation | 57% |
| Dedup | 1.12x |
| Health | ONLINE |

### ZFS Datasets (non-Docker)

| Dataset | Used | Mountpoint | Content |
|---------|------|------------|---------|
| TimeMachine | 951 GB | /pool01_n2_redundant/TimeMachine | macOS Time Machine backups |
| proxmox_nfs | 963 GB | /pool01_n2_redundant/proxmox_nfs | NFS share for Proxmox |
| homes | 455 GB | /pool01_n2_redundant/homes | User home directories |
| Download | 369 GB | /pool01_n2_redundant/Download | Downloads |
| HomeAssistant | 33.2 GB | /pool01_n2_redundant/HomeAssistant | HA config backup |
| proxmox_storage | 1 GB | /pool01_n2_redundant/proxmox_storage | Nearly empty |
| proxmox_iscsi | ~0 | /pool01_n2_redundant/proxmox_iscsi | Empty |
| proxmox_isci | ~0 | /pool01_n2_redundant/proxmox_isci | Empty (typo dataset) |
| Frigate | ~0 | /pool01_n2_redundant/Frigate | Empty — content deleted, dataset busy (can't zfs destroy yet) |
| docker | 781 GB | /var/lib/docker | Docker volumes + overlay |

## Completed: Docker Service Cleanup (2026-06-30)

### Removed: 10 Services (Images + Volumes + Bind Mounts)

All 10 services had NO running containers — only stale images, volumes, and bind-mount directories remained.

| Service | Images Removed | Volumes Removed | Bind Mounts Cleaned |
|---------|---------------|-----------------|---------------------|
| Frigate | 0.16.0-beta2, beta4, stable (~8.5 GB) | frigate_config, frigate_storage, frigate_postgres-data, frigate_double-take | /pool01_n2_redundant/Frigate/* (empty), /pool01_n2_redundant/HomeAssistant/frigate/ (clips+recordings) |
| Immich | server + ML + ML-openvino (~3 GB) | immich_pgdata, immich_model-cache | — |
| Paperless | 6 versions 2.9.0–2.17.1 (~8.3 GB) | paperless_data, paperless_media, paperless_redisdata | — |
| Scrypted | latest (2.2 GB) | scrypted | — |
| Homebridge | ubuntu-no-avahi (0.9 GB) | homebridge | /pool01_n2_redundant/homes/dominik/.homebridge/ (config + accessories) |
| Homegear | stable (1.5 GB) | homegear-data-etc, homegear-data-lib, homegear-data-log | — |
| openHAB | 3.4.5 (0.7 GB) | openhab4_openhab4_addons, openhab4_openhab4_conf, openhab4_openhab4_userdata | — |
| Node-RED | latest (0.6 GB) | nodered | — |
| ioBroker | latest (1.3 GB) | iobrokerdata | — |
| Jenkins | jekyll-docker (0.5 GB) | jenkins_home | — |

**Total reclaimed**: ~28 GB images + ~8.5 GB volumes + bind-mount data

### Issues Encountered During Cleanup

1. **Docker daemon lockup** — `development-dind-1` (docker:20-dind) in restart loop made all docker commands hang. Fixed by `sudo kill -9 $(pgrep dockerd)` — systemd auto-restarted with fresh daemon. See SKILL.md "Docker Daemon Lockup" section.

2. **FALSE NEGATIVE verification — daemon returned empty `docker images`** — After the first cleanup pass, `docker images | grep -iE "pattern"` returned empty (RC=1), suggesting all target images were deleted. I reported "ALL CLEAN". The user challenged this ("Hast du jetzt ALLE images gelöscht?"). After a daemon restart, `docker images` revealed 80+ images still present — only 3 of 10 target services (frigate beta2/beta4, immich-server, nodered) had actually been deleted. The degraded daemon had returned empty results despite images existing on disk. A second pass was needed to remove the remaining 12 images. **Lesson:** On potentially unstable Docker hosts, always sanity-check `docker images | wc -l` — if 0 but containers are running, the daemon is lying. See `references/docker-zfs-cleanup.md` "False Empty Results" section.

3. **ZFS slowness** — `docker rmi` and `docker volume rm` extremely slow on ZFS backend. Required batching (5-8 items per command) with 120s timeouts. Multiple passes needed — first batch removed ~half, second batch cleared the rest.

4. **ZFS dataset busy** — `zfs destroy pool01_n2_redundant/Frigate` failed with "dataset is busy" even after content deletion. Left empty (140K metadata) for later cleanup.

5. **SSH key deployment** — Initial SSH access was password-only. Deployed `hermes_pve_agent` pubkey via `sshpass` + `authorized_keys` append.

6. **Two-pass image deletion required** — First pass (background job) only deleted 3 of 19 target images before the daemon degraded. After daemon restart, a second `docker rmi -f` pass with all 15 remaining images succeeded in one command (120s timeout).

## Remaining: Docker Containers (still running)

| Container | Image | Ports | Status |
|-----------|-------|-------|--------|
| development-gitlab-1 | gitlab-ce:15.8.1-ce.0 | 22,80,443,2224→22 | DEPRECATED (Gitea replaces) |
| gitlab-runner | gitlab-runner:alpine | — | DEPRECATED |
| development-dind-1 | docker:20-dind | 2375-2376 | Restart loop — causes daemon lockups |
| Traefik | traefik:latest | 80→80 | Reverse proxy |
| portainer | portainer-ee:lts | 8000,9443 | Container management |
| spoolman-spoolman-1 | spoolman:latest | 7912→8000 | 3D filament mgmt |
| website-imkerei-website-1 | ghost:5.3 | 2368 | Imkerei blog |
| website-imkerei-proxy-1 | nginx:alpine | 80 | Proxy for imkerei.familie-schoen.com |
| website-imkerei-bienen-proxy-1 | nginx:alpine | 80 | Proxy for bienen.familie-schoen.com |

## Traefik Routing (Docker labels)

| Hostname | Target | Entrypoint |
|----------|--------|------------|
| git.familie-schoen.com | GitLab (port 80) | websecure (TLS) — NOW POINTS TO GITEA externally |
| registry.familie-schoen.com | GitLab registry (port 5000) | websecure |
| mgmt.familie-schoen.com | Traefik dashboard | web (basic auth) |
| imkerei.familie-schoen.com | nginx proxy → Ghost | web |
| bienen.familie-schoen.com | nginx proxy → Ghost | web |

## Completed: GitLab → Gitea Migration (2026-06-29)

All 12 GitLab repos migrated to Gitea (git.familie-schoen.com) via `git clone --mirror` + `git push --mirror` from 10.0.30.100. See `references/gitlab-to-gitea-migration.md` for technique.

## Decommissioning Plan (Remaining Phases)

### Phase 1: Data Migration (PARTIALLY COMPLETE)
✅ 10 Docker services removed (images + volumes + bind mounts)
⬜ Docker workloads → Proxmox LXC/VMs: Traefik, Ghost (Imkerei), Spoolman, Portainer (optional)
⬜ GitLab + Runner + DinD → remove (Gitea replaces ✅)
⬜ ZFS datasets → Ceph or NFS on Proxmox: TimeMachine (951 GB), homes (455 GB), Download (369 GB), proxmox_nfs (963 GB), HomeAssistant (33 GB)
⬜ Docker volumes for active services → migrate with containers

### Phase 2: Ceph Integration
1. Remove 9 HDDs from 10.0.30.100 (7 ZFS + 2 Ceph OSD)
2. Install in n5pro (10.0.20.91) — currently 0 OSDs, 120 GB NVMe only
3. Add as Ceph OSDs to Proxmox cluster
4. Adjust CRUSH rules if needed (HDD tier)

### Phase 3: Ceph OSD Decommission (on 10.0.30.100)
1. Mark osd.8, osd.9 as `out` → data redistributes
2. Wait for rebalance (cluster has 7.1 TB free — sufficient for 5.4 TB from 2 OSDs)
3. `ceph osd destroy` + `ceph osd purge`
4. Remove ceph-mon@ubuntu from cluster

### Phase 4: Shutdown
1. Export ZFS pool
2. Shut down server
3. Physically remove disks

### Capacity Check
- Ceph cluster: 12 TB raw, 5.1 TB used, 7.1 TB free
- ZFS data to migrate: ~6.8 TB (TimeMachine + homes + Download + proxmox_nfs + docker + misc)
- ⚠️ Tight capacity — may need to add disks to n5pro FIRST before draining ZFS data

### Ceph Health Warning (as of 2026-06-29)
- HEALTH_WARN: 3 OSDs slow operations, 1 stalled read in BlueFS, proxmox2 low on space
- osd.6 on proxmox6 is `destroyed` (not contributing)
- Should investigate before adding load