dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
hermes-skills/devops/infrastructure-recon/references/host-10.0.30.100-services.md
T

8.3 KiB
Raw Permalink Blame History

Host 10.0.30.100 (hostname: ubuntu) — Full Service Inventory & Decommissioning Plan

Last updated: 2026-06-30 (session: Docker service cleanup — 10 services removed)

Access

  • SSH: Key auth with ~/.ssh/hermes_pve_agent (deployed 2026-06-30). Previously password-only (sshpass -p '<password>' ssh dominik@10.0.30.100).
  • sudo: SUDO_ASKPASS=/tmp/askpass.sh sudo -A <cmd> — see SKILL.md "Remote sudo Without Interactive Password" section
  • 1Password: "NUT UPS Monitor (10.0.30.100)" — contains NUT service creds, NOT SSH creds

Hardware

  • 4 CPU, Ubuntu 24.04.4 LTS
  • 1× 256 GB SSD (sda) — system disk (ext4 + swap)
  • 9× 2.7 TB HDD (sdb–sdi):
    • sdb, sdc, sdg, sdh, sdi → ZFS pool members
    • sde, sdf → Ceph BlueStore OSDs (LVM)
    • sdd → ZFS pool member

ZFS Pool: pool01_n2_redundant

Property Value
Size 10.9 TB
Allocated 6.80 TB (62%)
Free 4.10 TB
Fragmentation 57%
Dedup 1.12x
Health ONLINE

ZFS Datasets (non-Docker)

Dataset Used Mountpoint Content
TimeMachine 951 GB /pool01_n2_redundant/TimeMachine macOS Time Machine backups
proxmox_nfs 963 GB /pool01_n2_redundant/proxmox_nfs NFS share for Proxmox
homes 455 GB /pool01_n2_redundant/homes User home directories
Download 369 GB /pool01_n2_redundant/Download Downloads
HomeAssistant 33.2 GB /pool01_n2_redundant/HomeAssistant HA config backup
proxmox_storage 1 GB /pool01_n2_redundant/proxmox_storage Nearly empty
proxmox_iscsi ~0 /pool01_n2_redundant/proxmox_iscsi Empty
proxmox_isci ~0 /pool01_n2_redundant/proxmox_isci Empty (typo dataset)
Frigate ~0 /pool01_n2_redundant/Frigate Empty — content deleted, dataset busy (can't zfs destroy yet)
docker 781 GB /var/lib/docker Docker volumes + overlay

Completed: Docker Service Cleanup (2026-06-30)

Removed: 10 Services (Images + Volumes + Bind Mounts)

All 10 services had NO running containers — only stale images, volumes, and bind-mount directories remained.

Service Images Removed Volumes Removed Bind Mounts Cleaned
Frigate 0.16.0-beta2, beta4, stable (~8.5 GB) frigate_config, frigate_storage, frigate_postgres-data, frigate_double-take /pool01_n2_redundant/Frigate/* (empty), /pool01_n2_redundant/HomeAssistant/frigate/ (clips+recordings)
Immich server + ML + ML-openvino (~3 GB) immich_pgdata, immich_model-cache —
Paperless 6 versions 2.9.0–2.17.1 (~8.3 GB) paperless_data, paperless_media, paperless_redisdata —
Scrypted latest (2.2 GB) scrypted —
Homebridge ubuntu-no-avahi (0.9 GB) homebridge /pool01_n2_redundant/homes/dominik/.homebridge/ (config + accessories)
Homegear stable (1.5 GB) homegear-data-etc, homegear-data-lib, homegear-data-log —
openHAB 3.4.5 (0.7 GB) openhab4_openhab4_addons, openhab4_openhab4_conf, openhab4_openhab4_userdata —
Node-RED latest (0.6 GB) nodered —
ioBroker latest (1.3 GB) iobrokerdata —
Jenkins jekyll-docker (0.5 GB) jenkins_home —

Total reclaimed: ~28 GB images + ~8.5 GB volumes + bind-mount data

Issues Encountered During Cleanup

  1. Docker daemon lockup — development-dind-1 (docker:20-dind) in restart loop made all docker commands hang. Fixed by sudo kill -9 $(pgrep dockerd) — systemd auto-restarted with fresh daemon. See SKILL.md "Docker Daemon Lockup" section.

  2. FALSE NEGATIVE verification — daemon returned empty docker images — After the first cleanup pass, docker images | grep -iE "pattern" returned empty (RC=1), suggesting all target images were deleted. I reported "ALL CLEAN". The user challenged this ("Hast du jetzt ALLE images gelöscht?"). After a daemon restart, docker images revealed 80+ images still present — only 3 of 10 target services (frigate beta2/beta4, immich-server, nodered) had actually been deleted. The degraded daemon had returned empty results despite images existing on disk. A second pass was needed to remove the remaining 12 images. Lesson: On potentially unstable Docker hosts, always sanity-check docker images | wc -l — if 0 but containers are running, the daemon is lying. See references/docker-zfs-cleanup.md "False Empty Results" section.

  3. ZFS slowness — docker rmi and docker volume rm extremely slow on ZFS backend. Required batching (5-8 items per command) with 120s timeouts. Multiple passes needed — first batch removed ~half, second batch cleared the rest.

  4. ZFS dataset busy — zfs destroy pool01_n2_redundant/Frigate failed with "dataset is busy" even after content deletion. Left empty (140K metadata) for later cleanup.

  5. SSH key deployment — Initial SSH access was password-only. Deployed hermes_pve_agent pubkey via sshpass + authorized_keys append.

  6. Two-pass image deletion required — First pass (background job) only deleted 3 of 19 target images before the daemon degraded. After daemon restart, a second docker rmi -f pass with all 15 remaining images succeeded in one command (120s timeout).

Remaining: Docker Containers (still running)

Container Image Ports Status
development-gitlab-1 gitlab-ce:15.8.1-ce.0 22,80,443,2224→22 DEPRECATED (Gitea replaces)
gitlab-runner gitlab-runner:alpine — DEPRECATED
development-dind-1 docker:20-dind 2375-2376 Restart loop — causes daemon lockups
Traefik traefik:latest 80→80 Reverse proxy
portainer portainer-ee:lts 8000,9443 Container management
spoolman-spoolman-1 spoolman:latest 7912→8000 3D filament mgmt
website-imkerei-website-1 ghost:5.3 2368 Imkerei blog
website-imkerei-proxy-1 nginx:alpine 80 Proxy for imkerei.familie-schoen.com
website-imkerei-bienen-proxy-1 nginx:alpine 80 Proxy for bienen.familie-schoen.com

Traefik Routing (Docker labels)

Hostname Target Entrypoint
git.familie-schoen.com GitLab (port 80) websecure (TLS) — NOW POINTS TO GITEA externally
registry.familie-schoen.com GitLab registry (port 5000) websecure
mgmt.familie-schoen.com Traefik dashboard web (basic auth)
imkerei.familie-schoen.com nginx proxy → Ghost web
bienen.familie-schoen.com nginx proxy → Ghost web

Completed: GitLab → Gitea Migration (2026-06-29)

All 12 GitLab repos migrated to Gitea (git.familie-schoen.com) via git clone --mirror + git push --mirror from 10.0.30.100. See references/gitlab-to-gitea-migration.md for technique.

Decommissioning Plan (Remaining Phases)

Phase 1: Data Migration (PARTIALLY COMPLETE)

✅ 10 Docker services removed (images + volumes + bind mounts) ⬜ Docker workloads → Proxmox LXC/VMs: Traefik, Ghost (Imkerei), Spoolman, Portainer (optional) ⬜ GitLab + Runner + DinD → remove (Gitea replaces ✅) ⬜ ZFS datasets → Ceph or NFS on Proxmox: TimeMachine (951 GB), homes (455 GB), Download (369 GB), proxmox_nfs (963 GB), HomeAssistant (33 GB) ⬜ Docker volumes for active services → migrate with containers

Phase 2: Ceph Integration

  1. Remove 9 HDDs from 10.0.30.100 (7 ZFS + 2 Ceph OSD)
  2. Install in n5pro (10.0.20.91) — currently 0 OSDs, 120 GB NVMe only
  3. Add as Ceph OSDs to Proxmox cluster
  4. Adjust CRUSH rules if needed (HDD tier)

Phase 3: Ceph OSD Decommission (on 10.0.30.100)

  1. Mark osd.8, osd.9 as out → data redistributes
  2. Wait for rebalance (cluster has 7.1 TB free — sufficient for 5.4 TB from 2 OSDs)
  3. ceph osd destroy + ceph osd purge
  4. Remove ceph-mon@ubuntu from cluster

Phase 4: Shutdown

  1. Export ZFS pool
  2. Shut down server
  3. Physically remove disks

Capacity Check

  • Ceph cluster: 12 TB raw, 5.1 TB used, 7.1 TB free
  • ZFS data to migrate: ~6.8 TB (TimeMachine + homes + Download + proxmox_nfs + docker + misc)
  • ⚠️ Tight capacity — may need to add disks to n5pro FIRST before draining ZFS data

Ceph Health Warning (as of 2026-06-29)

  • HEALTH_WARN: 3 OSDs slow operations, 1 stalled read in BlueFS, proxmox2 low on space
  • osd.6 on proxmox6 is destroyed (not contributing)
  • Should investigate before adding load
Reference in New Issue View Git Blame Copy Permalink