Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
@@ -0,0 +1,143 @@
|
||||
# Host 10.0.30.100 (hostname: ubuntu) — Full Service Inventory & Decommissioning Plan
|
||||
|
||||
Last updated: 2026-06-30 (session: Docker service cleanup — 10 services removed)
|
||||
|
||||
## Access
|
||||
|
||||
- **SSH**: Key auth with `~/.ssh/hermes_pve_agent` (deployed 2026-06-30). Previously password-only (`sshpass -p '<password>' ssh dominik@10.0.30.100`).
|
||||
- **sudo**: `SUDO_ASKPASS=/tmp/askpass.sh sudo -A <cmd>` — see SKILL.md "Remote sudo Without Interactive Password" section
|
||||
- **1Password**: "NUT UPS Monitor (10.0.30.100)" — contains NUT service creds, NOT SSH creds
|
||||
|
||||
## Hardware
|
||||
|
||||
- 4 CPU, Ubuntu 24.04.4 LTS
|
||||
- **1× 256 GB SSD** (sda) — system disk (ext4 + swap)
|
||||
- **9× 2.7 TB HDD** (sdb–sdi):
|
||||
- sdb, sdc, sdg, sdh, sdi → ZFS pool members
|
||||
- sde, sdf → Ceph BlueStore OSDs (LVM)
|
||||
- sdd → ZFS pool member
|
||||
|
||||
## ZFS Pool: pool01_n2_redundant
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Size | 10.9 TB |
|
||||
| Allocated | 6.80 TB (62%) |
|
||||
| Free | 4.10 TB |
|
||||
| Fragmentation | 57% |
|
||||
| Dedup | 1.12x |
|
||||
| Health | ONLINE |
|
||||
|
||||
### ZFS Datasets (non-Docker)
|
||||
|
||||
| Dataset | Used | Mountpoint | Content |
|
||||
|---------|------|------------|---------|
|
||||
| TimeMachine | 951 GB | /pool01_n2_redundant/TimeMachine | macOS Time Machine backups |
|
||||
| proxmox_nfs | 963 GB | /pool01_n2_redundant/proxmox_nfs | NFS share for Proxmox |
|
||||
| homes | 455 GB | /pool01_n2_redundant/homes | User home directories |
|
||||
| Download | 369 GB | /pool01_n2_redundant/Download | Downloads |
|
||||
| HomeAssistant | 33.2 GB | /pool01_n2_redundant/HomeAssistant | HA config backup |
|
||||
| proxmox_storage | 1 GB | /pool01_n2_redundant/proxmox_storage | Nearly empty |
|
||||
| proxmox_iscsi | ~0 | /pool01_n2_redundant/proxmox_iscsi | Empty |
|
||||
| proxmox_isci | ~0 | /pool01_n2_redundant/proxmox_isci | Empty (typo dataset) |
|
||||
| Frigate | ~0 | /pool01_n2_redundant/Frigate | Empty — content deleted, dataset busy (can't zfs destroy yet) |
|
||||
| docker | 781 GB | /var/lib/docker | Docker volumes + overlay |
|
||||
|
||||
## Completed: Docker Service Cleanup (2026-06-30)
|
||||
|
||||
### Removed: 10 Services (Images + Volumes + Bind Mounts)
|
||||
|
||||
All 10 services had NO running containers — only stale images, volumes, and bind-mount directories remained.
|
||||
|
||||
| Service | Images Removed | Volumes Removed | Bind Mounts Cleaned |
|
||||
|---------|---------------|-----------------|---------------------|
|
||||
| Frigate | 0.16.0-beta2, beta4, stable (~8.5 GB) | frigate_config, frigate_storage, frigate_postgres-data, frigate_double-take | /pool01_n2_redundant/Frigate/* (empty), /pool01_n2_redundant/HomeAssistant/frigate/ (clips+recordings) |
|
||||
| Immich | server + ML + ML-openvino (~3 GB) | immich_pgdata, immich_model-cache | — |
|
||||
| Paperless | 6 versions 2.9.0–2.17.1 (~8.3 GB) | paperless_data, paperless_media, paperless_redisdata | — |
|
||||
| Scrypted | latest (2.2 GB) | scrypted | — |
|
||||
| Homebridge | ubuntu-no-avahi (0.9 GB) | homebridge | /pool01_n2_redundant/homes/dominik/.homebridge/ (config + accessories) |
|
||||
| Homegear | stable (1.5 GB) | homegear-data-etc, homegear-data-lib, homegear-data-log | — |
|
||||
| openHAB | 3.4.5 (0.7 GB) | openhab4_openhab4_addons, openhab4_openhab4_conf, openhab4_openhab4_userdata | — |
|
||||
| Node-RED | latest (0.6 GB) | nodered | — |
|
||||
| ioBroker | latest (1.3 GB) | iobrokerdata | — |
|
||||
| Jenkins | jekyll-docker (0.5 GB) | jenkins_home | — |
|
||||
|
||||
**Total reclaimed**: ~28 GB images + ~8.5 GB volumes + bind-mount data
|
||||
|
||||
### Issues Encountered During Cleanup
|
||||
|
||||
1. **Docker daemon lockup** — `development-dind-1` (docker:20-dind) in restart loop made all docker commands hang. Fixed by `sudo kill -9 $(pgrep dockerd)` — systemd auto-restarted with fresh daemon. See SKILL.md "Docker Daemon Lockup" section.
|
||||
|
||||
2. **FALSE NEGATIVE verification — daemon returned empty `docker images`** — After the first cleanup pass, `docker images | grep -iE "pattern"` returned empty (RC=1), suggesting all target images were deleted. I reported "ALL CLEAN". The user challenged this ("Hast du jetzt ALLE images gelöscht?"). After a daemon restart, `docker images` revealed 80+ images still present — only 3 of 10 target services (frigate beta2/beta4, immich-server, nodered) had actually been deleted. The degraded daemon had returned empty results despite images existing on disk. A second pass was needed to remove the remaining 12 images. **Lesson:** On potentially unstable Docker hosts, always sanity-check `docker images | wc -l` — if 0 but containers are running, the daemon is lying. See `references/docker-zfs-cleanup.md` "False Empty Results" section.
|
||||
|
||||
3. **ZFS slowness** — `docker rmi` and `docker volume rm` extremely slow on ZFS backend. Required batching (5-8 items per command) with 120s timeouts. Multiple passes needed — first batch removed ~half, second batch cleared the rest.
|
||||
|
||||
4. **ZFS dataset busy** — `zfs destroy pool01_n2_redundant/Frigate` failed with "dataset is busy" even after content deletion. Left empty (140K metadata) for later cleanup.
|
||||
|
||||
5. **SSH key deployment** — Initial SSH access was password-only. Deployed `hermes_pve_agent` pubkey via `sshpass` + `authorized_keys` append.
|
||||
|
||||
6. **Two-pass image deletion required** — First pass (background job) only deleted 3 of 19 target images before the daemon degraded. After daemon restart, a second `docker rmi -f` pass with all 15 remaining images succeeded in one command (120s timeout).
|
||||
|
||||
## Remaining: Docker Containers (still running)
|
||||
|
||||
| Container | Image | Ports | Status |
|
||||
|-----------|-------|-------|--------|
|
||||
| development-gitlab-1 | gitlab-ce:15.8.1-ce.0 | 22,80,443,2224→22 | DEPRECATED (Gitea replaces) |
|
||||
| gitlab-runner | gitlab-runner:alpine | — | DEPRECATED |
|
||||
| development-dind-1 | docker:20-dind | 2375-2376 | Restart loop — causes daemon lockups |
|
||||
| Traefik | traefik:latest | 80→80 | Reverse proxy |
|
||||
| portainer | portainer-ee:lts | 8000,9443 | Container management |
|
||||
| spoolman-spoolman-1 | spoolman:latest | 7912→8000 | 3D filament mgmt |
|
||||
| website-imkerei-website-1 | ghost:5.3 | 2368 | Imkerei blog |
|
||||
| website-imkerei-proxy-1 | nginx:alpine | 80 | Proxy for imkerei.familie-schoen.com |
|
||||
| website-imkerei-bienen-proxy-1 | nginx:alpine | 80 | Proxy for bienen.familie-schoen.com |
|
||||
|
||||
## Traefik Routing (Docker labels)
|
||||
|
||||
| Hostname | Target | Entrypoint |
|
||||
|----------|--------|------------|
|
||||
| git.familie-schoen.com | GitLab (port 80) | websecure (TLS) — NOW POINTS TO GITEA externally |
|
||||
| registry.familie-schoen.com | GitLab registry (port 5000) | websecure |
|
||||
| mgmt.familie-schoen.com | Traefik dashboard | web (basic auth) |
|
||||
| imkerei.familie-schoen.com | nginx proxy → Ghost | web |
|
||||
| bienen.familie-schoen.com | nginx proxy → Ghost | web |
|
||||
|
||||
## Completed: GitLab → Gitea Migration (2026-06-29)
|
||||
|
||||
All 12 GitLab repos migrated to Gitea (git.familie-schoen.com) via `git clone --mirror` + `git push --mirror` from 10.0.30.100. See `references/gitlab-to-gitea-migration.md` for technique.
|
||||
|
||||
## Decommissioning Plan (Remaining Phases)
|
||||
|
||||
### Phase 1: Data Migration (PARTIALLY COMPLETE)
|
||||
✅ 10 Docker services removed (images + volumes + bind mounts)
|
||||
⬜ Docker workloads → Proxmox LXC/VMs: Traefik, Ghost (Imkerei), Spoolman, Portainer (optional)
|
||||
⬜ GitLab + Runner + DinD → remove (Gitea replaces ✅)
|
||||
⬜ ZFS datasets → Ceph or NFS on Proxmox: TimeMachine (951 GB), homes (455 GB), Download (369 GB), proxmox_nfs (963 GB), HomeAssistant (33 GB)
|
||||
⬜ Docker volumes for active services → migrate with containers
|
||||
|
||||
### Phase 2: Ceph Integration
|
||||
1. Remove 9 HDDs from 10.0.30.100 (7 ZFS + 2 Ceph OSD)
|
||||
2. Install in n5pro (10.0.20.91) — currently 0 OSDs, 120 GB NVMe only
|
||||
3. Add as Ceph OSDs to Proxmox cluster
|
||||
4. Adjust CRUSH rules if needed (HDD tier)
|
||||
|
||||
### Phase 3: Ceph OSD Decommission (on 10.0.30.100)
|
||||
1. Mark osd.8, osd.9 as `out` → data redistributes
|
||||
2. Wait for rebalance (cluster has 7.1 TB free — sufficient for 5.4 TB from 2 OSDs)
|
||||
3. `ceph osd destroy` + `ceph osd purge`
|
||||
4. Remove ceph-mon@ubuntu from cluster
|
||||
|
||||
### Phase 4: Shutdown
|
||||
1. Export ZFS pool
|
||||
2. Shut down server
|
||||
3. Physically remove disks
|
||||
|
||||
### Capacity Check
|
||||
- Ceph cluster: 12 TB raw, 5.1 TB used, 7.1 TB free
|
||||
- ZFS data to migrate: ~6.8 TB (TimeMachine + homes + Download + proxmox_nfs + docker + misc)
|
||||
- ⚠️ Tight capacity — may need to add disks to n5pro FIRST before draining ZFS data
|
||||
|
||||
### Ceph Health Warning (as of 2026-06-29)
|
||||
- HEALTH_WARN: 3 OSDs slow operations, 1 stalled read in BlueFS, proxmox2 low on space
|
||||
- osd.6 on proxmox6 is `destroyed` (not contributing)
|
||||
- Should investigate before adding load
|
||||
Reference in New Issue
Block a user