Auto-sync: 2026-10-08
This commit is contained in:
@@ -0,0 +1,44 @@
|
||||
---
|
||||
category: systems
|
||||
tags: [hermes-host, runner, systemd, backup, housekeeping, watchdog]
|
||||
updated: 2026-10-08
|
||||
---
|
||||
|
||||
# systems/hermes-agent-01 — Lokaler Hermes-Runner
|
||||
|
||||
## Identity
|
||||
- Debian 12, Kernel 6.1.0-53-amd64, Root-FS **79G ext4 (/dev/sda1)** — historisch knappe Resource!
|
||||
- Rolle: Hermes-Agent-Host (Telegram-Gateway, Crond, Playwright, avahi `hermes-agent-01.local`);
|
||||
Arbeitstiers laufen aus `~/hermes-agent` Worktree (`venv/bin/python -m hermes-cli main`).
|
||||
|
||||
## Storage-Governance (Stand 08.10., nach Krisenwoche)
|
||||
- **Worst Case erlebt:** 96–100 % volles Root-FS (10./08.10.) → korrupte write_file-Writes (mid-stream truncation); Syntax-Checker fing 2× korrupte Dateien.
|
||||
- Aktuell: **59 % belegt, 32 G frei** (von 96 % morgens).
|
||||
- **Permanenter Guards:**
|
||||
- journald: Drop-in `/etc/systemd/journald.conf.d/sizecap.conf` → `SystemMaxUse=200M` (1017M→199M)
|
||||
- uv-Cache: Guard-Cron `624a2f98fb10` (So 09:00, no_agent, Skript `clean_uv_cache.sh`, Threshold 3 GiB, leise wenn drunter)
|
||||
- **Podman bereinigt (08.10.):** 160 → 15 Images, ~23 G freigegeben (Inventory: `~/podman_inventory_2026-10-08.txt`). `schoenkitchen:latest` wurde versehentlich mitgeprünt (Namensregel-Fehler beim Pin-Container) → aus Git-Quelle rebuildet (`8c439711ceef`, 168 MB).
|
||||
- Producer der alten Krise: siehe Backup unten (tarballs in /tmp ohne Limit).
|
||||
|
||||
## Backup-Pipeline (produzierend seit 08.10.)
|
||||
- systemd `hermes-backup.timer` 04:00 UTC → `hermes-backup.service` (User debian) → `~/.hermes/scripts/backup-to-s3.sh`
|
||||
- Ziel: Norris S3 `rgw.nbg.nsc.noris.cloud`, Bucket `homelab-hermes-backup`, Retention 14 Tage; Creds via 1P Item `postgres-s3-backup` (Vault Kubernetes ESO).
|
||||
- **RCA 08.10.:** Exit 127 `aws: command not found` seit ≥27.09. (systemd-PATH ohne `~/.local/bin`) → Offsite TOOT + /tmp-Vollauf. Fix: Skript (absoluter AWS_BIN + `trap ... EXIT`-Cleanup) + Drop-in `/etc/systemd/system/hermes-backup.service.d/override.conf` (PATH + TimeoutStartSec 3600). Produktionslauf verifiziert (SUCCESS, S3-Readback 2.284.533.218 B). → PAT-015.
|
||||
- Belt-and-braces: manuelle Tarballs liegen zusätzlich auf NAS `10.0.20.100:/pool01_n2_redundant/backups/hermes-host/` (10 Stück, SHA256-verifiziert).
|
||||
|
||||
## Watchdogs (lokal registriert)
|
||||
- **NUT UPS Watchdog** Cron `4cbf03ccb4de` (1/min): rohes NUT-Proto gegen 10.0.20.92:3493, quiet-if-OK → Wiki systems/nut.
|
||||
- **uv Cache Guard** Cron `624a2f98fb10` (So 09:00).
|
||||
- HA-Backup-Alter Checker `2006a91fba21` (tägl. 12:00, Schwelle 30 h).
|
||||
|
||||
## Gotchas dieses Hosts
|
||||
- systemd/cron kennen `~/.local/bin` NICHT (aws, uv!) → Absolutpfade oder Drop-ins. PAT-015.
|
||||
- Root-FS-Volllauf = Korruptionsquelle Nr. 1 (write_file mid-stream truncation) — vor großen Writes Platz checken.
|
||||
- Terminal-Foreground verbietet `&`-Konstrukte; Async nur via `background=true`.
|
||||
- PDF-Generation via `/home/debian/.venvs/pdfkit/bin/python3` (PEP-668 sperrt System-pip).
|
||||
- `~/.hermes/state.db` 1,84 G + Snapshots 507 M — größte Nicht-Container-Posten.
|
||||
|
||||
## Related
|
||||
- [[systems/nut]] — UPS-Watchdog-Detail
|
||||
- [[reference/emhass-cascade]], [[systems/homeassistant]]
|
||||
- docs/solutions/bug-fixes/2026-10-08-hermes-offsite-backup-aws-path-rca.md
|
||||
@@ -77,6 +77,7 @@ modified: "2026-09-27"
|
||||
- **Kaskaden-Automation** (`1774430238377`, ON): EMHASS target → BYD (803 Auto default, 802 bei `target<=0 && pv<100`) + Marstek (Rest via `sensor.marstek_target_power`)
|
||||
- **Template Sensoren:** `sensor.smart_battery_target` (EMHASS), `sensor.marstek_target_power` (Plan A: smart_target − actual_BYD_discharge). `sensor.sma_target_power` ENTFERNT (Double-Counting Bug).
|
||||
- **Cron:** `a32005d62226` (no_agent, every 15m, runner script) ruft EMHASS Optimization auf.
|
||||
- **Shadow Phase-2 (2026-10-08):** Dual-Solve-Shadow live (Cron `d046e808f543` + Staleness-Watchdog `19aef2351273`); Deferrables als Phantom-Last erkannt und produktiv entfernt (CSV jetzt 17 Spalten). Details: [[reference/emhass-cascade-automation-architecture]]
|
||||
- **Backup:** `.bak.20261001_175804` für configuration.yaml + automations.yaml
|
||||
- **NFS-Backups (Fix 2026-10-07):** Supervisor-Mounts `Backup_NFS` (usage=backup) + `Medien_NFS` (usage=media) → **10.0.20.100:/pool01_n2_redundant/HomeAssistant** (vorher tote 10.0.30.100, verloren beim NAS-Crash-Reboot 03.10.). Volle automatische Backups ~03:27 UTC nightly. Monitoring: Hermes-Cron `2006a91fba21` (Alert bei Backup-Alter >30h). Lösungsdoku: `docs/solutions/bug-fixes/2026-10-07-ha-audit-nfs-mount-failure-after-nas-crash.md`
|
||||
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
title: "Home Assistant"
|
||||
category: systems
|
||||
tags: [homeassistant, smart-home, mqtt, automation]
|
||||
created: "2026-09-27"
|
||||
modified: "2026-09-27"
|
||||
---
|
||||
|
||||
# Home Assistant
|
||||
|
||||
> Smart Home Zentrale auf Proxmox. Steuerung, Automatisierung und Benachrichtigungen.
|
||||
|
||||
## Infrastruktur
|
||||
- **Host:** `10.0.30.10` (HAOS VM)
|
||||
- **SSH:** `hassio@10.0.30.10` (PW: 1P Vault "Hermes")
|
||||
- **URL:** `https://homeassistant.familie-schoen.com`
|
||||
- **Container:** `homeassistant` (Docker)
|
||||
|
||||
## MQTT
|
||||
- **Broker:** Mosquitto Add-on v7.1.1 auf localhost:1883
|
||||
- **HA MQTT User:** ehemals `opendtu` (BROKEN — ACL blockiert), gefixt
|
||||
- **ACL:** `/etc/mosquitto/acl` definiert `user homeassistant` + `user addons`
|
||||
- **Auth Plugin:** `go-auth.so` (files,http backends)
|
||||
|
||||
## Automations
|
||||
- **File:** `/config/automations.yaml` (35 Automations)
|
||||
- **Schreibmethode:** SSH → `docker exec homeassistant chmod 666`, danach restore 644
|
||||
- **Reload:** REST API mit JWT (HS256, signed from `/config/.storage/auth`)
|
||||
|
||||
### Frigate Einfahrt Notification (ID 1714065780832)
|
||||
- MQTT trigger `frigate/events` → filter `type=='new'` + cameras [Einfahrt,Terrasse] + labels [person,car,...]
|
||||
- Mode: `single`, cooldown 600s
|
||||
- Dedup: `input_text.frigate_last_event_id`
|
||||
- Flow: delay 10s → snapshot → notify iPhone → optional LLM Vision
|
||||
- Siehe [[systems/frigate]]
|
||||
|
||||
## Notify Services
|
||||
| Service | Status |
|
||||
|---------|--------|
|
||||
| `notify.mobile_app_iphone_dominik` | ✅ aktiv |
|
||||
| `notify.mobile_app_sarahs_iphone_app` | verfügbar |
|
||||
| `notify.mobile_app_ipad_2` | verfügbar |
|
||||
| `notify.mobile_app_sm_x205` | verfügbar |
|
||||
|
||||
## Entitäten
|
||||
- Kameras: `camera.einfahrt_2`, `camera.terrasse_2` (Suffix `_2` wegen verwaister Integration)
|
||||
- Motion: `binary_sensor.einfahrt_motion_2`, `binary_sensor.terrasse_motion_2`
|
||||
- Input Text: `input_text.frigate_last_event_id` (dedup storage, max 255 chars)
|
||||
|
||||
## Snapshots
|
||||
- Gespeichert: `/config/www/snapshots/{camera}_latest.jpg`
|
||||
- URL: `/local/snapshots/` (HTTP 200, keine Auth)
|
||||
|
||||
## JWT Auth
|
||||
1. `jwt_key` aus `/config/.storage/auth` lesen
|
||||
2. Client `Hermes_202606` (token id `1444b2c6757b4d66a6f8f8e5899b4e6a`)
|
||||
3. HS256 signieren, Bearer Header
|
||||
4. `?return_response=true` für Service-Call Responses
|
||||
|
||||
## Integrations
|
||||
- Frigate (MQTT)
|
||||
- LLM Vision (noris AI `gemma-4-31b-it`)
|
||||
- Tibber (Strom)
|
||||
- Marstek Speicher (VENUS-E, IP 10.0.50.113)
|
||||
- NUT (APC Back-UPS ES 700G via ms-a2-1, `10.0.20.92:3493`, User monuser; Sensors `sensor.apc_*`; Entry `5e1d2138235140eb9b476e9d6779c228`; deployed via `iac-homelab/epic-9-ups-nut`)
|
||||
- Various sensors (Xiaomi BLE, etc.)
|
||||
|
||||
## EMHASS Batteriesteuerung
|
||||
- **Endpoint:** `http://10.0.30.10:5000` (v0.18.4)
|
||||
- **Architektur:** EMHASS = Sole Optimizer. Virtuelle 16 kWh Batterie (BYD 11k + Marstek 5k).
|
||||
- **Config API:** `POST /set-config` überschreibt GESAMTE Konfig (kein Partial-Update!). Immer alle Felder senden. Siehe Solution Doc `integrations/2026-10-02-emhass-set-config-overwrite-pitfall.md`.
|
||||
- **Akuelle Config (2026-10-02):**
|
||||
- Capacity 16000Wh, Charge/Discharge 6700W, Efficiency 0.9
|
||||
- Target SoC 0.15 (15%), Min SoC 0.1, Max SoC 1.0
|
||||
- `set_battery_dynamic: True`, `set_use_battery: True`
|
||||
- SoC sensor: `sensor.total_soc`, Power sensor: `sensor.batteries_power_out_total`
|
||||
- **Kaskaden-Automation** (`1774430238377`, ON): EMHASS target → BYD (803 Auto default, 802 bei `target<=0 && pv<100`) + Marstek (Rest via `sensor.marstek_target_power`)
|
||||
- **Template Sensoren:** `sensor.smart_battery_target` (EMHASS), `sensor.marstek_target_power` (Plan A: smart_target − actual_BYD_discharge). `sensor.sma_target_power` ENTFERNT (Double-Counting Bug).
|
||||
- **Cron:** `a32005d62226` (no_agent, every 15m, runner script) ruft EMHASS Optimization auf.
|
||||
- **Backup:** `.bak.20261001_175804` für configuration.yaml + automations.yaml
|
||||
- **NFS-Backups (Fix 2026-10-07):** Supervisor-Mounts `Backup_NFS` (usage=backup) + `Medien_NFS` (usage=media) → **10.0.20.100:/pool01_n2_redundant/HomeAssistant** (vorher tote 10.0.30.100, verloren beim NAS-Crash-Reboot 03.10.). Volle automatische Backups ~03:27 UTC nightly. Monitoring: Hermes-Cron `2006a91fba21` (Alert bei Backup-Alter >30h). Lösungsdoku: `docs/solutions/bug-fixes/2026-10-07-ha-audit-nfs-mount-failure-after-nas-crash.md`
|
||||
|
||||
## Related
|
||||
- [[systems/frigate]] — NVR Integration
|
||||
- [[systems/noris-ai]] — LLM Vision Provider
|
||||
- [[reference/ip-map]] — IP Assignments
|
||||
- [[reference/emhass-cascade-automation-architecture]] — Kaskaden-Architektur Details
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: NUT / UPS Shutdown-Kaskade
|
||||
category: systems
|
||||
tags: [nut, ups, apc, proxmox, shutdown, epic-9]
|
||||
created: "2026-10-08"
|
||||
modified: "2026-10-08"
|
||||
---
|
||||
|
||||
# NUT / UPS Shutdown-Kaskade
|
||||
|
||||
## Überblick
|
||||
APC Back-UPS ES 700G (USB 051d:0002, SN `5B1940T54935␣␣` — 2 Padding-Spaces, nie per serial matchen!) hängt an **ms-a2-1 (10.0.20.92)**. NUT-Master dort, alle übrigen 7 PVE-Nodes + NAS sind Netclient-Slaves. IaC: `iac-homelab/epic-9-ups-nut/ansible/` (Commits bis `beee480`).
|
||||
|
||||
## Topologie
|
||||
- **Master ms-a2-1:** `nut-driver@apc` (usbhid-ups, vendorid/productid) → `upsd` 0.0.0.0:3493 → `nut-monitor` (upsmon MASTER) + `upssched`
|
||||
- **Users:** `monuser` (upsmon master, für HA) + `slavemon` (upsmon slave, für Cluster-Clients) — PWs in 1P Vault „Hermes": Items „NUT upsd monuser (ms-a2-1)" + „NUT upsd slavemon (cluster)"
|
||||
- **Slaves (7 PVE + NAS):** `MODE=netclient`, upsmon `slave` auf `apc@10.0.20.92`, FSD → `/sbin/shutdown -h now`, `nut-server` absent
|
||||
|
||||
## Shutdown-Kaskade (BY DESIGN)
|
||||
1. Stromausfall → UPS meldet ONBATT → upssched startet 60s-Timer
|
||||
2. Timer läuft ab (>60 s Dauerstörung) → `upsmon -c fsd` → FSD-Broadcast
|
||||
3. Alle Slaves sehen FSD → sofortiger geordneter Shutdown (PVE fährt Gäste via Guest-Agent)
|
||||
4. Master wartet `HOSTSYNC 45`s → fährt als letzter runter
|
||||
- **LOWBATT → sofortiger FSD** (kein Warten auf Timer)
|
||||
- Kurzunterbrechungen ≤60 s: Timer wird bei ONLINE gecancelt → kein Shutdown
|
||||
|
||||
## Betrieb & Verifikation
|
||||
```bash
|
||||
upsc apc@10.0.20.92 # Telemetrie (HA: sensor.apc_*)
|
||||
systemctl is-active nut-monitor # auf ALLEN 9 Maschinen active
|
||||
ansible-playbook -i inventory.ini site.yml # konvergentes Re-Deploy
|
||||
# Rollback (Kaskade entschärfen): deploy-nut.yml (disarmiert bewusst)
|
||||
```
|
||||
|
||||
## Grenzen / offene Risiken
|
||||
- **Runtime 277 s @ 76 % Last** — knapp für Nodes mit vielen Gästen. Remediation: `qm set <vmid> --startup down=60`, zweite UPS (LB>SPOF).
|
||||
- **n5pro:** pgdg-Repo deformiert (`dists/-pgdg`) — bricht apt-update; NUT-Deploy workaround über no-update_cache. Fix braucht GO.
|
||||
- Single-UPS = Single Point of Failure für den ganzen Cluster (auch NAS mit ZFS-Pool).
|
||||
- NAS `nut-driver.target not-found` = harmloses Phantom (Unit-Referenz ohne Paket).
|
||||
|
||||
## Monitoring
|
||||
- **NUT-Watchdog** (Hermes-Cron `4cbf03ccb4de`, jede Minute, no_agent): `~/.hermes/scripts/check_ups_health.sh` fragt upsd via rohem NUT-Protokoll ab (`GET VAR apc ups.status` — anonym möglich, kein Secret). Quiet bei OL; Instant-Alert bei Statuswechsel (OB/LB/upsd down), 10-min-Reminder bei Dauerstörung, einmalige Recovery-Info. State: `~/.hermes/cache/nut_watchdog.state`.
|
||||
|
||||
## Related
|
||||
- [[systems/homeassistant]] — NUT-Integration (Entry `5e1d2138…`, sensor.apc_*)
|
||||
- [[reference/ip-map]]
|
||||
- Bugfix-Doku: `docs/solutions/bug-fixes/2026-10-07-nut-ups-ms-a2-1-deployment.md`
|
||||
Reference in New Issue
Block a user