Auto-sync: 2026-10-08

This commit is contained in:
Dominik Schön
2026-10-08 22:00:16 +00:00
parent 8dca234bc6
commit b8d38c5640
13 changed files with 805 additions and 3 deletions
+3
View File
@@ -38,6 +38,8 @@
- [[systems/paperless]] — Dokumentenmgmt, OIDC, OCR via noris AI - [[systems/paperless]] — Dokumentenmgmt, OIDC, OCR via noris AI
- [[systems/sarah-hermes]] — VM107, zweites Hermes für Sarah (WebUI :8787, TG geplant) - [[systems/sarah-hermes]] — VM107, zweites Hermes für Sarah (WebUI :8787, TG geplant)
- [[systems/ubuntu-host]] — Physischer Ceph-Node 10.0.20.100, MSI B150M ECO, kein freier PCIe-x8-Slot (NVMe-Träger auf PEG) - [[systems/ubuntu-host]] — Physischer Ceph-Node 10.0.20.100, MSI B150M ECO, kein freier PCIe-x8-Slot (NVMe-Träger auf PEG)
- [[systems/nut]] — NUT/UPS Shutdown-Kaskade, APC ES 700G @ ms-a2-1, FSD-Cluster-Shutdown
- [[systems/hermes-agent-01]] — lokaler Hermes-Runner: Storage-Governance, Backup-Pipeline (RCA 08.10.), Watchdogs
## Concepts ## Concepts
- [[concepts/network-architecture]] — 10.0.X.Y Schema, VLANs - [[concepts/network-architecture]] — 10.0.X.Y Schema, VLANs
@@ -68,6 +70,7 @@
- [[patterns/placement-guardrails]] — 70% RAM-Deckel + Swap-Verbot als Prometheus-Rules (PAT-012) - [[patterns/placement-guardrails]] — 70% RAM-Deckel + Swap-Verbot als Prometheus-Rules (PAT-012)
- [[patterns/prometheus-config-rescue]] — grep -vn Gift/Crashloop, LXC-Rootfs-Direct-Mount-Rescue (PAT-013) - [[patterns/prometheus-config-rescue]] — grep -vn Gift/Crashloop, LXC-Rootfs-Direct-Mount-Rescue (PAT-013)
- [[patterns/ceph-dead-nvme-resurrection]] — PCI-Reset→lvchange→chown-Falle, Weight-Management bei Reintegration (PAT-014) - [[patterns/ceph-dead-nvme-resurrection]] — PCI-Reset→lvchange→chown-Falle, Weight-Management bei Reintegration (PAT-014)
- [[patterns/systemd-path-blindness]] — Exit-127-Klasse: ~/.local/bin unsichtbar im systemd-PATH, trap-EXIT-Cleanup (PAT-015)
- [[patterns/llm-backend-hang-diagnosis]] — Modell- vs. Infra-Defekt: code=000 bei erfolgreichem Connect (PAT-015) - [[patterns/llm-backend-hang-diagnosis]] — Modell- vs. Infra-Defekt: code=000 bei erfolgreichem Connect (PAT-015)
- [[patterns/skill-impact]] — Skill Modification Audit Trail - [[patterns/skill-impact]] — Skill Modification Audit Trail
+25
View File
@@ -1,5 +1,11 @@
# Memory Log # Memory Log
## [2026-10-08] emhass-phase2-shadow | Shadow-Dual-Solve live + Deferrables entfernt
- Shadow Phase-2 live: Cron `d046e808f543` (*/15m, silent-on-healthy) + Staleness-Watchdog `19aef2351273` (:35, >45 min -> Alert). Ledger `~/shadow_phase2/ledger.jsonl`, Report `~/scripts/report_phase2.py`.
- Bewiesene CSV-Gesetze: +P_batt=Entladung, P_grid signed (Export NEGATIV), Deferrable-Spalten=Phantom-Last (+12 kWh Summeninvarianz, keine Consumer).
- PRODUKTIV: `number_of_deferrable_loads` 2->0 (Direct-File-Edit + Addon-Restart im :50-Fenster, Backup `config.json.hermesbak-20261008`); Beweis: Incumbent-CSV jetzt 17 Spalten.
- Solution Doc: `docs/solutions/workflows/2026-10-08-emhass-phase2-shadow-methodology.md`
## [2026-10-02] hermes-gui-modelpicker | Stale-Katalog im Modelpicker behoben ## [2026-10-02] hermes-gui-modelpicker | Stale-Katalog im Modelpicker behoben
- Symptom: Modelpicker der GUI (Hermes Desktop via hermes-serve :9119) zeigt je Profil andere noris-Modellzahlen — personal-coach nur **25**, infra-sre **40** (3 Phantome), Rest 37-38. - Symptom: Modelpicker der GUI (Hermes Desktop via hermes-serve :9119) zeigt je Profil andere noris-Modellzahlen — personal-coach nur **25**, infra-sre **40** (3 Phantome), Rest 37-38.
- Root Cause: `$HERMES_HOME/provider_models_cache.json` (disk-Cache des `/v1/models`-Katalogs). `cached_fetch_api_models()` liefert bis **7 Tage** Altes aus (TTL 1h, stale-while-revalidate) und revalidiert im **Daemon-Thread** — der in kurzlebigen Prozessen / nie geöffneten Picker-Profilen nie landet. Kein Fehler sichtbar. - Root Cause: `$HERMES_HOME/provider_models_cache.json` (disk-Cache des `/v1/models`-Katalogs). `cached_fetch_api_models()` liefert bis **7 Tage** Altes aus (TTL 1h, stale-while-revalidate) und revalidiert im **Daemon-Thread** — der in kurzlebigen Prozessen / nie geöffneten Picker-Profilen nie landet. Kein Fehler sichtbar.
@@ -325,3 +331,22 @@
- Endstand: 3/3 Synced, Primary, MaxScale alle Server Running - Endstand: 3/3 Synced, Primary, MaxScale alle Server Running
- Solution Doc: docs/solutions/bug-fixes/2026-09-29-vm302-zombie-postmigration-galera-rejoin.md - Solution Doc: docs/solutions/bug-fixes/2026-09-29-vm302-zombie-postmigration-galera-rejoin.md
- Zusätzlich: Home Assistant Core Restart via REST API erfolgreich (Version 2026.9.3, RUNNING) - Zusätzlich: Home Assistant Core Restart via REST API erfolgreich (Version 2026.9.3, RUNNING)
## 2026-10-08 — NUT-Shutdown-Kaskade clusterweit (epic-9 Finale)
- GO: NAS-Treiber entfernen + nut-monitor-Kaskade aktivieren
- Master ms-a2-1: upsmon MASTER + upssched (ONBATT>60s → FSD, LOWBATT → Sofort-FSD, HOSTSYNC 45)
- 7 PVE-Slaves + NAS als NETCLIENTS, NAS-Altlast gepurged (Crashloop tot)
- FSWEEP: 9/9 nut-monitor active+enabled; HA-NUT-Entry bleibt loaded
- Commits adb4dcc/beee480; Neue Wikiseite systems/nut.md + Index
- Nebenfunde: n5pro pgdg-Repo kaputt (braucht GO); Hermes-Root-FS war 100% voll (Backups → NAS)
- Quality-Gate-Lektion: Syntax-Check fing 2 korrupte Writes (Volldisk-Ursache)
## 2026-10-08 — Disk-Audit hermes-agent-01 + Offsite-Backup-RCA
- Root-FS 96%→72% (21,7G frei): 10x hermes-backup Tarballs (18,9G) nach SHA256-Verifikation (10/10 identisch) auf NAS geloescht
- RCA Offsite-Backup: hermes-backup.service Exit 127 'aws: command not found' (systemd-PATH ohne ~/.local/bin) seit >=27.09. TA EGLICH; Offsite war TOOT
- Fix: Skript (AWS_BIN absolut + trap EXIT-Cleanup), systemd-Drop-in (PATH + TimeoutStartSec 3600); Prod-Lauf SUCCESS, S3-Objekt 2.284.533.218B verifiziert
- n5pro pgdg.list disabled (syntaktisch defekt, nie genutzt) -> apt update gruenn
- NUT-Watchdog-Cron 4cbf03ccb4de (1/min, rohes NUT-Proto, quiet-if-ok, 10min-Throttle) 7/7 Tests
- Naechste Aufraeum-Hebel (warten auf GO): podman prune 23,4G, uv-Cache 9,4G, playwright-alt ~1G, journal 0,8G, apt-cache 0,6G
- Welle 2: journald SystemMaxUse=200M (drop-in, permanent; 1017M->199M), uv cache clean (-70,8G logisch, FS 72->59%), Guard-Cron 624a2f98fb10 (So 09:00, 3GiB threshold)
- Closing-the-Loop: PAT-015 systemd-path-blindness (patterns/), Wiki systems/hermes-agent-01 neu, INDEX.md normalisiert (28 docs, beide Okt-Docs eingepflegt), skill-impact-Ledger 2 Rows gebucht, Hindsight Wave-2
+346
View File
@@ -0,0 +1,346 @@
# Memory Log
## [2026-10-02] hermes-gui-modelpicker | Stale-Katalog im Modelpicker behoben
- Symptom: Modelpicker der GUI (Hermes Desktop via hermes-serve :9119) zeigt je Profil andere noris-Modellzahlen — personal-coach nur **25**, infra-sre **40** (3 Phantome), Rest 37-38.
- Root Cause: `$HERMES_HOME/provider_models_cache.json` (disk-Cache des `/v1/models`-Katalogs). `cached_fetch_api_models()` liefert bis **7 Tage** Altes aus (TTL 1h, stale-while-revalidate) und revalidiert im **Daemon-Thread** — der in kurzlebigen Prozessen / nie geöffneten Picker-Profilen nie landet. Kein Fehler sichtbar.
- **Key-Falle:** Cache-Key ist `custom:<base_url>` (neu: `…#<fp>`), **nicht** der Provider-Slug. `cached_provider_model_ids("noris")` ist die falsche Funktion (schreibt fremden Key, liefert 19 statt 37).
- Fix: `~/.hermes/scripts/refresh-picker-catalog.py` (Dry-Run default, `--apply`, mit Backup) → **alle 5 Profile 37 Modelle, 0 fehlend**. Kein Restart nötig (Cache wird pro Call neu gelesen).
- E2E-verifiziert über `build_model_options_payload()` — die eine Quelle für Desktop-Picker, `/api/model/options`, TUI `model.options` und Bots-Profile-Editor. Config-Dateien unberührt (mtime 2026-09-27/29).
- Doku: `docs/solutions/integrations/2026-10-02-hermes-gui-modelpicker-stale-catalog.md`; Skill `hermes-agent` gepatcht.
- **Offen (nicht angefasst):** `vllm/qsu/glm-5-3-flash` = Default im default-Profil, **nicht in /v1/models**, konsistent **HTTP 503** → tot. 3 Profile auf `vllm/release/glm-5-2`, infra-sre auf `vllm/moonshotai/kimi-k2.6` (**403 model_blocked**). Von 37 gelisteten Modellen sind nur ~20 chatfähig (TTS/Reranker/Embedding liefern 400).
## [2026-10-02] sarah-hermes-timeout | glm-5-2 Upstream-Hang → Modellwechsel
- Symptom: Sarahs Hermes (VM107) läuft in Turn-Timeouts. `errors.log`: `httpx.ReadError: Broken pipe` → `openai.APIConnectionError` → 3 Retries → "unresponsive … 5 stale attempts".
- Root Cause: **`vllm/release/glm-5-2` hängt upstream bei ai.noris.de** — TCP-Connect OK (0.69s), aber 0 Bytes, stream + non-stream, von VM107 UND extern reproduzierbar.
- Ausgeschlossen: VM-Ressourcen (Load 0.11, 2.1GB frei), Netz (DNS 0.84s/TCP 0.69s/TLS 1.3s), Key-Scope (4 andere Modelle mit demselben Key = 200).
- Fix: Default `model.default` + `providers.noris.model` → `vllm/qsu/deepseek-v41-flash`. Backup `config.yaml.bak.20261002_075455`. Restart 07:55:41 (mtime config 07:55:23 < Start ✓).
- **E2E verifiziert:** Session meldet neues Modell, Turn completed in 15s, Transcript `user::…PONG` → `assistant::PONG`. Test-Session danach gelöscht (7 übrig).
- Nebenbefunde: `auxiliary.vision.model = release/gemma-4-31b-it` → **403 model_blocked**. `fallback_providers: []` = kein Fallback (Amplifier). Unit-File `0644` mit Passwort+API-Key im Klartext.
- Neu: Pattern [[patterns/llm-backend-hang-diagnosis]] (PAT-015), Wiki [[systems/sarah-hermes]] Pitfalls 4-6.
## [2026-10-02] emhass-cascade-fix | Kaskaden-Bugfix + Config-Pitfall + Overnight-Hold
- Double-Counting Bug behoben: `sensor.sma_target_power` entfernt, `sensor.marstek_target_power` nutzt jetzt echte BYD-Werte (discharge − charge) statt Forecast-Split. Summe BYD+Marstek = EMHASS target ✓.
- BYD 802 Hold-Bedingung erweitert: `target < 0` → `target <= 0` (Hold bei target=0 eingeschlossen).
- EMHASS `battery_target_state_of_charge` 0.30 → 0.15 (65% Overnight-Hold war zu hoch).
- **Pitfall:** `POST /set-config` überschreibt GESAMTE Konfig, nicht nur gesendete Felder. Reset auf Defaults (5kWh, 1000W, dynamic=false). Musste alle 14 Parameter zusammen resendieren.
- Solution Doc: `docs/solutions/integrations/2026-10-02-emhass-set-config-overwrite-pitfall.md`
- Wiki: `reference/emhass-cascade-automation-architecture.md` neu, `systems/homeassistant.md` EMHASS-Sektion ergänzt.
- Hindsight: Config + Pitfall semantisch indexiert.
## [2026-09-30] k8s-cp01-relief-descheduler | Workload-Migration + Descheduler-Deployment
- cp-01 bei 93% RAM → hindsight-api + hindsight-postgres + paperless nach worker-05 migriert (cordon/delete/uncordon). cp-01 RAM 93%→43%.
- ArgoCD selfHeal belebte alte ReplicaSets mit nodeSelector wieder → manuell auf 0 skalieren bis konvergiert.
- hindsight-api Live-Pin (Sep-17-Hotfix) war nie in Git → Live-Patch nötig.
- Descheduler v0.30.0 deployt (raw Manifests, nicht Helm — Chart hat params/args-Bug + falsche API-Version). Policy: LowNodeUtilization (30/50/20 → 70/70/60), 2m Intervall, nodeFit auf Profile-Ebene.
- RBAC-Lektion: Descheduler braucht `list/watch` auf `namespaces` in ClusterRole.
- ArgoCD repoURL: `ssh://git@10.0.30.200:22/...` (mit `git@` User-Prefix, wie alle anderen Apps).
- Solution Doc + INDEX + Wiki rke2-kubernetes.md aktualisiert. Commits b955d7e→8c1ecff.
## [2026-09-30] ceph-monitoring-hardened | Mgr-Failover-Resistenz + SMART-Korrektur
- Ceph-Scrape war Single-Target am aktiven mgr (10.0.20.60) → jeder mgr-Failover hätte ALLE Ceph-Alerts stumm gemacht (Standbys: 200/empty-body). Fix: Union über alle 5 mgr-Kandidaten (.50,.60,.70,.91,.92:9283) in prometheus.yml — aktiver mgr liefert, Standbys harmlos leer. TOTAL DOWN TARGETS: 0. Auch 10.0.20.70:9100 (node_exporter mit ceph-fill-collector) in Scrape-Ziele aufgenommen.
- Zwischenfail: `aliases:`-Field in Alert-Regel ungültig (RuleNode kennt das nicht) → Prometheus Fatal. Behoben durch Entfernen + force-recreate.
- SMART-Korrektur: Kingston SFYRDK2000G (osd.2-Träger) ist GESUND (wear 6 %, media_errors 0, spare 100 %, PoH 1469) — frühere "Austausch-Kandidatin"-These RETRAKIERT. Vorfall war Controller-/Fabric-Ebene. Neu beobachten: Temps 70/78 °C, T1-throttle 3×.
- PAT-014 erweitert (Smart-Log-Abschnitt), ceph-cluster.md korrigiert, Commit cc2f237.
## [2026-09-30] ceph-osd-resurrection | NVMe-Revival osd.2 + osd.3-Weight-Drama + ubuntu-Hosts-Fix (PAT-014)
- osd.2 (ubuntu, Kingston SFYRDK2000G) tot: NVMe-Controller state=dead, VG verschwunden, errno-5. Revival-Kette: PCI remove/rescan (Ctrl kam als nvme2 zurück!) → pvscan --cache → lvchange -ay -K (stale DM-Table) → dd-Lesetest 1,4 GB/s → chown ceph:ceph am Mapper-Device (udev-Falle) → ACTIVE, Weight 1.0, 495 GiB. Drive = Replacement-Kandidatin. *(Korrektur später am selben Tag: SMART-Audit → gesund, kein Austausch; siehe nächster Eintrag.)*
- osd.3-Lehre: re-in mit Weight 1.0 → instant 96 % voll → backfillfull, 13 PGs blockiert. Korrekt: `crush reweight osd.3 0.05` + in. Cluster 17/17 up/in, Degraded 0,78 % fallend.
- ubuntu-Hosts-Fix: `10.0.20.100 ubuntu` in /etc/hosts aller 8 PVE-Nodes → GUI-500 „hostname lookup failed" behoben.
- Doc: bug-fixes/2026-09-30-ceph-osd2-nvme-resurrection-osd3-drain.md, Pattern PAT-014.
## [2026-09-30] watchdog-self-healed | Guardrail-Rollout begleitender Incidents (PAT-013)
- Beim Guardrail-Rollout entdeckt: Phantom-ICMP-Targets .10/.20 lebten NOCHMAL im blackbox_icmp-Abschnitt (erste Bereinigung traf nur node_exporter-Liste) → HostUnreachableICMP-Alerts. Entfernt, 12 ICMP-Probes, alle grün.
- Eigener Fehler: `grep -vn`-Rewrite fügte Zeilennummer-Präfixe ("1:global:") in prometheus.yml ein → Prometheus Crashloop. **Rescue-Pfad etabliert:** CT-Rootfs direkt am Host mounten (`mount /dev/rbd2 /mnt/...` — rbd2 = CT141-Disk), Fix außerhalb des pct-Kanals, Container recreation. Prometheus wieder HEALTHY, 28 Targets, DOWN=[], alle Rules ok.
- **Lessons**: (a) NIEMALS `grep -n`-Ausgaben als Rewrite-Source verwenden; (b) LXC-Rootfs-Host-Mount = universeller Rescue-Kanal wenn pct exec zickt; (c) nach jedem Rewrite YAML-validieren BEVOR recreate.
- Doc: bug-fixes/2026-09-30-prometheus-yaml-prefix-crashloop.md, Commit <SHA>.
## [2026-09-30] guardrails-live | Placement-Policies maschinell erzwingbar (PAT-012)
- Zwei neue Rules in CT141 (`placement_guardrails`): PVEPlacementCeilingBreached (>70% RAM, 15m) + HVResidentSwapNonzero (>100MiB, 10m). Alle Rules health=ok.
- Baseline: proxmox3 80,6% / p4 79,3% / p5 75,5% / p6 72,6% ÜBER Deckel → Warnbursts erwartet (Rebalancing-Backlog Richtung ms-a2-1/-2 mit 74%/Headroom).
- Doc: architecture/2026-09-30-placement-guardrails-ram-ceiling.md, Commit 66f2172.
## [2026-09-30] webhook-fixed | PVE→Telegram Notification-Pipeline repariert (PAT-011)
- Ursachenkette (dreifach gestapelt): Endpoint-Drift .99→.141 (Bridge wohnt in CT141), fehlender `body`-Attr (Leere Posts → 400), unescapte Handlebars-Interpolation (Apostrophe/Multiline → invalides JSON).
- Fix: URL korrigiert, Body via pvesh (BASE64-Pflicht!) mit `{{escape title}}`/`{{escape message}}` (Space-Syntax, NICHT Colon). Offizieller Test grün, Bridge loggt POST /pve 200.
- Diagnose-Technik: Mini-Sniffer (temp URL-Redirect, Bytes kapern, URL restaurieren) enthüllte exakten Wire-Body.
- Doc: bug-fixes/2026-09-30-pve-webhook-notifications-drift-base64-escape.md, Commit 38622ff. Residue ge cleaned (HV+/tmp+lokales).
## [2026-09-30] legacy-alert-cleanup | Alerts bereinigt, mysqld_exporter VM300 nachgezogen
- proxmox3 /boot/efi 100%: 17 alte Kernel-Pakete gepurged (-17/-19/-12 behalten) → 30%.
- Phantom-Targets .10/.20 entfernt, ICMP→TCP-Probe für Offsite-PBS (ICMP upstream gefiltert) → 30 Targets.
- VM300 mysqld_exporter 0.15.1 nachdeployt (war aspirational Target): GitHub-Download via qm guest exec+b64, exporter-User (vorgeschädigte exporter@localhost-Shadow!) PW-Align, UFW 9104←10.0.30.141. Alle 3 Galera-Exporte UP.
- Cleanup: alle /tmp-Skripts (HV+CT141+VM300), lokale Scratch-Dirs entfernt.
## [2026-09-30] remediation-complete | Vorfall-Nacharbeiten: GPU restored, PSI/OOM-Alerts live, p6 entlastet
- GPU (VM102/ms-a2-2): hostpci0 ohne x-vga restauriert → renderD128 lebt. **x-vga=1 bricht AMD-Passthrough** (SeaBIOS Shadow-ROM → VBIOS-Zugriff tot, amdgpu -22). Doc-Update in 2026-07-21-amd-gpu-passthrough-rombar.md.
- Monitoring (CT141): Regelgruppe `pressure_alerts` (5 Regeln: PSI mem waiting/stalled, oom_kill, Swap-Churn, MajFault-Storm) live; node_exporter auf ms-a2-1/-2 nachinstalliert (Blindspots!), 32 Targets. **LXC-Bindmount-Inode-Trap:** sed-i/In-Place-Rewrites unsichtbar für Docker bis Force-Recreate → neues Doc.
- proxmox6: VM301 → proxmox7 (HA relocate, na-vm301 live verifiziert). Swap 6,3G→0B, RAM 8,4/15G. Nur noch CT151 auf p6. Altlast-Alerts sichtbar geworden (NodeDown .10/.20 Phantoms, proxmox3 /boot/efi 100%, ICMP 213.95.54.60) — Cleanup offen.
- Docs: bug-fixes/2026-09-30-prometheus-lxc-bindmount-inode-trap.md (neu), INDEX.md aktualisiert.
## [2026-09-30] incident-fix | worker-05 Freeze: proxmox6 Doppel-OOM → Zombie-VM (PAT-010)
- Symptom: KubeDaemonSetRolloutStuck (Traefik DS misscheduled=1), worker-05 NotReady 19h
- Root Cause: proxmox6 RAM-Oversubscription (~47.8 GB alloc / 16 GB, 7.3/8 GB Swap) → OOM-Killer tötete kvm (VM102) 2× (29.09. 18:43 + 20:44); 2. Revival = Zombie (QEMU running, Gast inert, RSS 239MB/12GB)
- Diagnose-Signatur: `qm status --verbose` RSS-Kollaps + tote Guest-Agent + statischer Tap-TX
- Fix: Laya CT152 → proxmox3 (Offline-Move 2s, shared RBD) → Druck raus; qm stop/start VM102 → HA replatzierte auf ms-a2-2 (60 GB frei, GPU-fähig!); uncordon → Node Ready, Traefik-Orphan self-reconciled, Alert cleared
- Drift-Fund: frühere node-affinity `na-vm102` (NICHT ms-a2-2) existiert live nicht mehr
- Offen: proxmox6 strukturell eng (28 GB alloc); PSI/OOM-Alerts pro PVE-Node fehlen komplett; GPU(renderD128)-Verifikation auf ms-a2-2 nach Return
- Docs: docs/solutions/bug-fixes/2026-09-30-proxmox6-oom-frozen-vm102-worker05.md, patterns/pve-oom-frozen-guest.md (PAT-010)
## [2026-09-30] deployment | Sarah-Hermes VM107 — zweite Hermes-Instanz live
- VM107 (n5pro, 10.0.30.66) via Tofu epic-8 deployed; Docker+UFW via Ansible (epic-7-Stil)
- hermes-webui Single-Container :8787, LAN-only, Password-Auth; Secrets in 1P (sarah-hermes-webui, sarah-hermes-noris-key)
- E2E verifiziert: Login 200, Agent-Antwort via vllm/release/glm-5-2 @ ai.noris.de
- PITFALL: hermes-agent pyproject pinnt Core-Deps hinter python_version>='3.14'-Markers → auf Python 3.12 installiert `pip install -e .` NULL Deps ("AIAgent not available"). Fix: manuelle Pin-Installation + Missing-Import-Loop bis `import run_agent`
- Backup: täglich 23:00 Job backup-2e8a34e3-66cb (all=1) deckt VM107; manueller Verify TASK OK 48s
- Wiki: systems/sarah-hermes.md, ip-map.md erweitert; iac-homelab commits 0d5f017/1ce4816
- OFFEN: Telegram-Bot blockiert auf BotFather-Token (Sarah/Dominik)
## [2026-09-29] bug-fix | KubeAPIErrorBudgetBurn — Ceph-CSI resize loop from missing controller-expand-secret
- Root Cause: `ceph-flash` + `ceph-hdd-replica` StorageClasses created manually WITHOUT `controller-expand-secret-name/namespace` params
- CNPG PVC resize (50→100Gi, 10→20Gi) triggered infinite CSI resizer retry loop ("provided secret is empty") → API server write pressure → etcd DeadlineExceeded → Handler timeout 5xx
- Cordoning cp-03 amplified: CNPG switchover → operator reconcile storm (~60/min)
- Fix: (1) Delete+recreate SCs with all secret refs, (2) Patch 6 PVs with controllerExpandSecretRef, (3) Restart CSI resizer, (4) Commit SCs to Git `clusters/main/storage/`, (5) Move Gitea off unstable worker-05
- Worker-05 cordoned (repeated reboots, likely hypervisor issue on ms-a2-2)
- Architectural risk documented: etcd on Ceph RBD (~20ms WAL fsync all CP nodes)
- Solution doc: docs/solutions/bug-fixes/2026-09-29-kubeapi-error-budget-burn-ceph-csi-resize-loop.md
## [2026-09-27] bug-fix | Gitea CSI RBAC Fix + ArgoCD Verknüpfungs-Audit
- Root Cause: Ceph CSI RBD `csi-attacher` fehlte `storage.k8s.io/csinodes` Berechtigung → VolumeAttachments pending → Gitea Pod 4+ Tage Init-crash
- Fix: ClusterRole `ceph-rbd-external-attacher-runner` patched, provisioner Pods neu gestartet → alle 24 VolumeAttachments `true`
- Gitea Pod + schoenkitchen runner durch Pod-Delete wiederhergestellt
- ArgoCD Audit: 17/25 Apps via Gitea SSH, 20/25 Synced+Healthy
- Known Issues: `gitea`/`authelia` Health=Progressing (Ingress-LB-IP Gap), `immich` doppelt verwaltet, `gitea-config` Deployment Drift
- `DEFAULT_ACTIONS_URL=https://gitea.com` deprecated → Helm override auf `self` empfohlen
- Wiki `systems/gitea.md` aktualisiert: Runner Status, Incident, ArgoCD Issues
- Solution Doc: `docs/solutions/bug-fixes/2026-09-27-gitea-csi-rbac-volumeattachment-fix.md`
## [2026-09-27] architecture | Memory Layer Restructuring + Laya Session-Nutzung
- USER.md bereinigt: Infra-Fakten entfernt, nur noch User-Preferences/Safety-Rules (1.087/1.375 chars)
- MEMORY.md ausgedünnt: 2.145→1.664 chars, alle Infra-Details zeigen auf Wiki-Seiten mit `→Wiki` Pointern
- 4 neue Wiki-Seiten: systems/noris-ai, systems/frigate, systems/homeassistant, systems/paperless
- Neues Concept: concepts/memory-layer-architecture — Entscheidungsbaum "was wohin gehört"
- Hindsight Audit: massiv überladen mit veralteter Infra-Topologie. Going-Forward-Policy: nur noch semantische Pointer + Entscheidungen
- LCM gesund: 7.590 messages, 34 DAG nodes, 21.2:1 compression ratio
- User-Preference: Laya künftig in normalen Sessions nutzen (choice/noul/score)
## [2026-09-27] architecture | Laya Email-Organizer Migration + Session-Nutzung
- Rechnungen-Organizer Cron `f773f8c23230` von LLM-Agent → `no_agent` Script mit Laya migriert
- 12 Kategorien, 3-Schichten-Safety (Confidence-Gate + Subject-Validierung + Move-Erfolg)
- Globale Ordner (Rechnungen/Bestellungen/Gutschriften/Gutscheine) statt Monatssortierung
- Wiki-Seite `systems/laya.md` erstellt mit Usage Guide für Session-Nutzung
- User-Preference: Laya künftig in normalen Sessions nutzen (choice/noul/score)
- Solution Doc: `docs/solutions/architecture/2026-09-27-laya-email-organizer-migration.md`
## [2026-09-17] workflow | CT108-Endausbau: Census, Ghost-Router CT99999, Alias-Repair, Stop
- **Census (auth):** K8s=25 / CT108=31 / gemeinsam=23 — 22 Tips identisch, dominik/memory=K8s-Superset (enthält CT-Tip de97cd6d), nur-CT=8× PoC-Müll. Archiv: 29 Bare-Bundles 143 MB unter `/home/debian/git-archive/ct108-final/` (2 Failures = legitim leere Repos).
- **Ghost-Router enttarnt:** CT99999 (Traefik-LXC auf proxmox7, 10.0.60.10) terminiert TLS für *.familie-schoen.com und forwardet plain HTTP an .203. `git.familie-schoen.com` → .105:3000 WAR der letzte Live-Konsument von CT108; `git.schoen.codes` lief längst per Double-Hop aufs K8s. Fix: gitea-service-Upstream → .203 (Backup `explicit-http.yml.bak-hermes-20260917`) + Alias-Host im K8s-Ingress (Commit `9e9b6ee`). Beide Hostnamen jetzt v1.27.0.
- **Stop vollzogen (genehmigt):** `pct stop 108` auf ms-a2-2 (10.0.20.93) — connection-refused-Beweis, Fleet 22/25 Synced, Runner unversehrt. Wiki-Lügen korrigiert (ip-map behauptete „stopped" seit Wochen).
- **Lessons:** 1P-SA braucht je Call `--vault`; `op read --reveal` existiert nicht (stdout=Secret); `op item get --reveal` maskiert nur Display (JSON-Captures intakt); `/repos/search` = `{ok,data}`-Envelope; blankes Token erzeugt glaubwürdig LEERE Census (Fast-Fehlentscheidung „K8s hat nur 2 Repos").
- Docs: `docs/solutions/workflows/2026-09-17-ct108-full-decommission-census-router-topology.md` · Wiki: systems/gitea.md, reference/ip-map.md
- Offen (je Freigabe): Zombie-Secret `argocd-repo-credentials` löschen; immich-Zwillings-App (toter rendered-Dump vom 01.08., Live gehört immich-config) entfernen.
## [2026-09-17] fix | Merge-Day-Kampagne abgeschlossen: Konvergenz + Autosync-Rennen + Live-State-Arbitrage
- **Konvergenz DONE:** Merge `d76d3e8` (fork 46fa168, 24.07.) + Fixups `4987341`/`8c741c5` auf BEIDEN Remotes (CT108 + K8s-Gitea SSH 10.0.30.200). Ahead37/behind63-Narrativ endgültig begraben (Cache-Phantom). Single-Remote-Ziel erreicht: beide Tipps identisch.
- **Autosync-Renne verarbeitet (4 Minentypen):** authelia Duplikat-Volume (union-merge) entfernt; homepage `authelia-auth`-Middleware restauriert (Jul-Entscheidung ging nie live); paperless plaintext-OIDC-Secret GELÖSCHT statt Wert-Rollback (ESO-Ownership seit 24.08., Quelle 1P); newborn-App ceph-csi-cephfs eingefroren (autosync-Block aus Git entfernt — Helm-Release 3.17.0 wartet auf Adoption; rbd-chart 3.10.1 weiterhin ungoverned, Backlog).
- **3 Sync-Failures via Live-State-Arbitrage geheilt (alle Synced/Healthy @ 8c741c5):** (1) backups: VSC-CRD-Flavor-Falle — RKE2-Addon-CRD ist FLAT (driver/deletionPolicy top-level, `spec:` verboten!), Jul-Files waren für Upstream-Flavor korrekt; beide Files geflattet, cephfs-snapclass erstmals LIVE ERZEUGT. (2) databases: CNPG grow-only — Git auf 100Gi/20Gi hochaligned (Shrink verboten). (3) hindsight: VCT storageClassName immutable (hdd→flash aligniert), Live-Hotfixes gespiegelt (Node-Pin cp-01, ReadinessProbe draußen, secretKeyRef statt Klartext-PW). hindsight-postgres-0 rotierte sauber, PVC 47d Bound, API pollt.
- **Push-Learned:** canonical-Push braucht EXPLIZITEN Key (`GIT_SSH_COMMAND="ssh -i ~/.ssh/id_ed25519_gitea-hermes-push -o IdentitiesOnly=yes"`) — kein ~/.ssh/config vorhanden, Default-Key lehnt ab.
- **Chronisch (prä-merge, offen):** kube-prometheus-stack Synced/FAILED (CRD-Annotation >256kB); residual OutOfSync gitea-config (Deployment/gitea-runner) + immich-config (SA/CM-Reste) = Alt-Backlog, keine Regression.
- **Finale Ordnung (nächste Schritte):** ① ArgoCD-Sources auf ssh://git@10.0.30.200:22 flippen (incl. Secret argocd-repo-credentials) ② CT108 final stoppen (explizite Freigabe Dominik) ③ CSI-Adoption ④ kpstack-CRD-Fix.
- Docs: `docs/solutions/workflows/2026-09-17-argocd-autosync-race-four-mine-types.md` + `docs/solutions/bug-fixes/2026-09-17-live-state-arbitration-crd-flavors-hotfix-mirroring.md` · Morgen-Doc (.202-Empfehlung) korrigiert auf .200.
## [2026-09-17] bug-fix | Ghost-Instance-Regression: CT108-Gitea als ArgoCD-Source + gitea-backup RCA-Quality
- gitea-backup-29826930 (03:30Z) failed: BackoffLimitExceeded, 3 Instant-Crashes in 73s. Zwei Auto-RCAs attribuierten auf 1P/ESO-Rate-Limit — STRUKTURELL widerlegt (crashing init-container gitea-files konsumiert keine Secrets; mounted Secrets intakt).
- Echter Fund: ArgoCD-Apps (root/proxy/paperless/schoenkitchen/gitea-config) zogen von http://10.0.30.105:3000 = CT108-Ghost (nach Dekommissionierung 24.07. wieder eingeschaltet, stale Mirror ohne Hardening-Commit 2014673) → "Synced" maskierte fehlende failedJobsHistoryLimit-Felder live.
- Sofortmassnahme: manueller Rerun gitea-backup-manual-161538 SUCCESS (201,7 MB, S3-Upload verifiziert), Failed-Job gelöscht, Alert clear.
- Offen (awaiting owner): ArgoCD-Sources auf ssh://git@10.0.30.202:22 flippen, Repo-Divergenz ahead37/behind63 + tote HTTPS-Auth forensisch, CT108 final stoppen, ESO-Nachtsättigung (00:00Z-Fenster, cf. PAT-003) analysieren.
- Doc: `docs/solutions/bug-fixes/2026-09-17-gitops-ghost-instance-regression-masked-hardening.md` · Wiki: systems/gitea (CT108-Status), concepts/gitops-workflow (Single-Remote-Regel)
- **Abend-Phase:** SSH-Deny-Rootcause = keine Keys/Tokens im K8s-Gitea-DB registriert (Opfer der 01.08.-Migration) → via 1P-Token (hermes-gitops) re-registriert: User-Key + ArgoCD-Deploy-Key (beide end-to-end verifiziert, push dry-run OK). Echter Branch-Split am 24.07. entdeckt: K8s-Gitea-main=29.07.-Stand (b9d7441, 70 Commits incl. mariadb:11.4-Fix für gitea-backup), CT108/local=38 Commits ab 04.09. Hardening TTL=86400 + Exit-42-Guard via ArgoCD gelanded (a0c9b6b). Heutiger DB-Dump validiert (116 Tables, kompletter Trailer). Nächste Schritte: Historien-Konvergenz → Source-Flip → CT108-Stop (Freigabe).
## [2026-08-30] retro | Compound Learning Retrospective (Last 30 Days)
- Reviewed sessions from Jul 31 – Aug 30, 2026
- **4 new solution docs** written by subagents:
- `bug-fixes/2026-08-05-ceph-squid-ec-pool-mark-complete-bug.md` — EC pool mark-complete fails in Squid
- `bug-fixes/2026-08-06-traefik-cross-namespace-routing-pitfall.md` — Two Traefik 404 incidents
- `architecture/2026-08-01-k8s-full-cluster-rebuild-procedure.md` — Full RKE2 rebuild sequence
- `architecture/2026-08-01-immich-data-loss-no-backup.md` — Ceph redundancy ≠ backup
- **2 new patterns** extracted:
- PAT-008: Ansible default_ipv6 fact missing on fresh VMs
- PAT-009: Stale NBD devices after RBD volume swap
- **4 Hindsight entries** indexed with solution summaries
- Key themes: K8s cluster disaster recovery, Ceph EC pool unrecoverability, Traefik routing complexity, backup gap identification
## [2026-08-30] feat | WikiSkill Patterns Directory + Skill-Impact Tracker
- Analysed arXiv:2608.27454 (WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution)
- **New `patterns/` directory** in LLM Wiki — structured failure-mode patterns inspired by WikiSkill's Wiki Layer
- Created 7 initial pattern pages from existing MEMORY.md entries + solution docs:
- PAT-001: Galera DDL TOI Deadlock
- PAT-002: Traefik reload unreliable
- PAT-003: 1Password account-level rate-limit
- PAT-004: VFIO GPU passthrough race condition
- PAT-005: Finanzblick sync WAF-blocked
- PAT-006: LinkedIn React nativeInputSetter
- PAT-007: Ceph EC pool + SSD wear-level
- **Skill-Impact Tracker** (`patterns/skill-impact.md`) — audit trail for skill modifications (accept/reject history)
- **Template** (`patterns/_template.md`) for future pattern creation
- **compound-learning skill patched** — added Phase 4.8 (Pattern Extraction) + Phase 4.9 (Skill-Impact Tracking)
- Updated index.md with Patterns section
- Hindsight: indexed with tags architecture, wikiskill, patterns, skill-evolution
## [2026-08-13] feat | Memory Architecture Enhancement (arxiv 2602.06052v4)
- Implemented 3 new automated memory capabilities from survey paper analysis:
- **Decay & Importance Scoring** (`hindsight_decay_scoring.py`): Monthly cron (1st 03:30). Score = log(1+access)/log(51) × 0.5^(age/90d). Archives score<0.15 with 0 accesses. First run: 200 evaluated, 50 archived.
- **User Drift Detection** (`user_drift_detection.py`): Weekly cron (Mon 04:00). Compares 14d vs 90d baseline for message length, frustration, corrections, topic shifts, style changes. First run: detected 3 drifts (41% longer msgs, new focus gitops, declining ai_ml).
- **Skill Health Check** (`skill_health_check.py`): Weekly cron (Mon 04:30). Scans 187 active SKILL.md files for stubs, broken refs, missing pitfalls, staleness, duplicates. First run: 70 healthy, 117 warnings, 0 critical.
- Cron jobs: 112c106c0954, 280458a2b035, 761b6694d99b (all no_agent, deliver=origin)
- Wiki updated: systems/hindsight.md (meta-memory automation table expanded)
- Hindsight: indexed with tags architecture, memory-system, cron, arxiv-2602.06052
## [2026-08-13] fix | Seafile Recovery + Paperless Upgrade + Authelia Ingress
- **Gitea SSH access**: No SSH key on Hermes VM was authorized for Gitea. HTTP port 3000 has no external IngressRoute. Generated dedicated ED25519 key (`id_ed25519_gitea-hermes-push`), stored in 1Password (ID: lqaebdtewj5xatvpzfrkag73v4), registered in Gitea as key ID 4 for user dominik. SSH via `10.0.30.202:22` (LoadBalancer) is the reliable push path.
- **Paperless v3 IngressRoute**: Host was `dokumente.familie-schoen.com` instead of `dokumente-neu.familie-schoen.com`. Two-track fix: kubectl patch (immediate) + Git commit `fa4184b` (permanent via ArgoCD self-heal). Backtick escaping in Traefik Host() match requires `--patch-file` not inline `--patch`.
- Wiki updated: systems/gitea.md (SSH push key section), reference/ssh-keys.md (new key entry)
- Solution docs: workflows/2026-07-26-gitea-ssh-access-for-hermes.md, bug-fixes/2026-07-26-paperless-ingressroute-hostname-fix.md
- Hindsight: both solutions indexed with tags
## [2026-07-25] fix | Loki 500 Error + CNPG Leader Election + Stale Pods Cleanup
- **Loki HTTP 500**: `replication_factor: 3` in hash ring with only 1 SingleBinary instance → "too many unhealthy instances in the ring". Chart v6.42.0 ignores `loki.common.replication_factor` — correct path is `loki.commonConfig.replication_factor`. Fix: commit `cb70d36`
- **Memcached caches disabled**: `chunksCache.enabled: false`, `resultsCache.enabled: false` (SingleBinary doesn't need them). Commit `f8852e8`
- **Fluent Bit**: cascading failure from Loki 500s — fixed automatically once Loki accepted pushes
- **CNPG 28+129 restarts**: Leader election lease renewal failed during API server latency spikes (Ceph recovery I/O). Default 15s/10s too short. Fix: `--leader-lease-duration=60 --leader-renew-deadline=40` via `additionalArgs`. Commit `6ed4a53`
- **17 stale node-debugger pods** deleted from default namespace
- **OSDs 0+2 destroyed+purged** (proxmox2, SSDs with 92% wear + slow ops). CRUSH host proxmox2 removed. 13 OSDs remaining on 8 hosts.
- Wiki updated: ceph-cluster.md, loki-fluentbit.md
## [2026-07-25] fix | Memory Sync Broken — HTTP→HTTPS + Ceph Duplicate
- Git push failed: remote URL used `http://` but Gitea redirects to `https://` — git doesn't follow auth redirects
- Token itself was valid (same as ArgoCD `argocd-repo-credentials`), just wrong protocol
- Fixed: `git remote set-url` to `https://`
- Eliminated duplicate: `systems/ceph.md` (82 lines, no frontmatter) merged into `systems/ceph-cluster.md` (canonical, referenced in index.md)
- Fixed sync script: removed `--allow-empty` flag, improved change detection
## [2026-07-24] fix | Velero CSI Volume Snapshots Three-Layer Fix
- Velero backups PartiallyFailed for 121 days — all PVCs skipped
- Fix 1: Created VolumeSnapshotClass `ceph-rbd-snapclass` (rbd.csi.ceph.com)
- Fix 2: Migrated ArgoCD repoURLs from dead Gitea (10.0.30.105) to new (ssh://git@10.0.30.202:22)
- Gitea SSH user is `git` not `gitea`; git.schoen.codes → Traefik (10.0.30.208), SSH on 10.0.30.202
- New SSH deploy key generated, added to Gitea repo
- Fix 3: `features: EnableCSI` must be under `configuration:` in Velero Helm values (not top-level)
- Must be string, not array — array form breaks Helm template
- Verified: test backup created VolumeSnapshot + VolumeSnapshotContent, uploaded to S3 ✅
- Solution doc: docs/solutions/bug-fixes/2026-07-24-velero-csi-snapshots-three-layer-fix.md
- Commits: 5ebfa30, 19f35bc, cd18c58, baf85f5, f652bfb
## [2026-07-24] fix | VFIO Module Loading Race Condition RCA + Fix
- RCA: GPU passthrough failed on ms-a2-1 due to missing modprobe config (NOT kernel version)
- Root cause: Missing `softdep amdgpu pre: vfio-pci` + incomplete DRM blacklist → race condition (302s vs 2.4s bind time)
- Fix applied to ms-a2-1: Added `blacklist drm`, `blacklist drm_kms_helper`, `softdep amdgpu pre: vfio-pci`, rebuilt initramfs
- Verification pending (reboot required, 4 VMs + OSD 13 on ms-a2-1)
- Solution doc: docs/solutions/bug-fixes/2026-07-24-vfio-pci-module-loading-race-condition.md
- Also documented: K8s persistent storage architecture (all PVCs on Ceph RBD, 3x replication)
- Solution doc: docs/solutions/architecture/2026-07-24-k8s-persistent-storage-ceph-rbd.md
## [2026-07-24] fix | worker-05 GPU Passthrough — Move to ms-a2-2
- Problem: worker-05 (VM 102) on ms-a2-1 had kernel BUG with 1002:13c0 GPU — /dev/dri/renderD128 missing
- ms-a2-1 and ms-a2-2 have identical hardware (AMD Ryzen 9 9955HX, GPU 1002:13c0)
- Solution: Moved VM 102 from ms-a2-1 to ms-a2-2 (offline migration, Ceph RBD shared storage)
- Result: renderD128 ✅ visible in guest, amdgpu driver loaded, all pods running
- IaC: main.tf cleaned up — single unified rke2_worker block, worker-05 node=ms-a2-2, worker-06 removed
- PCI mapping: amd-gpu now includes n5pro + ms-a2-1 + ms-a2-2
- Commit: 21bd335
- Also: ms-a2-2 upgraded PVE 9.1.1→9.2.5, kernel 7.0.14-6-pve, VFIO persistent after reboot, HA integrated
## [2026-07-24] init | ms-a2-2 Node Initialization
- New Proxmox node ms-a2-2 (10.0.20.93, nodeid 1) joined cluster
- Hardware: AMD Ryzen 9 9955HX (32 threads), 91GB RAM, 1.8TB NVMe
- GPU: AMD Radeon 1002:13c0 (identical to ms-a2-1) — VFIO passthrough configured
- PCI mapping: amd-gpu-ms-a2-2 created, added to generic amd-gpu mapping
- Ceph OSD 14: created on nvme0n1 (1.8TB SSD), class ssd, weight 1.82
- Fluent-bit: installed + configured (hostname adapted to ms-a2-2)
- APT sources: fixed from enterprise to no-subscription (ceph + pve)
- LLM Wiki: ip-map.md, ceph.md updated; AGENTS.md updated (9 nodes)
- Note: PVE 9.1.1 on ms-a2-2 (rest of cluster is 9.2.3) — upgrade pending
## [2026-07-24] cleanup | proxmox1 Permanent Removal + Ceph PG Fixes
- Removed proxmox1 from Ceph CRUSH (empty host bucket, weight 0)
- Also removed stale empty buckets: proxmox, px-tmp20
- Updated corosync.conf: removed proxmox1 node entry, config_version 14→15, expected_votes 9→8
- Removed stale /etc/pve/nodes/{proxmox1,proxmox,pve,px-tmp20}
- Updated IaC: proxmox_api_url 10.0.20.10→10.0.20.91, template_node proxmox1→n5pro, target_nodes list updated
- Updated AGENTS.md topology (9→8 nodes), LLM Wiki ip-map.md + ceph.md
- Commit: 469d29d
- Additional: Manual pg-upmap for stuck PGs 5.13→[1,6,7], 6.6c→[11,8,7] — all clean+remapped eliminated
## [2026-07-24] fix | Ceph Remapped PGs — Weight Imbalance + pg_num Mismatch
- RCA: 11 active+clean+remapped PGs caused by CRUSH placement failure
- Root cause 1: Pool 6 (hdd_disk) pg_num=120 ≠ pgp_num=112 (autoscaler mid-merge)
- Root cause 2: Extreme HDD host weight imbalance (n5pro=10TB vs proxmox6=0.3TB)
- Fix: Equalized pg_num→112, set nopgchange=true, reweighted osd.7 0.80→1.0
- Result: 9/11 PGs recovering (backfilling at 36MiB/s), 2 cosmetic clean+remapped remain
- Created systems/ceph.md with full OSD/pool/CRUSH documentation
## [2026-07-24] update | Gitea Actions K8s Runner Bug + DinD Pitfalls
- Documented Gitea 1.27.0 bug: actions_ready_job queue doesn't assign tasks to K8s-based runners
- Updated systems/gitea.md with known issue, DinD pitfalls, runner management via DB
- Solution doc: docs/solutions/bug-fixes/2026-07-24-gitea-actions-k8s-runner-task-assignment-bug.md
- IaC committed: runner-deployment.yaml + runner-config.yaml (DinD TCP sidecar pattern)
## [2026-04-28] init | Memory System Created
- Created memory directory structure at ~/.hermes/memory
- Installed qmd 2.1.0 (Query Markup Documents)
- Created initial index.md with entities, concepts, and sources
- Created Entity pages: Infrastructure, Email-System, Health-Fitness, Team-Structure, Memory-System
- Created Concept pages: Network-Architecture, Email-Organization, Monitoring-System, Proxmox-Cluster, qmd-Knowledge-Base
- Initialized git repo, pushed to origin/main
## [2026-04-29] fix | Memory Process Established
- Corrected stale memory-entry-001.md
- Established protocol: memory update + git sync on every session
## [2026-04-30] ingest | cloud.familie-schoen.com
- Full port scan, DNS, SSL, HTTP headers, API analysis
- Identified: Seafile 13.0.19 on nginx, Let's Encrypt SSL
## [2026-07-24] restructure | Full Wiki Restructuring
- Deleted 5 root-duplicate files (dominik-schoen, email-organization, team-structure, llm-wiki-pattern, embedding-model)
- Deleted obsolete Entities/Memory-System.md, Concepts/qmd-Knowledge-Base.md
- Deleted raw/articles/ and memory-entry-001/002.md (stale)
- Renamed Entities/ → entities/, Concepts/ → concepts/ (lowercase)
- Created new directories: systems/, reference/
- Moved Proxmox-Cluster, Monitoring-System, seafile to systems/
- Updated all existing pages with current data (April → July 2026):
- Infrastructure: 2 Nodes → 9 Nodes, added Ceph/RKE2/Galera/Loki
- Proxmox-Cluster: PVE 9.2.3, 9 Nodes, Fluent Bit details, PVE Tasks, Ceph Audit
- Network-Architecture: cleaned up, links to reference pages
- Monitoring: Prometheus/Grafana/Loki/HolmesGPT (was just "weekly network scans")
- Created 6 new system pages: ceph-cluster, rke2-kubernetes, galera-maxscale, loki-fluentbit, gitea, hindsight
- Created 3 reference pages: ip-map, ssh-keys, ports
- Created 2 concept pages: gitops-workflow, credential-policy
- Rewrote index.md with new structure + Memory Layer Architecture table
- Total: 18 pages (was 21 with dupes, now 18 clean unique pages)
## [2026-09-29] update | VM302 Zombie-Recovery nach Migration
- VM302 (Galera db3) reagierte nach Migration auf ms-a2-2 nicht: QEMU "running", aber SSH/MariaDB/QGA tot
- Root Cause: Post-Migration-Zombie; -incoming/-S in QEMU-Cmdline ist Artefakt, kein Beweis für Pause
- Fix: qm stop/start trotz HA-Guard, SST-Rejoin ~2-3min
- Endstand: 3/3 Synced, Primary, MaxScale alle Server Running
- Solution Doc: docs/solutions/bug-fixes/2026-09-29-vm302-zombie-postmigration-galera-rejoin.md
- Zusätzlich: Home Assistant Core Restart via REST API erfolgreich (Version 2026.9.3, RUNNING)
## 2026-10-08 — NUT-Shutdown-Kaskade clusterweit (epic-9 Finale)
- GO: NAS-Treiber entfernen + nut-monitor-Kaskade aktivieren
- Master ms-a2-1: upsmon MASTER + upssched (ONBATT>60s → FSD, LOWBATT → Sofort-FSD, HOSTSYNC 45)
- 7 PVE-Slaves + NAS als NETCLIENTS, NAS-Altlast gepurged (Crashloop tot)
- FSWEEP: 9/9 nut-monitor active+enabled; HA-NUT-Entry bleibt loaded
- Commits adb4dcc/beee480; Neue Wikiseite systems/nut.md + Index
- Nebenfunde: n5pro pgdg-Repo kaputt (braucht GO); Hermes-Root-FS war 100% voll (Backups → NAS)
- Quality-Gate-Lektion: Syntax-Check fing 2 korrupte Writes (Volldisk-Ursache)
## 2026-10-08 — Disk-Audit hermes-agent-01 + Offsite-Backup-RCA
- Root-FS 96%→72% (21,7G frei): 10x hermes-backup Tarballs (18,9G) nach SHA256-Verifikation (10/10 identisch) auf NAS geloescht
- RCA Offsite-Backup: hermes-backup.service Exit 127 'aws: command not found' (systemd-PATH ohne ~/.local/bin) seit >=27.09. TA EGLICH; Offsite war TOOT
- Fix: Skript (AWS_BIN absolut + trap EXIT-Cleanup), systemd-Drop-in (PATH + TimeoutStartSec 3600); Prod-Lauf SUCCESS, S3-Objekt 2.284.533.218B verifiziert
- n5pro pgdg.list disabled (syntaktisch defekt, nie genutzt) -> apt update gruenn
- NUT-Watchdog-Cron 4cbf03ccb4de (1/min, rohes NUT-Proto, quiet-if-ok, 10min-Throttle) 7/7 Tests
- Naechste Aufraeum-Hebel (warten auf GO): podman prune 23,4G, uv-Cache 9,4G, playwright-alt ~1G, journal 0,8G, apt-cache 0,6G
- Welle 2: journald SystemMaxUse=200M (drop-in, permanent; 1017M->199M), uv cache clean (-70,8G logisch, FS 72->59%), Guard-Cron 624a2f98fb10 (So 09:00, 3GiB threshold)
- Closing-the-Loop: PAT-015 systemd-path-blindness (patterns/), Wiki systems/hermes-agent-01 neu, INDEX.md normalisiert (28 docs, beide Okt-Docs eingepflegt), skill-impact-Ledger 2 Rows gebucht, Hindsight Wave-2
+3
View File
@@ -14,6 +14,7 @@ The entry records WHAT changed, WHY, and WHETHER it helped (if validated).
``` ```
| Date | Skill | Change Type | Description | Validation | Outcome | Ref | | Date | Skill | Change Type | Description | Validation | Outcome | Ref |
``` ```
| 2026-10-08 | smart-home-automation | adopted | EMHASS-References erweitert: Status-Block in defect-analysis (Pointer-Datei) + neue Sektion "CSV Laws & Shadow Testing" in api-and-config | tested | accepted | docs/solutions/workflows/2026-10-08-emhass-phase2-shadow-methodology.md |
- **Change Type**: `created` | `patched` | `deleted` | `adopted` - **Change Type**: `created` | `patched` | `deleted` | `adopted`
- **Validation**: How was success measured? (`manual` | `tested` | `benchmark` | `n/a`) - **Validation**: How was success measured? (`manual` | `tested` | `benchmark` | `n/a`)
@@ -38,3 +39,5 @@ The entry records WHAT changed, WHY, and WHETHER it helped (if validated).
`validation: manual` — they may enable future improvements. `validation: manual` — they may enable future improvements.
- **Rollbacks** are explicitly tracked — if a skill patch caused regressions and was - **Rollbacks** are explicitly tracked — if a skill patch caused regressions and was
reverted, the entry stays with `outcome: rolled-back`. reverted, the entry stays with `outcome: rolled-back`.
| 2026-10-08 | smart-home | patched | NUT-Absatz ergänzt: Kaskade, Master/slaves, roh-Proto-Details, Watchdog-Cron 4cbf03ccb4de | tested | accepted | docs/solutions/bug-fixes/2026-10-07-nut-ups-ms-a2-1-deployment.md |
| 2026-10-08 | (none) | adopted | PAT-015 systemd-path-blindness erstellt (Exit-127-Klasse: ~/.local/bin unsichtbar im Service-PATH; trap-EXIT-Cleanup-Pflicht) | tested | accepted | docs/solutions/bug-fixes/2026-10-08-hermes-offsite-backup-aws-path-rca.md |
+42
View File
@@ -0,0 +1,42 @@
# Skill Impact Tracker
> Inspired by WikiSkill (arXiv:2608.27454) — tracks skill modifications, their validation
> outcomes, and acceptance decisions. Prevents repeating failed modifications and provides
> an audit trail of skill evolution.
## How to Use
When a skill is patched, created, or deleted, append an entry to the table below.
The entry records WHAT changed, WHY, and WHETHER it helped (if validated).
### Entry Format
```
| Date | Skill | Change Type | Description | Validation | Outcome | Ref |
```
- **Change Type**: `created` | `patched` | `deleted` | `adopted`
- **Validation**: How was success measured? (`manual` | `tested` | `benchmark` | `n/a`)
- **Outcome**: `accepted` | `rejected` | `rolled-back` | `pending`
- **Ref**: Session ID or solution doc path
## Audit Trail
| Date | Skill | Change Type | Description | Validation | Outcome | Ref |
|------|-------|-------------|-------------|------------|---------|-----|
| 2026-08-30 | compound-learning | patched | Added WikiMaintainer phase + patterns/ guidance from WikiSkill paper analysis | tested | accepted | this session |
| 2026-08-30 | (retrospective) | created | 4 solution docs + 2 new patterns (PAT-008, PAT-009) from 30-day retrospective | tested | accepted | this session |
| 2026-08-13 | hindsight | patched | Expanded meta-memory automation table (decay scoring, drift detection, skill health) | tested | accepted | session 2026-08-13 |
| 2026-08-13 | (3 cron jobs) | created | Decay scoring, drift detection, skill health check — no_agent cron jobs | tested | accepted | session 2026-08-13 |
## Notes
- **Rejected proposals persist here** — they are NOT deleted. Future skill updates can
consult this log to avoid repeating failed approaches (the WikiSkill paper showed this
is critical for effective evolution).
- **Neutral changes** (no improvement, no regression) are recorded as `accepted` with
`validation: manual` — they may enable future improvements.
- **Rollbacks** are explicitly tracked — if a skill patch caused regressions and was
reverted, the entry stays with `outcome: rolled-back`.
| 2026-10-08 | smart-home | patched | NUT-Absatz ergänzt: Kaskade, Master/slaves, roh-Proto-Details, Watchdog-Cron 4cbf03ccb4de | tested | accepted | docs/solutions/bug-fixes/2026-10-07-nut-ups-ms-a2-1-deployment.md |
| 2026-10-08 | (none) | adopted | PAT-015 systemd-path-blindness erstellt (Exit-127-Klasse: ~/.local/bin unsichtbar im Service-PATH; trap-EXIT-Cleanup-Pflicht) | tested | accepted | docs/solutions/bug-fixes/2026-10-08-hermes-offsite-backup-aws-path-rca.md |
+49
View File
@@ -0,0 +1,49 @@
---
pattern_id: PAT-015
title: "systemd-PATH-Blindness: binaries aus ~/.local/bin verschwinden im Service-Kontext"
category: infrastructure
severity: high
status: active
first_observed: 2026-10
last_updated: 2026-10-08
related_systems: [hermes-agent-01, systemd]
related_solution_docs: [docs/solutions/bug-fixes/2026-10-08-hermes-offsite-backup-aws-path-rca.md]
related_skills: []
---
# PAT-015 — systemd-PATH-Blindness
**Trigger**: Ein Skript, das interaktiv/per SSH fehlerfrei läuft, stirbt als systemd-Service mit
`command not found` (Exit 127) — typischerweise bei pip-user-installierten CLIs (`aws`, `uv`),
Node/npm-Tools oder allem unter `$HOME/.local/bin`, `$HOME/.cargo/bin`, `/opt/*`.
## Symptom
- Journal: `<script>: line NN: aws: command not found` → `status=127`
- Dieselbe Binary funktioniert problemlos in der interaktiven Shell desselben Users.
- Bei `set -e`-Skripten mit Cleanup AM ENDE: sekundär Müll-Akkumulation, weil der Exit
vor dem Aufräumschritt liegt.
## Root Cause
systemd startet Units mit definiertem Minimal-PATH (`/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`),
OHNE die User-Shell-Profile zu sourcen. Pip `--user`-Installationen landen in `~/.local/bin`,
das nur via `~/.profile`/shell-Init in PATH kommt. Interaktiv≠Serviceland.
## Mitigation
1. **Robuste Variante (im Skript):** Absolutpfad mit Fallback-Kette:
`BIN="$(command -v foo || true)"; [ -z "$BIN" ] && BIN="/home/user/.local/bin/foo"; [ -x "$BIN" ] || exit 1`
2. **Unit-seitig:** Drop-in `Environment="PATH=/home/user/.local/bin:$PFAD"` — macht ALLE
Scripts der Unit robust, auch solche Dritter.
3. **Cleanup-Traps:** `trap 'rm -f "$TMPFILE"' EXIT` — garantiert Aufräumen bei JEDEM Exit,
egal wo `set -e` zuschlägt.
## Prevention
- Beim Schreiben neuer systemd-Units mit User-Skripten: PATH-Annahme niemals erben lassen.
- Watchdog-Muster: Backup-Jobs brauchen Alter-des-letzten-Erfolgs-Monitoring (still sterbende
Jobs bleiben sonst Wochen unsichtbar — hier 10+ Tage TOOT).
- Bei Exit-127 im Journal zuerst PATH-Differenz interaktiv vs. Service prüfen (`systemctl show -p Environment`).
## Evidence
- 2026-10-08: hermes-backup.service (backup-to-s3.sh, täglich 04:00 UTC) Exit 127 seit ≥27.09.,
10× ~2G-Tarballs in /tmp bis Root-FS 100 %. Fix: Skript-AWS_BIN + Drop-in-PATH + trap.
Verifikation: Produktionslauf SUCCESS + S3-Readback 2.284.533.218 B.
→ docs/solutions/bug-fixes/2026-10-08-hermes-offsite-backup-aws-path-rca.md
@@ -3,7 +3,7 @@ title: "EMHASS Kaskaden-Automations-Architektur"
category: reference category: reference
tags: [emhass, battery, cascade, homeassistant, byd, marstek] tags: [emhass, battery, cascade, homeassistant, byd, marstek]
created: "2026-10-02" created: "2026-10-02"
modified: "2026-10-02" modified: "2026-10-08"
--- ---
# EMHASS Kaskaden-Automations-Architektur # EMHASS Kaskaden-Automations-Architektur
@@ -104,9 +104,23 @@ Siehe Solution Doc: `docs/solutions/integrations/2026-10-02-emhass-set-config-ov
| Job ID | Schedule | Typ | Funktion | | Job ID | Schedule | Typ | Funktion |
|--------|----------|-----|----------| |--------|----------|-----|----------|
| `a32005d62226` | every 15m | no_agent script | Ruft EMHASS Optimization auf, published Plan zu HA | | `a32005d62226` | every 15m | no_agent script | Ruft EMHASS Optimization auf, published Plan zu HA |
| `d046e808f543` | */15m | LLM-Agent (silent-on-healthy) | Shadow-Dual-Solve A/B + Ledger |
| `19aef2351273` | hourly :35 | no_agent script | Shadow-Staleness-Watchdog (>45 min -> Alert) |
Runner: `/home/debian/.hermes/scripts/emhass-optimizer-runner.sh` Runner: `/home/debian/.hermes/scripts/emhass-optimizer-runner.sh`
Optimizer: `/home/debian/scripts/emhass_optimizer.py` Optimizer: `/home/debian/scripts/emhass_optimizer.py`
Shadow: `/home/debian/scripts/emhass_optimizer_v2.py --shadow` - Ledger `~/shadow_phase2/ledger.jsonl` - Report `~/scripts/report_phase2.py`
## Shadow Phase-2 (live seit 2026-10-08)
Dual-Solve-Validierung (A=Standard, B=grid-charge erlaubt) auf IDENTISCHEN
Inputs; Incumbent unangetastet. Bewiesene CSV-Gesetze: `+P`=Entladung,
`P_grid` signed net import (**Export NEGATIV**), Deferrable-Spalten =
Phantom-Last (+12 kWh Summeninvarianz, KEINE Consumer) ->
**produktiv entfernt 2026-10-08** (`number_of_deferrable_loads: 2->0`,
Direct-File-Edit + Addon-Restart im :50-Fenster, Backup
`config.json.hermesbak-20261008`; Beweis: Incumbent-CSV faellt auf 17
Spalten). Auswertung: `python3 ~/scripts/report_phase2.py`.
## Deaktivierte Komponenten ## Deaktivierte Komponenten
@@ -0,0 +1,141 @@
---
title: "EMHASS Kaskaden-Automations-Architektur"
category: reference
tags: [emhass, battery, cascade, homeassistant, byd, marstek]
created: "2026-10-02"
modified: "2026-10-02"
---
# EMHASS Kaskaden-Automations-Architektur
> Steuerung der dualen Batterie-Systems (BYD 11kWh + Marstek 5kWh) via EMHASS Optimizer + HA Kaskaden-Automation.
## Architektur-Übersicht
```
EMHASS (LP Solver) → sensor.smart_battery_target
↓
┌───────────────────┴──────────────────┐
│ Kaskaden-Automation (jede 2 Min) │
├───────────────────────────────────────┤
│ BYD/SMA: 803 Auto (default) │
│ 802 Manual (bei target≤0) │
│ Marstek: force_mode via │
│ sensor.marstek_target_power │
│ (= smart_target − BYD_actual)│
└───────────────────────────────────────┘
```
## EMHASS Configuration (16 kWh Virtual Battery)
| Parameter | Value | Hinweis |
|-----------|-------|---------|
| `battery_nominal_energy_capacity` | 16000 | BYD 11k + Marstek 5k kombiniert |
| `battery_charge_power_max` | 6700 | 4200 (BYD) + 2500 (Marstek) |
| `battery_discharge_power_max` | 6700 | dto. |
| `battery_charge_efficiency` | 0.9 | Konservativ, gemischt |
| `battery_discharge_efficiency` | 0.9 | dto. |
| `battery_target_state_of_charge` | 0.15 | 15% Reserve (geändert von 0.30 am 2026-10-02) |
| `battery_soc_deficit_cost` | 0.5 | Strafe für Unterschreitung |
| `battery_minimum_state_of_charge` | 0.1 | 10% Hard Floor |
| `battery_maximum_state_of_charge` | 1.0 | 100% |
| `set_battery_dynamic` | True | Dynamische Peeling erlaubt |
| `set_use_battery` | True | Batterie aktiv |
| `sensor_battery_state_of_charge` | `sensor.total_soc` | Gewichteter Gesamt-SoC |
| `sensor_power_battery` | `sensor.batteries_power_out_total` | Entladeleistung |
| `use_websocket` | True | |
## ⚠️ Known Pitfall: set-config Full Overwrite
`POST /set-config` ersetzt die **gesamte** Konfiguration, nicht nur gesendete Felder.
Nicht angegebene Parameter fallen auf Defaults zurück (capacity→5000, power→1000, etc.).
**Regel:** Immer ALLE Parameter zusammen senden. Nach jedem Write mit `GET /get-config` verifizieren.
Siehe Solution Doc: `docs/solutions/integrations/2026-10-02-emhass-set-config-overwrite-pitfall.md`
## Kaskaden-Automation
**Automation ID:** `1774430238377` (ON)
**Schedule:** Alle 2 Minuten
**File:** `/mnt/data/supervisor/homeassistant/automations.yaml`
### BYD/SMA Steuerung
| EMHASS target | PV | BYD Modus | Bedeutung |
|--------------|-----|-----------|-----------|
| `> 0` (Entladung) | egal | **803 Auto** | BYD self-managed, Marstek kriegt Rest |
| `≤ 0` (Hold/Ladung) | `< 100W` | **802 Manual** | BYD hält/zwingt Ladung, EVCC hold |
| `< 0` (Ladung) | `≥ 100W` | **803 Auto** | PV reicht, BYD lädt selbst |
- BYD läuft permanent in Auto/Normal — nur bei EMHASS-Hold/Zwangsladung auf 802
- Modbus Register: 44425 (Mode), 44427 (Power)
- BYD IP: `10.0.50.202:502`
### Marstek Steuerung
- `sensor.marstek_target_power` = `smart_battery_target − actual_BYD_discharge + actual_BYD_charge`
- Cap: ±2500W
- Force modes: `charge`, `discharge`, `standby`
- Writable entities: `select.marstek_venus_modbus_force_mode`,
`number.marstek_venus_modbus_ladeleistung_einstellen`,
`number.marstek_venus_modbus_entladeleistung_einstellen`
- Marstek IP: `10.0.50.113`
### Gelöste Bugs
1. **Double-Counting (behoben 2026-10-01):** `sensor.sma_target_power` nahm volle EMHASS-Target,
ließ Marstek 0W. Fix: `sma_target_power` gelöscht, `marstek_target_power` nutzt jetzt
echte BYD-Werte (discharge - charge) statt Forecast-Split.
2. **Overnight Hold Too High (behoben 2026-10-02):** EMHASS hielt bei 65% weil
`target_state_of_charge=0.30`. Geändert auf 0.15.
3. **BYD ignoriert Hold (behoben 2026-10-02):** 802-Bedingung war `target < 0`,
jetzt `target <= 0` (Hold bei target=0 eingeschlossen).
## Template Sensoren (configuration.yaml)
- `sensor.smart_battery_target` — EMHASS p_batt_forecast, gerundet
- `sensor.marstek_target_power` — Plan A: smart_target − actual_BYD_net_discharge
- Guards: SoC-Checks, ±2500W cap
- `sensor.sma_target_power` — **ENTFERNT** (war Double-Counting Source)
## Cron Jobs
| Job ID | Schedule | Typ | Funktion |
|--------|----------|-----|----------|
| `a32005d62226` | every 15m | no_agent script | Ruft EMHASS Optimization auf, published Plan zu HA |
Runner: `/home/debian/.hermes/scripts/emhass-optimizer-runner.sh`
Optimizer: `/home/debian/scripts/emhass_optimizer.py`
## Deaktivierte Komponenten
| Component | Status | Grund |
|-----------|--------|-------|
| Tibber Smart Charging Bridge | OFF | EMHASS ist sole optimizer |
| Marstek Kaskaden-Steuerung (Modbus) | OFF | Alte PV-only Kaskade, ersetzt durch EMHASS-Kaskade |
## Hardware Referenz
| Device | IP | Capacity | Max Power | Interface |
|--------|----|----------|-----------|-----------|
| BYD/SMA SBS | 10.0.50.202:502 | 11 kWh | 4.2 kW | Modbus 44425/44427 |
| Marstek Venus E | 10.0.50.113 | 5 kWh | 2.5 kW | HA Entities (Modbus) |
| SMA SB #1 | 10.0.50.201:502 | — | — | PV Wechselrichter |
| SMA SB #2 | 10.0.50.203:502 | — | — | PV Wechselrichter |
| Hoymiles | 10.0.40.55 | — | — | OpenDTU |
## Zugriff
- EMHASS API: `http://10.0.30.10:5000`
- HA intern: `http://10.0.30.10:8123`
- HA extern: `https://homeassistant.familie-schoen.com`
- SSH: `ssh -i ~/.ssh/id_ed25519_proxmox -p 22222 root@10.0.30.10`
- HA Token: `/tmp/ha_token.txt`
- HA Config: `/mnt/data/supervisor/homeassistant/`
- Backups: `.bak.20261001_175804` (configuration.yaml + automations.yaml)
## Related
- [[systems/homeassistant]] — HA System Seite
- Solution Doc: `docs/solutions/integrations/2026-10-02-emhass-set-config-overwrite-pitfall.md`
+2 -2
View File
@@ -17,8 +17,8 @@ modified: "2026-09-30"
| 10.0.20.60 | proxmox6 | 6 | | | 10.0.20.60 | proxmox6 | 6 | |
| 10.0.20.70 | proxmox7 | 7 | MON, Traefik CT99999 | | 10.0.20.70 | proxmox7 | 7 | MON, Traefik CT99999 |
| 10.0.20.91 | n5pro | 8 | Templates 9000/9001/9002, MON, MGR (active) | | 10.0.20.91 | n5pro | 8 | Templates 9000/9001/9002, MON, MGR (active) |
| 10.0.20.92 | ms-a2-1 | 9 | OSD 13 (SSD 1.8TB), GPU 1002:13c0, MON, **NUT-Server (APC UPS USB, :3493)** | | 10.0.20.92 | ms-a2-1 | 9 | OSD 13 (SSD 1.8TB), GPU 1002:13c0, MON, **NUT-Master (APC UPS USB, upsd :3493, Shutdown-Kaskade)** |
| 10.0.20.93 | ms-a2-2 | 1 | OSD 14 (SSD 1.8TB), GPU 1002:13c0 | | 10.0.20.93 | ms-a2-2 | 1 | OSD 14 (SSD 1.8TB), GPU 1002:13c0, NUT-Slave |
> ⚠️ n5pro (10.0.20.91, nodeid 8) ≠ proxmox7 (10.0.20.70, nodeid 7) — separate Nodes! > ⚠️ n5pro (10.0.20.91, nodeid 8) ≠ proxmox7 (10.0.20.70, nodeid 7) — separate Nodes!
> proxmox1 (10.0.20.10) und proxmox2 (10.0.20.20) wurden dauerhaft entfernt. > proxmox1 (10.0.20.10) und proxmox2 (10.0.20.20) wurden dauerhaft entfernt.
+44
View File
@@ -0,0 +1,44 @@
---
category: systems
tags: [hermes-host, runner, systemd, backup, housekeeping, watchdog]
updated: 2026-10-08
---
# systems/hermes-agent-01 — Lokaler Hermes-Runner
## Identity
- Debian 12, Kernel 6.1.0-53-amd64, Root-FS **79G ext4 (/dev/sda1)** — historisch knappe Resource!
- Rolle: Hermes-Agent-Host (Telegram-Gateway, Crond, Playwright, avahi `hermes-agent-01.local`);
Arbeitstiers laufen aus `~/hermes-agent` Worktree (`venv/bin/python -m hermes-cli main`).
## Storage-Governance (Stand 08.10., nach Krisenwoche)
- **Worst Case erlebt:** 96–100 % volles Root-FS (10./08.10.) → korrupte write_file-Writes (mid-stream truncation); Syntax-Checker fing 2× korrupte Dateien.
- Aktuell: **59 % belegt, 32 G frei** (von 96 % morgens).
- **Permanenter Guards:**
- journald: Drop-in `/etc/systemd/journald.conf.d/sizecap.conf` → `SystemMaxUse=200M` (1017M→199M)
- uv-Cache: Guard-Cron `624a2f98fb10` (So 09:00, no_agent, Skript `clean_uv_cache.sh`, Threshold 3 GiB, leise wenn drunter)
- **Podman bereinigt (08.10.):** 160 → 15 Images, ~23 G freigegeben (Inventory: `~/podman_inventory_2026-10-08.txt`). `schoenkitchen:latest` wurde versehentlich mitgeprünt (Namensregel-Fehler beim Pin-Container) → aus Git-Quelle rebuildet (`8c439711ceef`, 168 MB).
- Producer der alten Krise: siehe Backup unten (tarballs in /tmp ohne Limit).
## Backup-Pipeline (produzierend seit 08.10.)
- systemd `hermes-backup.timer` 04:00 UTC → `hermes-backup.service` (User debian) → `~/.hermes/scripts/backup-to-s3.sh`
- Ziel: Norris S3 `rgw.nbg.nsc.noris.cloud`, Bucket `homelab-hermes-backup`, Retention 14 Tage; Creds via 1P Item `postgres-s3-backup` (Vault Kubernetes ESO).
- **RCA 08.10.:** Exit 127 `aws: command not found` seit ≥27.09. (systemd-PATH ohne `~/.local/bin`) → Offsite TOOT + /tmp-Vollauf. Fix: Skript (absoluter AWS_BIN + `trap ... EXIT`-Cleanup) + Drop-in `/etc/systemd/system/hermes-backup.service.d/override.conf` (PATH + TimeoutStartSec 3600). Produktionslauf verifiziert (SUCCESS, S3-Readback 2.284.533.218 B). → PAT-015.
- Belt-and-braces: manuelle Tarballs liegen zusätzlich auf NAS `10.0.20.100:/pool01_n2_redundant/backups/hermes-host/` (10 Stück, SHA256-verifiziert).
## Watchdogs (lokal registriert)
- **NUT UPS Watchdog** Cron `4cbf03ccb4de` (1/min): rohes NUT-Proto gegen 10.0.20.92:3493, quiet-if-OK → Wiki systems/nut.
- **uv Cache Guard** Cron `624a2f98fb10` (So 09:00).
- HA-Backup-Alter Checker `2006a91fba21` (tägl. 12:00, Schwelle 30 h).
## Gotchas dieses Hosts
- systemd/cron kennen `~/.local/bin` NICHT (aws, uv!) → Absolutpfade oder Drop-ins. PAT-015.
- Root-FS-Volllauf = Korruptionsquelle Nr. 1 (write_file mid-stream truncation) — vor großen Writes Platz checken.
- Terminal-Foreground verbietet `&`-Konstrukte; Async nur via `background=true`.
- PDF-Generation via `/home/debian/.venvs/pdfkit/bin/python3` (PEP-668 sperrt System-pip).
- `~/.hermes/state.db` 1,84 G + Snapshots 507 M — größte Nicht-Container-Posten.
## Related
- [[systems/nut]] — UPS-Watchdog-Detail
- [[reference/emhass-cascade]], [[systems/homeassistant]]
- docs/solutions/bug-fixes/2026-10-08-hermes-offsite-backup-aws-path-rca.md
+1
View File
@@ -77,6 +77,7 @@ modified: "2026-09-27"
- **Kaskaden-Automation** (`1774430238377`, ON): EMHASS target → BYD (803 Auto default, 802 bei `target<=0 && pv<100`) + Marstek (Rest via `sensor.marstek_target_power`) - **Kaskaden-Automation** (`1774430238377`, ON): EMHASS target → BYD (803 Auto default, 802 bei `target<=0 && pv<100`) + Marstek (Rest via `sensor.marstek_target_power`)
- **Template Sensoren:** `sensor.smart_battery_target` (EMHASS), `sensor.marstek_target_power` (Plan A: smart_target − actual_BYD_discharge). `sensor.sma_target_power` ENTFERNT (Double-Counting Bug). - **Template Sensoren:** `sensor.smart_battery_target` (EMHASS), `sensor.marstek_target_power` (Plan A: smart_target − actual_BYD_discharge). `sensor.sma_target_power` ENTFERNT (Double-Counting Bug).
- **Cron:** `a32005d62226` (no_agent, every 15m, runner script) ruft EMHASS Optimization auf. - **Cron:** `a32005d62226` (no_agent, every 15m, runner script) ruft EMHASS Optimization auf.
- **Shadow Phase-2 (2026-10-08):** Dual-Solve-Shadow live (Cron `d046e808f543` + Staleness-Watchdog `19aef2351273`); Deferrables als Phantom-Last erkannt und produktiv entfernt (CSV jetzt 17 Spalten). Details: [[reference/emhass-cascade-automation-architecture]]
- **Backup:** `.bak.20261001_175804` für configuration.yaml + automations.yaml - **Backup:** `.bak.20261001_175804` für configuration.yaml + automations.yaml
- **NFS-Backups (Fix 2026-10-07):** Supervisor-Mounts `Backup_NFS` (usage=backup) + `Medien_NFS` (usage=media) → **10.0.20.100:/pool01_n2_redundant/HomeAssistant** (vorher tote 10.0.30.100, verloren beim NAS-Crash-Reboot 03.10.). Volle automatische Backups ~03:27 UTC nightly. Monitoring: Hermes-Cron `2006a91fba21` (Alert bei Backup-Alter >30h). Lösungsdoku: `docs/solutions/bug-fixes/2026-10-07-ha-audit-nfs-mount-failure-after-nas-crash.md` - **NFS-Backups (Fix 2026-10-07):** Supervisor-Mounts `Backup_NFS` (usage=backup) + `Medien_NFS` (usage=media) → **10.0.20.100:/pool01_n2_redundant/HomeAssistant** (vorher tote 10.0.30.100, verloren beim NAS-Crash-Reboot 03.10.). Volle automatische Backups ~03:27 UTC nightly. Monitoring: Hermes-Cron `2006a91fba21` (Alert bei Backup-Alter >30h). Lösungsdoku: `docs/solutions/bug-fixes/2026-10-07-ha-audit-nfs-mount-failure-after-nas-crash.md`
+87
View File
@@ -0,0 +1,87 @@
---
title: "Home Assistant"
category: systems
tags: [homeassistant, smart-home, mqtt, automation]
created: "2026-09-27"
modified: "2026-09-27"
---
# Home Assistant
> Smart Home Zentrale auf Proxmox. Steuerung, Automatisierung und Benachrichtigungen.
## Infrastruktur
- **Host:** `10.0.30.10` (HAOS VM)
- **SSH:** `hassio@10.0.30.10` (PW: 1P Vault "Hermes")
- **URL:** `https://homeassistant.familie-schoen.com`
- **Container:** `homeassistant` (Docker)
## MQTT
- **Broker:** Mosquitto Add-on v7.1.1 auf localhost:1883
- **HA MQTT User:** ehemals `opendtu` (BROKEN — ACL blockiert), gefixt
- **ACL:** `/etc/mosquitto/acl` definiert `user homeassistant` + `user addons`
- **Auth Plugin:** `go-auth.so` (files,http backends)
## Automations
- **File:** `/config/automations.yaml` (35 Automations)
- **Schreibmethode:** SSH → `docker exec homeassistant chmod 666`, danach restore 644
- **Reload:** REST API mit JWT (HS256, signed from `/config/.storage/auth`)
### Frigate Einfahrt Notification (ID 1714065780832)
- MQTT trigger `frigate/events` → filter `type=='new'` + cameras [Einfahrt,Terrasse] + labels [person,car,...]
- Mode: `single`, cooldown 600s
- Dedup: `input_text.frigate_last_event_id`
- Flow: delay 10s → snapshot → notify iPhone → optional LLM Vision
- Siehe [[systems/frigate]]
## Notify Services
| Service | Status |
|---------|--------|
| `notify.mobile_app_iphone_dominik` | ✅ aktiv |
| `notify.mobile_app_sarahs_iphone_app` | verfügbar |
| `notify.mobile_app_ipad_2` | verfügbar |
| `notify.mobile_app_sm_x205` | verfügbar |
## Entitäten
- Kameras: `camera.einfahrt_2`, `camera.terrasse_2` (Suffix `_2` wegen verwaister Integration)
- Motion: `binary_sensor.einfahrt_motion_2`, `binary_sensor.terrasse_motion_2`
- Input Text: `input_text.frigate_last_event_id` (dedup storage, max 255 chars)
## Snapshots
- Gespeichert: `/config/www/snapshots/{camera}_latest.jpg`
- URL: `/local/snapshots/` (HTTP 200, keine Auth)
## JWT Auth
1. `jwt_key` aus `/config/.storage/auth` lesen
2. Client `Hermes_202606` (token id `1444b2c6757b4d66a6f8f8e5899b4e6a`)
3. HS256 signieren, Bearer Header
4. `?return_response=true` für Service-Call Responses
## Integrations
- Frigate (MQTT)
- LLM Vision (noris AI `gemma-4-31b-it`)
- Tibber (Strom)
- Marstek Speicher (VENUS-E, IP 10.0.50.113)
- NUT (APC Back-UPS ES 700G via ms-a2-1, `10.0.20.92:3493`, User monuser; Sensors `sensor.apc_*`; Entry `5e1d2138235140eb9b476e9d6779c228`; deployed via `iac-homelab/epic-9-ups-nut`)
- Various sensors (Xiaomi BLE, etc.)
## EMHASS Batteriesteuerung
- **Endpoint:** `http://10.0.30.10:5000` (v0.18.4)
- **Architektur:** EMHASS = Sole Optimizer. Virtuelle 16 kWh Batterie (BYD 11k + Marstek 5k).
- **Config API:** `POST /set-config` überschreibt GESAMTE Konfig (kein Partial-Update!). Immer alle Felder senden. Siehe Solution Doc `integrations/2026-10-02-emhass-set-config-overwrite-pitfall.md`.
- **Akuelle Config (2026-10-02):**
- Capacity 16000Wh, Charge/Discharge 6700W, Efficiency 0.9
- Target SoC 0.15 (15%), Min SoC 0.1, Max SoC 1.0
- `set_battery_dynamic: True`, `set_use_battery: True`
- SoC sensor: `sensor.total_soc`, Power sensor: `sensor.batteries_power_out_total`
- **Kaskaden-Automation** (`1774430238377`, ON): EMHASS target → BYD (803 Auto default, 802 bei `target<=0 && pv<100`) + Marstek (Rest via `sensor.marstek_target_power`)
- **Template Sensoren:** `sensor.smart_battery_target` (EMHASS), `sensor.marstek_target_power` (Plan A: smart_target − actual_BYD_discharge). `sensor.sma_target_power` ENTFERNT (Double-Counting Bug).
- **Cron:** `a32005d62226` (no_agent, every 15m, runner script) ruft EMHASS Optimization auf.
- **Backup:** `.bak.20261001_175804` für configuration.yaml + automations.yaml
- **NFS-Backups (Fix 2026-10-07):** Supervisor-Mounts `Backup_NFS` (usage=backup) + `Medien_NFS` (usage=media) → **10.0.20.100:/pool01_n2_redundant/HomeAssistant** (vorher tote 10.0.30.100, verloren beim NAS-Crash-Reboot 03.10.). Volle automatische Backups ~03:27 UTC nightly. Monitoring: Hermes-Cron `2006a91fba21` (Alert bei Backup-Alter >30h). Lösungsdoku: `docs/solutions/bug-fixes/2026-10-07-ha-audit-nfs-mount-failure-after-nas-crash.md`
## Related
- [[systems/frigate]] — NVR Integration
- [[systems/noris-ai]] — LLM Vision Provider
- [[reference/ip-map]] — IP Assignments
- [[reference/emhass-cascade-automation-architecture]] — Kaskaden-Architektur Details
+47
View File
@@ -0,0 +1,47 @@
---
title: NUT / UPS Shutdown-Kaskade
category: systems
tags: [nut, ups, apc, proxmox, shutdown, epic-9]
created: "2026-10-08"
modified: "2026-10-08"
---
# NUT / UPS Shutdown-Kaskade
## Überblick
APC Back-UPS ES 700G (USB 051d:0002, SN `5B1940T54935␣␣` — 2 Padding-Spaces, nie per serial matchen!) hängt an **ms-a2-1 (10.0.20.92)**. NUT-Master dort, alle übrigen 7 PVE-Nodes + NAS sind Netclient-Slaves. IaC: `iac-homelab/epic-9-ups-nut/ansible/` (Commits bis `beee480`).
## Topologie
- **Master ms-a2-1:** `nut-driver@apc` (usbhid-ups, vendorid/productid) → `upsd` 0.0.0.0:3493 → `nut-monitor` (upsmon MASTER) + `upssched`
- **Users:** `monuser` (upsmon master, für HA) + `slavemon` (upsmon slave, für Cluster-Clients) — PWs in 1P Vault „Hermes": Items „NUT upsd monuser (ms-a2-1)" + „NUT upsd slavemon (cluster)"
- **Slaves (7 PVE + NAS):** `MODE=netclient`, upsmon `slave` auf `apc@10.0.20.92`, FSD → `/sbin/shutdown -h now`, `nut-server` absent
## Shutdown-Kaskade (BY DESIGN)
1. Stromausfall → UPS meldet ONBATT → upssched startet 60s-Timer
2. Timer läuft ab (>60 s Dauerstörung) → `upsmon -c fsd` → FSD-Broadcast
3. Alle Slaves sehen FSD → sofortiger geordneter Shutdown (PVE fährt Gäste via Guest-Agent)
4. Master wartet `HOSTSYNC 45`s → fährt als letzter runter
- **LOWBATT → sofortiger FSD** (kein Warten auf Timer)
- Kurzunterbrechungen ≤60 s: Timer wird bei ONLINE gecancelt → kein Shutdown
## Betrieb & Verifikation
```bash
upsc apc@10.0.20.92 # Telemetrie (HA: sensor.apc_*)
systemctl is-active nut-monitor # auf ALLEN 9 Maschinen active
ansible-playbook -i inventory.ini site.yml # konvergentes Re-Deploy
# Rollback (Kaskade entschärfen): deploy-nut.yml (disarmiert bewusst)
```
## Grenzen / offene Risiken
- **Runtime 277 s @ 76 % Last** — knapp für Nodes mit vielen Gästen. Remediation: `qm set <vmid> --startup down=60`, zweite UPS (LB>SPOF).
- **n5pro:** pgdg-Repo deformiert (`dists/-pgdg`) — bricht apt-update; NUT-Deploy workaround über no-update_cache. Fix braucht GO.
- Single-UPS = Single Point of Failure für den ganzen Cluster (auch NAS mit ZFS-Pool).
- NAS `nut-driver.target not-found` = harmloses Phantom (Unit-Referenz ohne Paket).
## Monitoring
- **NUT-Watchdog** (Hermes-Cron `4cbf03ccb4de`, jede Minute, no_agent): `~/.hermes/scripts/check_ups_health.sh` fragt upsd via rohem NUT-Protokoll ab (`GET VAR apc ups.status` — anonym möglich, kein Secret). Quiet bei OL; Instant-Alert bei Statuswechsel (OB/LB/upsd down), 10-min-Reminder bei Dauerstörung, einmalige Recovery-Info. State: `~/.hermes/cache/nut_watchdog.state`.
## Related
- [[systems/homeassistant]] — NUT-Integration (Entry `5e1d2138…`, sensor.apc_*)
- [[reference/ip-map]]
- Bugfix-Doku: `docs/solutions/bug-fixes/2026-10-07-nut-ups-ms-a2-1-deployment.md`