Auto-sync: 2026-10-26
This commit is contained in:
@@ -1,25 +1,25 @@
|
|||||||
# User Drift Report — 2026-10-19
|
# User Drift Report — 2026-10-26
|
||||||
|
|
||||||
**Recent window:** Last 14 days (817 msgs)
|
**Recent window:** Last 14 days (1738 msgs)
|
||||||
**Baseline:** Previous 90 days (7645 msgs)
|
**Baseline:** Previous 90 days (8124 msgs)
|
||||||
|
|
||||||
## Detected Drifts
|
## Detected Drifts
|
||||||
|
|
||||||
- **message_length**: Messages 166% longer (baseline: 1943 chars → recent: 5171)
|
- **message_length**: Messages 288% longer (baseline: 1989 chars → recent: 7725)
|
||||||
- **frustration_increase**: Frustration markers rose from 6.7% to 23.4%
|
- **frustration_increase**: Frustration markers rose from 6.8% to 29.6%
|
||||||
- **correction_increase**: Corrections increased from 9.7% to 27.8% — preferences may have shifted
|
- **correction_increase**: Corrections increased from 10.4% to 43.0% — preferences may have shifted
|
||||||
- **new_focus**: New dominant topics: monitoring, security
|
- **new_focus**: New dominant topics: gitops, ai_ml, monitoring
|
||||||
- **declining_focus**: Topics fading from focus: proxmox, ceph
|
- **declining_focus**: Topics fading from focus: backup, security, ceph
|
||||||
|
|
||||||
## Signal Summary
|
## Signal Summary
|
||||||
|
|
||||||
| Metric | Baseline | Recent |
|
| Metric | Baseline | Recent |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Avg msg length | 1943 | 5171 |
|
| Avg msg length | 1989 | 7725 |
|
||||||
| Frustration rate | 6.7% | 23.4% |
|
| Frustration rate | 6.8% | 29.6% |
|
||||||
| Correction rate | 9.7% | 27.8% |
|
| Correction rate | 10.4% | 43.0% |
|
||||||
| Action-first | 1.6% | 2.8% |
|
| Action-first | 1.5% | 3.8% |
|
||||||
| Top topics | ceph, proxmox, k8s | security, backup, gitops |
|
| Top topics | ceph, proxmox, k8s | proxmox, gitops, ai_ml |
|
||||||
|
|
||||||
## Recommendations
|
## Recommendations
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,77 @@
|
|||||||
# Memory Log
|
# Memory Log
|
||||||
|
|
||||||
|
## [2026-10-26 mittag] p0-mcp ALLE 4 FACILITIES GRÜN + Deadman-Cold-Start-Fix + Design-Docs
|
||||||
|
- **Phase 0.8 komplett:** org-registry + board-service + **telemetry**
|
||||||
|
(601435d3) + **provisioning** (fa39b7cd) live, alle E2E-grün aus dem
|
||||||
|
Cluster (ConfigMap-Pod-Muster), psql-Beweise erbracht. Baukasten-Muster
|
||||||
|
dokumentiert (Solution-Doc architecture/2026-10-26-fleet-mcp-facility-baukasten.md).
|
||||||
|
- **Deadman v2 (Cold-Start-Fix):** nie gesehene Agenten = pending (kein
|
||||||
|
Alarm), Beacon-Unerreichbarkeit = EIN fail-closed-Alarm, DEADMAN-ALARM.md
|
||||||
|
auf Board-PVC mit RESOLVED-Zeilen. Digest f700e772, Unit-Tests + live
|
||||||
|
bewiesen (agents_pending 8, beacon_ok 1).
|
||||||
|
- **Beacon 0.11 final:** psql heartbeats-Beweis (count=2, ceo 09:27 UTC).
|
||||||
|
- **Design-Docs (blockieren 0.9/0.13):** sandbox-design.md (21,4 KB, 8
|
||||||
|
Abnahmekriterien, VK-Platzhalter-Fluss, Strict-Gate-422, Blaupausen) +
|
||||||
|
tower-design.md (17,2 KB, Fork-Inventar, 3 Tasks, Manifest-Deploy).
|
||||||
|
- **Subagent-Bilan:** 600s-Timeout systematisch (API-Latenz) — Chair-direct
|
||||||
|
ist der effektive Pfad; Config-Erhöhung child_timeout_seconds greift erst
|
||||||
|
nach Session-Neustart.
|
||||||
|
|
||||||
|
## [2026-10-26 früh] p0-spool + p0-db GRÜN — Spool/Watcher E2E + Fleet-DB + ArgoCD-ignoreDifferences
|
||||||
|
- **Phase 0.7 (Spool+Watcher) abgeschlossen:** `submit_task.py` (Schema-
|
||||||
|
Validierung, Journal, `_invalid/`, fail-loud) + Watcher-Deployment
|
||||||
|
`fleet-spool-watcher` (Image fleet-spool-watcher:fleet-v1, digest
|
||||||
|
c51684fd…, 25/25 Smoke) live. E2E beide Richtungen bewiesen: completed
|
||||||
|
(touch-CMD) + failed (exit-127-Writeback). CephFS-Pitfall gelöst via
|
||||||
|
initContainer chown 1000:1000 (Root-Dir root:root). TASK_RUN_CMD-Override
|
||||||
|
NUR via Git — ArgoCD selfHeal reverts kubectl-Patches sofort.
|
||||||
|
- **Phase 0.10 (Fleet-DB) abgeschlossen:** CNPG `fleet-db` healthy, 7 Tabellen,
|
||||||
|
App-User `app_fleet` als **CNPG Managed Role** (Bootstrap-Owner hat kein
|
||||||
|
CREATEROLE — CREATE ROLE im Job failt, Managed Roles sind der Weg).
|
||||||
|
DROP-Negativtest ok, App-Secret `fleet-db-app` via K8s-API-Readback.
|
||||||
|
- **ArgoCD-Lektion (neu):** `jqPathExpressions` brauchen **jq-Punktnotation**
|
||||||
|
(`.spec.probes`, Wildcards `[]`); JSON-Pointer-Syntax (`/spec/probes`) →
|
||||||
|
`unexpected token "/"`, Mischform → `function not defined`. Plus: K8s-1.35
|
||||||
|
`status.terminatingReplicas` blockte ALLE fleet-Syncs (ComparisonError) —
|
||||||
|
Fix jsonPointers. Nach Application-Spec-Änderung `argocd app get root
|
||||||
|
--refresh` (app-of-apps). Solution Doc:
|
||||||
|
`docs/solutions/bug-fixes/2026-10-26-argocd-ignoredifferences-jq-syntax.md`
|
||||||
|
- **Wiki:** systems/agent-fleet.md um p0-spool/p0-db-Live-Facts erweitert,
|
||||||
|
Phasenliste aktualisiert (0.7 + 0.10 GRÜN).
|
||||||
|
|
||||||
|
## [2026-10-26 früh] p0-bifrost GRÜN — 9 VKs je Agent, Smoke + Prometheus bestätigt
|
||||||
|
- **Phase 0.6 abgeschlossen:** 9 Virtual Keys (ceo-v3, cto-v2, ciso-v2, chro-v2,
|
||||||
|
cmo-v2, agile-coach, coo-v2, devops-v2, research), Werte in 1P „Kubernetes ESO",
|
||||||
|
ExternalSecrets `fleet-vk-<agent>` in NS fleet (ArgoCD fleet Synced/Healthy,
|
||||||
|
Commit 5d5b26b + 5aa4d31), Mapping `fleet-truth/bifrost-vks.yaml`
|
||||||
|
(→ `FLEET_BIFROST_API_KEY` für Phase 0.9). Smoke 9/9 HTTP 200, Deny-by-default
|
||||||
|
403, Prometheus `bifrost_input_tokens_total` je Key 19–38.
|
||||||
|
- **Pitfalls:** `POST /api/governance/virtual-keys` verwirft `allow_all_keys` →
|
||||||
|
`PUT …/{id}` mit `key_ids:["*"]` (PATCH 405); echte API nur `/api/…`
|
||||||
|
(`/api/v1/…` liefert SPA-HTML mit 200); Modellnamen brauchen `noris/`-Präfix;
|
||||||
|
Reasoning-Modelle: Smoke über Status + usage, nicht Content; ESO-Defaults
|
||||||
|
(`conversionStrategy/decodingStrategy/metadataPolicy`, `deletionPolicy: Retain`)
|
||||||
|
in Git pinnen, sonst ArgoCD-Drift bei leerem Diff.
|
||||||
|
- **Altlast:** 1P-Items `VK ceo (v2)`/`coo (v2)`/`cmo (v2)`/`devops (v2)` halten
|
||||||
|
13-Zeichen-Platzhalter (401) — Löschung vorgeschlagen. 9 Alt-VKs noch aktiv
|
||||||
|
(Deaktivierung = Owner-GO).
|
||||||
|
- **Solution Doc:** `docs/solutions/integrations/2026-10-26-bifrost-virtual-keys-per-agent.md`
|
||||||
|
|
||||||
|
## [2026-10-25 spät] Fleet-Rebuild p0-image GRÜN — Gitea-OCI-Push-Kette + ESO-b64dec
|
||||||
|
- **Phase 0.5 abgeschlossen:** Agent-Image `git.schoen.codes/dominik/hermes-agent:fleet-v1`
|
||||||
|
= digest `sha256:32eda413…` in Gitea-Registry (1,07 GB), digest-gepinnt in
|
||||||
|
OpenShell-Values (Commit 640e051), ArgoCD Synced/Healthy, gateway.toml
|
||||||
|
`default_image` live. E2E: Kubelet-Pull 34,6 s, jq intakt. Pull-Secret
|
||||||
|
`agent-fleet-registry-pull` out-of-band NS fleet.
|
||||||
|
- **Solution Docs:** integrations/2026-10-25-gitea-oci-registry-push-chain.md
|
||||||
|
(http-Realm-Bug Gitea 1.27, Traefik-499-Manifest-PITFALL: Blobs idempotent →
|
||||||
|
einfach erneut pushen; port-forward überlebt keine SSH-Session) und
|
||||||
|
bug-fixes/2026-10-25-eso-decodingstrategy-pruned-b64dec.md (ESO v2.2.0 prunt
|
||||||
|
decodingStrategy → b64dec IM Template; Mischung = Doppel-Dekodierung schreibt
|
||||||
|
Fehlerstring als Secret).
|
||||||
|
- **Wiki:** systems/agent-fleet.md (Rebuild-v2.5-Abschnitt + Freeze-Status) und
|
||||||
|
systems/gitea.md (Container-Registry-Sektion + Push-Runbook) aktualisiert.
|
||||||
|
|
||||||
## [2026-10-25 vormittag] Model-Split fleet-weit — 3 Modelle statt 1
|
## [2026-10-25 vormittag] Model-Split fleet-weit — 3 Modelle statt 1
|
||||||
- Owner-Direktive: Last der Agenten auf unterschiedliche Modelle verteilen
|
- Owner-Direktive: Last der Agenten auf unterschiedliche Modelle verteilen
|
||||||
(glm-5.3, glm flash, deepseek). Release-Lane-Verfügbarkeit geprüft:
|
(glm-5.3, glm flash, deepseek). Release-Lane-Verfügbarkeit geprüft:
|
||||||
@@ -674,3 +746,5 @@
|
|||||||
- DST: CEST→CET am So 25.10. — Cron-Umstellung (Morgen 0 6, Mittag 30 11, Abend 30 19 UTC) als Memory-Reminder dokumentiert.
|
- DST: CEST→CET am So 25.10. — Cron-Umstellung (Morgen 0 6, Mittag 30 11, Abend 30 19 UTC) als Memory-Reminder dokumentiert.
|
||||||
- **22.10. 20:15** Agent-Fleet Genesis W0: CEO deployed (CT 161, 10.0.30.162). Herzschlag→Beacon→PG/Prom verifiziert, Inferenz via Bifrost verifiziert (9× 200). W1-Board-Memo (Occam/Hadrian/Diogenes) vom CEO erarbeitet, liegt zur Genehmigung. Spec v5.2 (A15/ADR-018), Commit f8f43d7. Details: systems/agent-fleet.md.
|
- **22.10. 20:15** Agent-Fleet Genesis W0: CEO deployed (CT 161, 10.0.30.162). Herzschlag→Beacon→PG/Prom verifiziert, Inferenz via Bifrost verifiziert (9× 200). W1-Board-Memo (Occam/Hadrian/Diogenes) vom CEO erarbeitet, liegt zur Genehmigung. Spec v5.2 (A15/ADR-018), Commit f8f43d7. Details: systems/agent-fleet.md.
|
||||||
- **23.10. 06:45** Agent-Fleet W1: Board deployed (CT163 cto/Occam, CT164 ciso/Hadrian, CT165 chro/Diogenes — Vollklone CT161, sanitisiert, eigene Heartbeat-Timer). Ghost-Heartbeat-Zwischenfall (geerbte CEO-Timer, 1438 Rows) per Phase-Clustering gepurgt, 308 echte CEO-Rows erhalten. A16: Board hiret ab W2 kollektiv; Slim-Sizing für Worker. Spec v5.3, Commit d04935f. Details: systems/agent-fleet.md.
|
- **23.10. 06:45** Agent-Fleet W1: Board deployed (CT163 cto/Occam, CT164 ciso/Hadrian, CT165 chro/Diogenes — Vollklone CT161, sanitisiert, eigene Heartbeat-Timer). Ghost-Heartbeat-Zwischenfall (geerbte CEO-Timer, 1438 Rows) per Phase-Clustering gepurgt, 308 echte CEO-Rows erhalten. A16: Board hiret ab W2 kollektiv; Slim-Sizing für Worker. Spec v5.3, Commit d04935f. Details: systems/agent-fleet.md.
|
||||||
|
|
||||||
|
- **26.10. (Chair) p0-beacon 0.11 + p0-deadman 0.12 GRÜN** — Beacon: Deployment 1/1, healthz 200, E2E POST /heartbeat ceo → 200 + DB-Row, unbekannter Agent → 422, /metrics live. Drei Root-Causes behoben: fehlende `DATABASE_URL` (envFrom liefert nur Einzelkeys → secretKeyRef auf `uri`), Patch-Indent-Bug (`name:` neben statt unter `secretRef` → ArgoCD ComparisonError blockierte App-Sync), fehlende **Sequence**-Grants für app_fleet (BIGSERIAL → `permission denied for sequence heartbeats_heartbeat_id_seq`; GRANT USAGE,SELECT + ALTER DEFAULT PRIVILEGES in schema.yaml). Deadman: LLM-freier Watchdog (Image ohne hermes/LLM bewiesen), 3 Kaskaden (Agent-Silence 1800 s / Watcher-Tick 180 s / Gateway 120 s anhaltend), Edge-Trigger → 1 P1-Eskalation je Alarm an ceo, Heilung löscht Alert. Watcher v2 mit `_journal/watcher.tick` als Liveness-Signal. Kritischer Validator-Konflikt: from/to waren auf die 9er-Org beschränkt → System-Sender `deadman-watchdog` (nur type=escalation) ergänzt; Envelope-Feld ist `body`, `input` wird quarantäniert (9/9 Tasks durch validate_schema_loose bewiesen). E2E Watcher-Kaskade live: replicas 0 → Tick 105→300 s → ALARM bei 196 s + Task → replicas 1 → GEHEILT. Neue ArgoCD-Lektion: Jobs blockieren App-Sync bei Immutabilität; `Replace=true` scheitert (fehlender selector) → Job als **PostSync-Hook** (hook-delete-policy BeforeHookCreation). `kill -STOP 1` im Pod wirkungslos (PID-Namespace-Init). Details: systems/agent-fleet.md.
|
||||||
|
|||||||
+231
-3
@@ -3,15 +3,243 @@ title: Agent-Fleet
|
|||||||
category: systems
|
category: systems
|
||||||
status: active
|
status: active
|
||||||
created: 2026-10-22
|
created: 2026-10-22
|
||||||
updated: 2026-10-23
|
updated: 2026-10-26
|
||||||
---
|
---
|
||||||
|
|
||||||
# Agent-Fleet (Multi-Agent-Plattform)
|
# Agent-Fleet (Multi-Agent-Plattform)
|
||||||
|
|
||||||
Projekt: Corporation-scale Agent-Fleet mit CEO+SRE-Team, Org-as-Code, zentralem
|
Projekt: Corporation-scale Agent-Fleet mit CEO+SRE-Team, Org-as-Code, zentralem
|
||||||
LLM-Chokepoint und objektiver Telemetrie. Phase 0 (Fundament) weitgehend gebaut.
|
LLM-Chokepoint und objektiver Telemetrie.
|
||||||
|
|
||||||
## Repos & Struktur
|
> **STATUS 25.10.2026:** Iteration 1 (MVP, 7 Hermes-CTs 161/163/164/165/202/205/206)
|
||||||
|
> per Q8a-GO **EINGEFROREN** (Crons `.frozen-20261025`, Protokoll
|
||||||
|
> `/srv/fleet-board-freeze/` auf PVE); Deprovisionierung erst nach Phase-1-Grün.
|
||||||
|
> **Aktiv ist der Rebuild v2.5** (lebende Verfassung, Commit 391c9fa): 9 Agenten
|
||||||
|
> (ceo; cto/ciso/chro/coo/cmo unter ceo; agile-coach unter chro; research unter
|
||||||
|
> cto; devops unter cto) auf RKE2 (CP 10.0.30.51–53, Worker .63–65), Git-Truth
|
||||||
|
> `iac-homelab/clusters/main/fleet-truth/org.yaml`. Dieser Wiki-Abschnitt
|
||||||
|
> **Iteration-1-Betrieb** ist historisch (CTs, Poller, WebUI v0.x, B-Inputs,
|
||||||
|
> Chair-Ticks, Value-Streams) — Rebuild-Details siee Abschnitt
|
||||||
|
> „Fleet-Rebuild v2.5 (Phase 0)" unten.
|
||||||
|
|
||||||
|
## Fleet-Rebuild v2.5 (Phase 0, ab 25.10.2026)
|
||||||
|
|
||||||
|
- **Verfassung:** Prompt v2.5 = lebendes Dokument (Q9b), Owner-GO 25.10.
|
||||||
|
Kernfeatures 1–5 unabänderlich; ereignisgetriebene Starts statt Zeittakt;
|
||||||
|
max. 26 Agenten; Handover an Org nach Phase-1-E2E (danach Chair read-only).
|
||||||
|
- **Phasen:** 0.1 Landlock ✓ · 0.2 Git-Truth ✓ · 0.3 CephFS ✓ · 0.4 OpenShell-
|
||||||
|
Gateway ✓ (Chart 0.1.5, OIDC Authelia, Enforcement via grpcurl verifiziert) ·
|
||||||
|
**0.5 p0-image ✓ GRÜN (25.10. spät)** · **0.6 Bifrost-VKs ✓ GRÜN (26.10.)** ·
|
||||||
|
**0.7 Spool+Watcher ✓ GRÜN (26.10.)** · **0.8 MCP-Facilities ✓ ALLE 4 GRÜN
|
||||||
|
(26.10.)** · 0.9 Sandboxes ·
|
||||||
|
**0.10 Fleet-DB ✓ GRÜN (26.10.)** · **0.11 Beacon ✓ GRÜN (26.10.)** ·
|
||||||
|
**0.12 Deadman ✓ GRÜN (26.10.)** · 0.13 Control Tower · 0.14 E2E → dann Phase 1 Onboarding.
|
||||||
|
- **p0-image (0.5) final:** Agent-Image `git.schoen.codes/dominik/hermes-agent`
|
||||||
|
Tag `fleet-v1` = **digest `sha256:32eda4136f62ef15920f9414c30dd33e9c7736b53a6f02c1658b5584c45a41b7`**
|
||||||
|
(1,07 GB; Basis nousresearch/hermes-agent digest-gepinnt + jq; Dockerfile
|
||||||
|
`clusters/main/fleet/agent-image/Dockerfile`; gebaut auf mgmt-runner .124).
|
||||||
|
**Digest-Pin live** in OpenShell-Values (`sandbox.image.digest`, Commit 640e051,
|
||||||
|
App-Spec + fleet-truth synchron) → gateway.toml `default_image` → ArgoCD
|
||||||
|
Synced/Healthy. Pull-Secret `agent-fleet-registry-pull` out-of-band im NS fleet
|
||||||
|
(default-SA imagePullSecrets; nicht im Git). E2E: Kubelet-Pull 34,6 s auf
|
||||||
|
rke2-cp-01, Container-Start, jq 1.7. Push-Root-Causes + Runbook: siee
|
||||||
|
`systems/gitea.md` „Container Registry".
|
||||||
|
- **OpenShell:** NS openshell, Pod openshell-0, Helm-Chart ghcr.io/nvidia 0.1.5
|
||||||
|
digest-gepinnt; sandbox_image_pull_policy if_not_present; Workspace IMMER
|
||||||
|
/sandbox, UID 1000. Helm-Pitfall: gatewayConfig-Mapping mit EINEM quoted dotted
|
||||||
|
Key "openshell.gateway" (inline `values:`, nie ArgoCD-parameters — dotted Keys
|
||||||
|
brechen --set-Parsing, CrashLoop).
|
||||||
|
- **ArgoCD-Lektionen (wiederkehrend):** dotted Keys nur inline `values:`; nach
|
||||||
|
Git-Push Root-App (app-of-apps) refreshen; Sync via
|
||||||
|
`kubectl -n argocd exec sts/argocd-application-controller -- argocd app sync <app> --core`;
|
||||||
|
org.yaml NICHT im ArgoCD-App-Pfad (CR ohne CRD blockiert Sync) → fleet-truth/.
|
||||||
|
- **Bifrost (Wiederverwendung Iteration 1):** NS agent-fleet, VIP 10.0.30.210:8080,
|
||||||
|
App 2.3.1; 18 VKs live, davon 9 im Rebuild genutzt (s. Tabelle). EIN VK je Member
|
||||||
|
benannt `<agent_id>[-v<n>]`, Rotation `-v<n>`; Allowlist
|
||||||
|
`^(noris/)?vllm/release/.*`, `key_ids ["*"]`, deny-by-default. Abnahme je VK:
|
||||||
|
Completion-Call 200 + Prometheus-Gegenprobe 10.0.30.202:9090.
|
||||||
|
|
||||||
|
### p0-bifrost VK-Tabelle (26.10., live + Secrets in NS fleet)
|
||||||
|
|
||||||
|
| agent_id | VK | Modell (noris/vllm/release/…) | Secret |
|
||||||
|
|---|---|---|---|
|
||||||
|
| ceo | ceo-v3 | glm-5-3 | fleet-vk-ceo |
|
||||||
|
| cto | cto-v2 | glm-5-3 | fleet-vk-cto |
|
||||||
|
| ciso | ciso-v2 | glm-5-3 | fleet-vk-ciso |
|
||||||
|
| chro | chro-v2 | glm-5-3-flash | fleet-vk-chro |
|
||||||
|
| cmo | cmo-v2 | glm-5-3-flash | fleet-vk-cmo |
|
||||||
|
| agile-coach | agile-coach | glm-5-3-flash | fleet-vk-agile-coach |
|
||||||
|
| coo | coo-v2 | deepseek-v41-flash (reasoning_effort low) | fleet-vk-coo |
|
||||||
|
| devops | devops-v2 | deepseek-v41-flash | fleet-vk-devops |
|
||||||
|
| research | research | deepseek-v41-flash | fleet-vk-research |
|
||||||
|
|
||||||
|
- Secrets: `kubernetes.io` ESO-ExternalSecrets aus `clusters/main/fleet/vk-secrets.yaml`
|
||||||
|
(1P-Vault „Kubernetes ESO", Items `Agent-Fleet VK <vk>`, Feld `password`);
|
||||||
|
Keys `api-key` + `key-name`; Zieldatei `FLEET_BIFROST_API_KEY` definiert in
|
||||||
|
`fleet-truth/bifrost-vks.yaml` (Phase-0.9-Injektion).
|
||||||
|
- **Bifrost-API-Pitfalls:** `POST /api/governance/virtual-keys` verwirft
|
||||||
|
`allow_all_keys` → Key ohne Schlüssel (400 „no keys found"); Fix per
|
||||||
|
`PUT …/{id}` mit `key_ids:["*"]` (PATCH = 405). Echte API nur unter `/api/…`
|
||||||
|
(`/api/v1/…` = SPA-HTML 200!). Modellnamen brauchen Provider-Präfix `noris/`.
|
||||||
|
Reasoning-Modelle liefern bei `max_tokens`-Knappheit leeren Content → Smoke
|
||||||
|
über HTTP-Status + `usage.total_tokens`. ESO-Defaults
|
||||||
|
(`conversionStrategy/decodingStrategy/metadataPolicy`, `deletionPolicy: Retain`)
|
||||||
|
müssen in Git gepinnt werden, sonst ArgoCD-Dauerdrift bei leerem Diff.
|
||||||
|
- **Offen (Owner-GO):** 9 Alt-VKs (`fleet`, `ceo`, `ceo-v2`, `cto`, `ciso`, `chro`,
|
||||||
|
`coo`, `cmo`, `devops`) noch aktiv; 4 1P-Items mit 13-Zeichen-Platzhaltern
|
||||||
|
(`VK ceo (v2)`, `coo (v2)`, `cmo (v2)`, `devops (v2)`) löschen.
|
||||||
|
- **Solution Doc:** `docs/solutions/integrations/2026-10-26-bifrost-virtual-keys-per-agent.md`
|
||||||
|
|
||||||
|
### p0-spool (0.7) ✓ GRÜN (26.10.) — Live-Facts
|
||||||
|
- **Deployment `fleet-spool-watcher`** (NS fleet, 1 Replika, Recreate,
|
||||||
|
automountServiceAccountToken false, Label `fleet.noris.de/deadman-cascade: "true"`).
|
||||||
|
Image `git.schoen.codes/dominik/fleet-spool-watcher` Tag `fleet-v1` =
|
||||||
|
**digest `sha256:c51684fd67c95b1aadf91c8930ced0644d7b0180931348d386b7eb51e6e073bf`**
|
||||||
|
(python:3.11-slim + pyyaml 6.0.2, non-root UID 1000).
|
||||||
|
Code: `clusters/main/fleet/spool/` (submit_task.py, watcher/watcher_main.py
|
||||||
|
591 Z., watcher/deployment.yaml, tests/smoke_test_local.py — 25/25 PASS).
|
||||||
|
- **Watcher-Konfig:** scan 10 s, WIP max 2 je to-Agent, Agent-Cooldown 30 s,
|
||||||
|
Budget 2/Zyklus, global max 4 inflight, run_timeout 900 s; TASK_RUN_CMD
|
||||||
|
default `hermes chat {input}` (Platzhalter {input} {agent} {task_id}).
|
||||||
|
- **CephFS-Fix:** initContainer `spool-perms` (busybox:1.36, root) legt
|
||||||
|
`_journal/ _runs/ _invalid/` an + chown 1000:1000 — sonst PermissionError
|
||||||
|
(CephFS-Root root:root vs. Watcher non-root). RWO/RWX-PVC-Mount /srv/fleet/tasks.
|
||||||
|
- **E2E bewiesen:** submitted→dispatched→run_started→run_finished, Status-
|
||||||
|
Writeback ins Envelope; completed-Variante via touch-CMD, failed-Variante
|
||||||
|
(exit 127) — beides im Journal + Envelope nachgewiesen.
|
||||||
|
- **Befehlsmuster:** Toolbox-Pod `kubectl -n fleet run spool-e2e --image=python:3.11-slim`
|
||||||
|
+ PVC-Mount per --overrides; Einreichung via `kubectl cp` + `kubectl exec -i`
|
||||||
|
heredoc; Watcher-E2E-Overrides NUR via Git (selfHeal reverts kubectl-Patches
|
||||||
|
sofort).
|
||||||
|
|
||||||
|
### p0-db (0.10) ✓ GRÜN (26.10.) — Live-Facts
|
||||||
|
- **CNPG-Cluster `fleet-db`** (NS fleet, 1 Instanz, pg 16.4, ceph-flash 5Gi,
|
||||||
|
healthy; App-Secrets via CNPG: `fleet-db-app` Service fleet-db-rw:5432 DB fleet).
|
||||||
|
Manifeste: `clusters/main/fleet/db/` (fleet-db.yaml, schema.yaml).
|
||||||
|
- **App-User `app_fleet`** = CNPG **Managed Role** (`spec.managed.roles`:
|
||||||
|
login, connectionLimit 50, passwordSecret-Ref `fleet-db-app-user` [username+
|
||||||
|
password]; KEIN superuser/createdb/createrole). Bootstrap-Owner `fleet`
|
||||||
|
(Secret `fleet-db-owner` out-of-band) hat KEIN CREATEROLE — CREATE ROLE im
|
||||||
|
Schema-Job schlägt fehl → Managed Roles sind der Weg.
|
||||||
|
- **7 Tabellen** (additiv, IF NOT EXISTS): members, org_snapshots,
|
||||||
|
provisioning_events, sandboxes, trace_meta, heartbeats, gov_samples.
|
||||||
|
heartbeats-Spalten: agent_id, status, task_id, ts, tok_in, tok_out, meta.
|
||||||
|
- **Beweise:** psql select count je Tabelle (Selftest), DROP-Negativtest
|
||||||
|
(verweigert für app_fleet), `SECRET_READBACK_OK`, ArgoCD Synced+Healthy.
|
||||||
|
- **ArgoCD ignoreDifferences-Pitfalls** (Commits 3931c63, 34179d4): K8s-1.35
|
||||||
|
`status.terminatingReplicas` (jsonPointers) + ~45 CNPG-Webhook-Defaults via
|
||||||
|
**jqPathExpressions in jq-Punktnotation** (`.spec.probes` — Slash-Syntax
|
||||||
|
failt im Diff-Normalizer; Wildcards `[]` nicht `*`). Solution Doc:
|
||||||
|
`docs/solutions/bug-fixes/2026-10-26-argocd-ignoredifferences-jq-syntax.md`.
|
||||||
|
|
||||||
|
### p0-beacon (0.11) ✓ GRÜN (26.10.) — Live-Facts
|
||||||
|
- **Deployment `fleet-beacon`** (NS fleet, 1/1, digest `55718277…`), Code aus
|
||||||
|
`/home/debian/workspace/agent-fleet/src/beacon/` (FastAPI, 14/14 Tests),
|
||||||
|
deployt unter `clusters/main/fleet/beacon/` (app.py, deployment.yaml,
|
||||||
|
containerPort 8000, sync-wave 6). Persistenz via Secret `fleet-db-app`.
|
||||||
|
- **3 Root-Causes auf dem Weg (alle gefixt):** (1) app.py liest `DATABASE_URL`,
|
||||||
|
`envFrom: secretRef` liefert aber nur Einzelkeys → healthz 503 (fail-closed).
|
||||||
|
Fix: `valueFrom.secretKeyRef{name: fleet-db-app, key: uri}` (Commit f13c744).
|
||||||
|
(2) Patch-Indent-Bug: `name:` neben statt unter `secretRef` → ArgoCD
|
||||||
|
`ComparisonError: envFrom[0].name: field not declared in schema` blockierte
|
||||||
|
den GESAMTEN App-Sync (Commit fe5cc10). (3) `app_fleet` hatte Tabellen- aber
|
||||||
|
keine **Sequence**-Rechte → `permission denied for sequence
|
||||||
|
heartbeats_heartbeat_id_seq` beim INSERT. Live-GRANT + Git-Fix in
|
||||||
|
schema.yaml (`GRANT USAGE, SELECT ON ALL SEQUENCES` + ALTER DEFAULT
|
||||||
|
PRIVILEGES, Commit ca7660b).
|
||||||
|
- **E2E:** POST /heartbeat ceo → 200 + DB-Row; unbekannter Agent → **422**
|
||||||
|
(nicht 403 — Strict-Gate); /metrics counter + age-Gauge; TTL 3600 s.
|
||||||
|
- **Sequence-Lektion:** BIGSERIAL-PKs brauchen GRANT USAGE auf Sequences —
|
||||||
|
Tabellen-Grants allein reichen nicht (gilt für alle künftigen Facility-Inserts).
|
||||||
|
|
||||||
|
### p0-deadman (0.12) ✓ GRÜN (26.10.) — Live-Facts
|
||||||
|
- **Deployment `fleet-deadman`** (NS fleet, 1/1, sync-wave 7, digest
|
||||||
|
`sha256:f700e772dc434af15b9d88330cbe7e3392332d5c753e1b29ce955331e55e267e`
|
||||||
|
— v2 „coldstart-fix", Pod fleet-deadman-68cb6d9f6f-*), Code
|
||||||
|
`clusters/main/fleet/deadman/{deadman_main.py, Dockerfile, deployment.yaml}`.
|
||||||
|
Image python:3.11-slim + pyyaml, **KEIN LLM, kein hermes** (bewiesen: `NO_HERMES`,
|
||||||
|
pip list = stdlib+PyYAML), non-root UID 1000, `/metrics`+`/healthz` auf 8100,
|
||||||
|
readiness/liveness tcpSocket. **Board-PVC rw gemountet** (FLEET_BOARD_DIR).
|
||||||
|
- **VIER Kaskaden (§3.5, v2):** Agent-Silence 1800 s (Quelle Beacon
|
||||||
|
`/metrics` → `beacon_last_heartbeat_age_seconds`), Watcher-Tick 180 s
|
||||||
|
(`_journal/watcher.tick` mtime), Bifrost-Gateway 120 s anhaltend
|
||||||
|
(`/health`), **Beacon-Unerreichbarkeit fail-closed** (EIN `beacon`-Alarm,
|
||||||
|
KEINE 9 Fake-Alarme; Zustand unbekannt ≠ Stille; kein Heal bestehender
|
||||||
|
Alarme). **Edge-Trigger**: ein P1-Eskalations-Task je OK→ALARM an ceo;
|
||||||
|
Heilung löscht den Alert (`GEHEILT`-Log), kein Task-Spam.
|
||||||
|
- **Cold-Start-Semantik (Fix 26.10.):** Agent OHNE je gesehenen Heartbeat
|
||||||
|
= **pending** (KEIN Alarm; Metrik `deadman_agents_pending`). ALARM erst
|
||||||
|
wenn Heartbeat da WAR und dann 30 min still. Begründung: Phase-1-Agenten
|
||||||
|
senden noch keine Heartbeats — 9 Fehlalarme ab Tag 1 wären Spec-Verstoß.
|
||||||
|
Unit-Test bewiesen: Cold-Start → 0 `agent:*`-Alarme; Heilung → RESOLVED.
|
||||||
|
- **DEADMAN-ALARM.md** (Board-PVC, HITL-Kanal §3.5): je Ursache EINE
|
||||||
|
`[AKTIV]`-Zeile, Zustandswechsel-Updates statt Spam, Heilung schreibt
|
||||||
|
`[RESOLVED]` in den append-only-Verlauf; temporäre Datei + os.replace
|
||||||
|
(atomic). Schreibfehler = Log, nie Crash (Task+/metrics bleiben primär).
|
||||||
|
- **Metrics:** deadman_silence_seconds{agent}, deadman_alert_active{source},
|
||||||
|
deadman_agents_pending, deadman_beacon_ok, deadman_watcher_tick_age_seconds,
|
||||||
|
deadman_gateway_ok, deadman_last_scrape_age_seconds, deadman_up.
|
||||||
|
- **Live-Beweis (26.10.):** `agents_pending 8`, `beacon_ok 1`, keine
|
||||||
|
`agent:*`-Alarme, Watcher-Tick 6 s; Board-rw (touch-Test) OK.
|
||||||
|
- **Validator-Konflikt (kritisch):** submit_task.py UND watcher_main.py
|
||||||
|
erlaubten from/to nur aus der 9er-Org → Deadman-Tasks wären quarantäniert.
|
||||||
|
Fix: `VALID_SYSTEM_SENDERS=["deadman-watchdog"]` / `SUBMIT_SYSTEM_SENDERS`,
|
||||||
|
nur als Absender, nur `type=escalation`, `to` bleibt 9er-Org.
|
||||||
|
Envelope-Feld heißt **`body`** (Watcher-Whitelist REQUIRED_FIELDS +
|
||||||
|
KNOWN_OPTIONAL_FIELDS = reply_to|requires_ack|agent_card_ref|thread_id|topic|body;
|
||||||
|
`input` → Quarantäne). Beweis: 9/9 Tasks durch `validate_schema_loose`.
|
||||||
|
- **Watcher v2** (digest `sha256:e7bc819cf8b83ba64d0f4494972f812422b786e11dc73d396e0c9ac3f7ed36a8`):
|
||||||
|
schreibt `_journal/watcher.tick` (epoch-int, fsync) je Zyklus als Liveness-Signal;
|
||||||
|
System-Sender-Whitelist.
|
||||||
|
- **E2E live:** 8 Agenten ohne Heartbeat → 8 Alarme (ceo korrekt aus, hatte
|
||||||
|
Heartbeat); Gateway 0, Watcher 0. **Watcher-Kaskade bewiesen**: replicas 0
|
||||||
|
(via Git, Commit-Historie) → Tick 105→300 s → bei 196 s ALARM + Task
|
||||||
|
`ac15c65b` → replicas 1 → `GEHEILT watcher`, alert 0, Tick 3 s.
|
||||||
|
- **ArgoCD-Job-Immutabilität (neue Lektion):** `fleet-db-schema` Job blockierte
|
||||||
|
JEDEN App-Sync („field is immutable"; `Replace=true` scheitert an fehlendem
|
||||||
|
selector). Lösung: Job als **PostSync-Hook** (`argocd.argoproj.io/hook: PostSync`
|
||||||
|
+ `hook-delete-policy: BeforeHookCreation`) → nicht getrackt, konvergiert
|
||||||
|
idempotent bei jedem Sync. Commit 077c5c8.
|
||||||
|
- **Grenze des Kill-Tests:** `kill -STOP 1` im Pod wirkt NICHT (PID-Namespace-Init
|
||||||
|
ignoriert Signale ohne Handler, kein CAP_KILL) — Kaskadentest deshalb über
|
||||||
|
Git-replicas, nicht über Signale. `kubectl scale` wird von ArgoCD selfHeal
|
||||||
|
binnen ~3 min revertiert → Overrides IMMER via Git.
|
||||||
|
|
||||||
|
### p0-mcp (0.8) ✓ ALLE 4 FACILITIES GRÜN (26.10.) — Live-Facts
|
||||||
|
- **Baukasten:** alle 4 Facilities nach demselben Muster (mcp==2.3.0,
|
||||||
|
`MCPServer` + `run(transport="streamable-http", stateless_http=True)`,
|
||||||
|
tcpSocket-Probe — httpGet auf /mcp = 406, sync-wave 5, Recreate,
|
||||||
|
automountServiceAccountToken false, deadman-cascade-Label, UID 1000,
|
||||||
|
Digest-Pins). Vorlage: `clusters/main/fleet/mcp/org-registry/`.
|
||||||
|
Solution-Doc: `docs/solutions/architecture/2026-10-26-fleet-mcp-facility-baukasten.md`.
|
||||||
|
- **1/4 org-registry** (digest `sha256:dc3f53e2f57e91ba5796426046cbda08c51c4f832103f9ee028bd9ae6949913e`):
|
||||||
|
Tools list_members/get_member/health aus org.yaml (CM fleet-org-truth,
|
||||||
|
Datei-Mount, fail-loud SystemExit 2). In-cluster E2E 9 Agents.
|
||||||
|
- **2/4 board-service** (digest `sha256:b2b3c6125004103e8c83889c0ffc6f80ad75291c8e458f49cb9210b7cb66be6a`,
|
||||||
|
Commit 356bfcb): 7 Tools (add_backlog_card, add_release_note, get_backlog,
|
||||||
|
get_kanban, health, list_value_streams, move_card); Single-Writer aufs
|
||||||
|
Board-PVC, Spool-Submit; E2E 13/13 (Karte e2e-001, Spool-Quittung).
|
||||||
|
- **3/4 telemetry** (digest `sha256:601435d3ca1fb84e6ccc4f05c97e201cbc066560e2729531267025c33e12bba7`,
|
||||||
|
Commit aec669d): Tools query_prometheus (NUR GET, PromQL frei),
|
||||||
|
get_agent_tokens (9 Default-VKs, rate[2m] in+out), get_beacon_status
|
||||||
|
(Heartbeat-Age je Agent), health. E2E 6/6 in-cluster + fail-loud
|
||||||
|
(kaputtes PromQL → isError true). Prometheus/Beacon-ENV konfigurierbar.
|
||||||
|
- **4/4 provisioning** (digest `sha256:fa39b7cd3abd99158e9de5918507ea4d75a9596740c5e70f656b378cf54fd4b1`,
|
||||||
|
Commit f07c7d0): Tools validate_hire (Pflichtfelder + member_id-Doppelt-
|
||||||
|
ausschluss + A16-Vote-Datei im Board-PVC + CISO/CHRO-Gates → ok/fehlerliste),
|
||||||
|
record_provisioning_event (INSERT provisioning_events, psql-bewiesen
|
||||||
|
event_id=1), get_sandbox_status (Strict-Gate: member_id ∈ org.yaml sonst
|
||||||
|
isError; `none` = noch nicht provisioniert), health (DB-Connect).
|
||||||
|
**Gateway-Call STUBBED** (provision_sandbox_stub → NotImplementedError,
|
||||||
|
Aktivierung 0.9 laut sandbox-design.md). Board-PVC RO + initContainer
|
||||||
|
board-perms chown. envFrom fleet-db-app: Keys heißen **user/password/host/
|
||||||
|
port/db** (NICHT DB_USER — ENV-Name = Secret-Key!).
|
||||||
|
- **E2E-Muster (wiederverwendbar):** Test-Skript als ConfigMap + python:3.11-alpine-
|
||||||
|
Pod mit --overrides-Mount; RPC via POST + `Accept: application/json,
|
||||||
|
text/event-stream`, letzte `data: `-Zeile parsen. python:3.11-alpine hat KEIN
|
||||||
|
kubectl — Board-Test-Dateien vom Host via kubectl exec anlegen.
|
||||||
|
|
||||||
|
## Repos & Struktur (Iteration 1 — historisch)
|
||||||
- **agent-fleet** (Gitea, lokal `/home/debian/workspace/agent-fleet`): Org-YAML,
|
- **agent-fleet** (Gitea, lokal `/home/debian/workspace/agent-fleet`): Org-YAML,
|
||||||
Beacon-Scaffold (`src/beacon/`), Pläne (`docs/plans/2026-10-22-agent-fleet-phase0.md`,
|
Beacon-Scaffold (`src/beacon/`), Pläne (`docs/plans/2026-10-22-agent-fleet-phase0.md`,
|
||||||
inkl. Bifrost-Pivot-Vermerk — LiteLLM-Abschnitte A3/A4 sind superseded).
|
inkl. Bifrost-Pivot-Vermerk — LiteLLM-Abschnitte A3/A4 sind superseded).
|
||||||
|
|||||||
@@ -43,6 +43,30 @@ modified: "2026-09-27"
|
|||||||
- **DinD Pitfalls:** docker:27-dind creates docker.sock as directory → use TCP mode (--host=tcp://, --tls=false). act_runner:0.2.13 has no docker CLI → use nc for port check. act_runner:0.2.13 doesn't auto-register → wrapper script needed.
|
- **DinD Pitfalls:** docker:27-dind creates docker.sock as directory → use TCP mode (--host=tcp://, --tls=false). act_runner:0.2.13 has no docker CLI → use nc for port check. act_runner:0.2.13 doesn't auto-register → wrapper script needed.
|
||||||
- **Solution Doc:** `docs/solutions/bug-fixes/2026-07-24-gitea-actions-k8s-runner-task-assignment-bug.md`
|
- **Solution Doc:** `docs/solutions/bug-fixes/2026-07-24-gitea-actions-k8s-runner-task-assignment-bug.md`
|
||||||
|
|
||||||
|
## Container Registry (OCI, seit 25.10.2026 verifiziert push+pull-fähig)
|
||||||
|
|
||||||
|
- **Repos:** `git.schoen.codes/v2/dominik/hermes-agent` (Fleet-Agent-Image, digest-gepinnt
|
||||||
|
`sha256:32eda413…` = Tag `fleet-v1`, 1,07 GB) und `dominik/beacon` (Iteration 1).
|
||||||
|
- **Zertifikat:** Internes Wildcard `*.schoen.codes` via ESO/1P (`git-schoen-codes-tls`,
|
||||||
|
template `b64dec`, Commit e8b006b) — Traefik serviert es per SNI. Push/CI NUR gegen
|
||||||
|
`git.schoen.codes`; `git.familie-schoen.com` terminiert am externen Proxy mit
|
||||||
|
LE-Cert (deckt familie-schoen.com, nicht *.schoen.codes) und erzeugt den
|
||||||
|
http-Realm-Bug.
|
||||||
|
- **KNOWN BUG (Gitea 1.27.0):** Token-Realm-URL wird per
|
||||||
|
`GuessCurrentHostURL` aus Request-Headern gebaut, nicht aus ROOT_URL. Ohne
|
||||||
|
`X-Forwarded-Proto` (öffentlicher Pfad) → `realm="http://…/v2/token"` → Docker
|
||||||
|
stript Auth beim 301→https-Redirect → anonymes Token → **Push 401** (Pull
|
||||||
|
klappt, wenn Repo public). Intern hinter Traefik: Header vorhanden, Realm https.
|
||||||
|
- **Push-Runbook:** (1) `docker login git.schoen.codes` VOR dem Push; (2) bei
|
||||||
|
HTTP 499 am Manifest-PUT NICHT neu anfangen — Blobs sind idempotent drin,
|
||||||
|
einfach erneut pushen; (3) kubectl port-forward überlebt keine SSH-Session und
|
||||||
|
ist für Bulk-Pushes zu langsam; (4) tags/list zeigt nur Tags mit eigener
|
||||||
|
Manifest-Revision — Beweis via `GET /v2/…/manifests/<tag>` (200).
|
||||||
|
- **Kubelet-Pull:** Node-DNS löst git.schoen.codes auf; Pull-Secret
|
||||||
|
`agent-fleet-registry-pull` (NS fleet, default-SA imagePullSecrets, out-of-band,
|
||||||
|
nicht im Git). E2E: 1,07 GB in 34,6 s auf rke2-cp-01.
|
||||||
|
- **Solution Doc:** `docs/solutions/integrations/2026-10-25-gitea-oci-registry-push-chain.md`
|
||||||
|
|
||||||
## IaC Remotes
|
## IaC Remotes
|
||||||
- iac-homelab hat EINEN remote: `origin` = K8s Gitea (git.schoen.codes)
|
- iac-homelab hat EINEN remote: `origin` = K8s Gitea (git.schoen.codes)
|
||||||
- Alter CT108 remote wurde entfernt (CT108 decommissioned)
|
- Alter CT108 remote wurde entfernt (CT108 decommissioned)
|
||||||
|
|||||||
Reference in New Issue
Block a user