Auto-sync: 2026-07-24

2026-07-24 22:00:15 +00:00
parent de97cd6d20
commit 1643a0a6b5
36 changed files with 920 additions and 499 deletions
@@ -1,48 +0,0 @@
 ---
 title: Monitoring-System
 category: Infrastructure
 tags: [monitoring, network-scans, security, cronjob]
 created: "2026-04-28"
 modified: "2026-04-28"
 ---
 # Monitoring-System
 ## Netzwerk-Scans
 - **Häufigkeit:** Wöchentlich
 - **Cronjob:** SRE-Agent (infra-sre)
 - **Scan-Bereiche:**
  - 10.0.10.x/24 (Gateway)
  - 10.0.20.x/24 (Hypervisoren)
  - 10.0.30.x/24 (Container/VMs)
  - 10.0.40.x/24 (Clients)
  - 10.0.50.x/24 (Smart Home, 26 hosts)
  - 10.0.60.x/24 (DMZ/Reverse Proxies)
 ## Smart Home Monitoring
 - **Hosts:** 26 in 10.0.50.x
 - **Scan-Methode:** SEGMENTS Dictionary
 - **Ziel:** Geräte-Status und Verfügbarkeit
 ## Security-Checks
 - **SSH-Versionen:** 9.2p1-10.0p2 (gemischt)
 - **TLS-Zertifikate:** Keine CN, self-signed oder SAN-only
 - **Ports:** 2222 (SSH), 3306 (MySQL)
 ## Wichtige Befehle
 ```bash
 # Netzwerk-Scan durchführen
 nmap -sn 10.0.20.0/24
 # SSH-Version prüfen
 ssh -V
 # TLS-Zertifikat prüfen
 openssl s_client -connect <host>:<port>
 ```
 ## Cronjobs
 - **Netzwerk-Scan:** Wöchentlich
 - **Smart Home Scan:** Täglich
 - **Rechnungen-Organizer:** Täglich 09:00 Uhr
 - **Memory-Sync:** Täglich 06:00, 12:00, 18:00 Uhr
@@ -1,41 +0,0 @@
 ---
 title: Proxmox Cluster
 category: Infrastructure
 tags: [proxmox, virtualization, containers, lxc]
 created: "2026-04-28"
 modified: "2026-04-28"
 ---
 # Proxmox Cluster
 ## Cluster-Konfiguration
 - **Hypervisoren:** In 10.0.20.x
 - **Container:** In 10.0.30.x
 - **VLAN-basiertes Netzwerk** mit Bridges
 - **Shared Storage:** ZFS/ISCSI
 ## Wichtige Befehle
 ```bash
 # Container-Status prüfen
 pct status <vmid>
 # Container starten/stoppen
 pct start <vmid>
 pct stop <vmid>
 # Container konfigurieren
 pct setcfg <vmid> <config>
 # Cluster-Status
 pvecm status
 ```
 ## VLAN-Konfiguration
 - **Bridge:** vmbr0 für Netzwerk
 - **VLAN Tagging:** Für Segmentierung
 - **IP-Adressierung:** 10.0.X.Y (siehe Netzwerk-Architektur)
 ## Monitoring
 - **Netzwerk-Scans:** Wöchentlich via cronjob
 - **Smart Home:** 26 hosts in 10.0.50.x
 - **SEGMENTS Dictionary:** Zentrale Konfiguration für Scan-Intervalle
@@ -1,48 +0,0 @@
 ---
 title: qmd Wissenssystem
 category: Tools
 tags: [qmd, knowledge-base, embeddings, search]
 created: "2026-04-28"
 modified: "2026-04-28"
 ---
 # qmd Wissenssystem
 ## Überblick
 qmd ist ein markdown-basiertes Wissenssystem mit:
 - **Embeddings:** Semantische Suche
 - **Suche:** qmd search
 - **Index:** ~/.cache/qmd/
 - **MCP Server:** qmd mcp --http
 ## Installation
 ```bash
 npm install -g @tobilu/qmd
 ```
 ## Verwendung
 ```bash
 # Embeddings generieren
 qmd embed
 # Semantische Suche
 qmd search "query"
 # MCP Server starten
 qmd mcp --http
 # Server als Daemon starten
 qmd mcp --http --daemon
 ```
 ## Memory-Integration
 - **Verzeichnis:** ~/.hermes/memory/
 - **Struktur:** Entities/, Concepts/, Projects/, Templates/
 - **Sync:** Git + Cronjobs
 - **Index:** qmd embed für neue/änderte Einträge
 ## Arbeitsweise
 1. **Manuell:** Markdown-Dateien werden direkt geschrieben
 2. **Automatisch:** Cronjobs syncen und indexieren
 3. **Suche:** qmd search für semantische Suche
 4. **Index:** qmd embed für neue/änderte Einträge
@@ -1,43 +0,0 @@
 ---
 title: Infrastruktur-Übersicht
 category: Infrastructure
 tags: [networking, proxmox, monitoring, gitea]
 created: "2026-04-28"
 modified: "2026-04-28"
 ---
 # Infrastruktur-Übersicht
 ## Netzwerk-Architektur (Schön Consulting)
 - **Addressing:** 10.0.X.Y (nicht standard /8-Notation)
  - 10.0.10.x = Gateway
  - 10.0.20.x = Hypervisoren (Proxmox Cluster)
  - 10.0.30.x = Container/VMs
  - 10.0.40.x = Clients
  - 10.0.50.x = Smart Home (26 hosts)
  - 10.0.60.x = DMZ/Reverse Proxies
 - **TLS:** Alle Zertifikate haben kein CN (self-signed oder SAN-only), keine reverse-DNS
 - **MySQL:** 3306 auf mehreren VMs in 30.x Subnetz exponiert
 - **SSH:** Port 2222 auf allen Servern
 - **SSH Versionen:** Gemischt (9.2p1-10.0p2)
 ## Proxmox Cluster
 - **Hypervisoren:** In 10.0.20.x
 - **Container:** In 10.0.30.x
 - **VLAN-basiertes Netzwerk** mit Bridges
 ## Gitea Instance
 - **URL:** https://git.familie-schoen.com
 - **Auth:** SSH Key-basiert
 - **Git User:** Dominik Schön <dominik@familie-schoen.com>
 - **Repository:** memory.git (für Memory-Sync)
 ## Monitoring
 - **Netzwerk-Scans:** Wöchentlich via cronjob
 - **Smart Home:** 26 hosts in 10.0.50.x
 - **SEGMENTS Dictionary:** Zentrale Konfiguration für Scan-Intervalle
 ## Wichtige Dienste
 - **Embedding Model:** embeddinggemma-300M-Q8_0.gguf auf 10.0.30.92:8080 (OpenAI-kompatibel, CLS pooling)
 - **Himalaya v1.2.0:** iCloud Mail (Email-Client)
 - **qmd:** Markdown-basiertes Wissenssystem mit Embeddings
@@ -1,50 +0,0 @@
 ---
 title: Memory-System
 category: Tools
 tags: [memory, hindsight, qmd, markdown]
 created: "2026-04-28"
 modified: "2026-04-28"
 ---
 # Memory-System
 ## Hindsight Memory System
 - **Version:** 1.0
 - **Provider:** OpenAI (via custom endpoint)
 - **Model:** embeddinggemma-300M-Q8_0.gguf
 - **Endpoint:** http://10.0.30.92:8080
 - **OpenAI-kompatibel:** Ja
 - **CLS Pooling:** Ja
 ## Hindsight Daemon
 - **Status:** Läuft auf 10.0.30.92
 - **Config:** ~/.hindsight/profiles/hermes.env
 - **API Key:** In hermes.env gespeichert
 - **Log:** ~/.hindsight/daemon.log
 ## qmd Integration
 - **qmd:** Markdown-basiertes Wissenssystem
 - **Embeddings:** Via qmd embed
 - **Suche:** Via qmd search
 - **MCP Server:** qmd mcp --http
 - **Index:** ~/.cache/qmd/
 ## Memory-Verzeichnis
 - **Pfad:** ~/.hermes/memory/
 - **Struktur:**
  - Entities/ (Konkrete Dinge, Personen, Systeme)
  - Concepts/ (Abstrakte Ideen, Prinzipien, Methoden)
  - Projects/ (Aktuelle und abgeschlossene Projekte)
  - Templates/ (Vorlagen für neue Einträge)
 ## Git-Sync
 - **Remote:** https://git.familie-schoen.com/dominik/memory.git
 - **Branch:** main
 - **User:** Dominik Schön <dominik@familie-schoen.com>
 - **Commit-Messages:** "Auto-sync: YYYY-MM-DD HH:MM"
 ## Arbeitsweise
 1. **Manuell:** Memory-Einträge werden direkt in Markdown geschrieben
 2. **Automatisch:** Cronjobs syncen und indexieren
 3. **Suche:** qmd search für semantische Suche
 4. **Index:** qmd embed für neue/änderte Einträge
@@ -0,0 +1,47 @@
 ---
 title: Credential Policy
 category: concepts
 tags: [credentials, security, 1password, secrets]
 created: "2026-07-24"
 modified: "2026-07-24"
 ---
 # Credential Policy
 ## Regel
 Credentials, API-Keys und Secrets gehören in den 1Password Vault "Hermes" und werden von dort abgerufen (via `op` CLI). Passwörter in .env-Dateien sind ebenfalls akzeptabel. **Keine Secrets in Git-Repositories.**
 ## 1Password Vaults
 | Vault | Zweck | Used By |
 |-------|-------|---------|
 | Hermes | General secrets | Hermes Agent |
 | Kubernetes ESO | K8s ExternalSecrets Operator | ESO → K8s Secrets |
 ## ESO (External Secrets Operator)
 - Liest aus Vault "Kubernetes ESO"
 - Failing ExternalSecrets burn 1000 requests/hr
 - Bei Problemen: ESO scale 0, warten, fixen, wieder scale 1
 ## Bekannte Issues
 - **`mariadb-root` Item in 1Password**: PASSWORT IST FALSCH — echtes PW in `/root/.my.cnf` auf VM300
 - **1Password CLI v2.35**: assignment statements (`username=val`), NOT `--username` flags
 - **op CLI**: `--reveal` für blocked fields
 - **Account-Level Rate Limit**: 1Password rate-limited auf ACCOUNT-Ebene, nicht per SA. ESO (25 ExternalSecrets × 1h) kann das gemeinsame Kontingent erschöpfen → op item get auf Hermes Host schlägt fehl. Fix: ESO scale 0, 1h warten, refresh intervals auf 6h erhöhen. Siehe Solution Doc: `docs/solutions/architecture/2026-07-24-1password-account-level-rate-limit.md`
 ## K8s Secret Pattern
 ```yaml
 # ExternalSecret referenziert 1Password Item
 apiVersion: external-secrets.io/v1beta1
 kind: ExternalSecret
 spec:
  data:
    - secretKey: password
      remoteRef:
        key: "Paperless DB Pass"  # 1Password Item Title
        property: password
 ```
 ## Related
 - [[reference/ssh-keys]]
 - [[systems/gitea]] (NIEMALS öffentlich — enthält Secrets)
 - [[systems/hindsight]]
@@ -0,0 +1,41 @@
 ---
 title: GitOps Workflow
 category: concepts
 tags: [gitops, argocd, iac, terraform, ansible, workflow]
 created: "2026-07-24"
 modified: "2026-07-24"
 ---
 # GitOps Workflow
 ## Prinzip
 Infra-Changes **bevorzugt** über GitOps. Geht nicht immer. Ausnahmen müssen von Dominik freigegeben werden.
 ## Workflow
 1. IaC Repo lokal auschecken (`/root/iac-homelab` auf VM200)
 2. Ändern (Tofu/Ansible/K8s Manifeste)
 3. Commit + Push (zu BEIDEN Gitea Remotes: origin + k8s)
 4. ArgoCD sync (oder auto-sync)
 5. Verify (kubectl get, curl, etc.)
 6. Lokale Kopie löschen
 ## ArgoCD
 - 15 Applications via SSH Deploy Keys
 - `ssh://gitea@git.schoen.codes:22/dominik/iac-homelab.git`
 - `--force` kann nicht mit ServerSideApply kombiniert werden
 - Bei hängendem Operation-State: App terminieren und neu anstoßen
 ## IaC Tools
 - **OpenTofu**: VM/CT Provisioning, GPU PCI Mapping
 - **Ansible**: RKE2 Installation, Node Taints, Fluent Bit
 - **K8s Manifeste**: clusters/main/apps/*.yaml
 ## Was NICHT über GitOps geht
 - Direkte DB-Operationen (SQL)
 - Ceph `ceph` CLI Befehle
 - Proxmox `pct`/`qm` Live-Operationen
 - Notfall-Reparaturen
 ## Related
 - [[systems/rke2-kubernetes]]
 - [[systems/gitea]]
@@ -1,25 +1,24 @@
 ---
 title: Netzwerk-Architektur
-category: Infrastructure
+category: concepts
 tags: [networking, ip-addressing, subnets, vlan]
 created: "2026-04-28"
-modified: "2026-04-28"
+modified: "2026-07-24"
 ---
 # Netzwerk-Architektur
 ## IP-Adressierung (10.0.X.Y)
 Nicht-standard /8-Notation, aber mit klarem Schema:
 - 10.0.10.x = Gateway
- 10.0.20.x = Hypervisoren (Proxmox Cluster)
+- 10.0.20.x = Hypervisoren (Proxmox Cluster, 9 Nodes)
- 10.0.30.x = Container/VMs
+- 10.0.30.x = Container/VMs (Services)
 - 10.0.40.x = Clients
 - 10.0.50.x = Smart Home (26 hosts)
 - 10.0.60.x = DMZ/Reverse Proxies
 ## VLAN-Struktur
 - Jedes Segment hat eigenes VLAN
- Proxmox Bridge-Konfiguration
+- Proxmox Bridge-Konfiguration (vmbr0)
 - Inter-Segment-Kommunikation via Gateway
 ## TLS-Zertifikate
@@ -28,6 +27,13 @@ Nicht-standard /8-Notation, aber mit klarem Schema:
 - Keine reverse-DNS Einträge
 ## Wichtige Ports
- **MySQL:** 3306 (auf mehreren VMs in 30.x)
+Siehe [[reference/ports]] für vollständige Liste.
- **SSH:** 2222 (alle Server)
+
- **Embedding Model:** 8080 (10.0.30.92)
+## DNS
 - externe Domains: familie-schoen.com, schoen.codes, essen.schoen.eu
 - interne Auflösung über K8s CoreDNS
 ## Related
 - [[reference/ip-map]]
 - [[reference/ports]]
 - [[systems/proxmox-cluster]]
@@ -1,33 +0,0 @@
 # Dominik Schön
 ## Overview
 CEO Schön Consulting Inc. PhD (Dr.). Former CodeCamp:N, Noris Network.
 ## Personal Details
 - **Family:** Sarah (Wife, loves carbs), Cleo (4y, loves pasta)
 - **Health Goals:** -15kg target (currently 95kg, 1.75m)
 - **Food Preferences:** Asian cuisine preferred. No allergies.
 - **Shopping:** Aldi (Fridays). HelloFresh occasionally.
 ## Work Rhythm
 - Home Office: Mon/Wed/Fri
 - Meal Prep: Tue/Thu
 - Meals: Daily dinner. Mon/Wed/Fri. Weekend: 2 warm meals.
 - Cleo Kitabox: Mon-Thu
 ## Communication
 - Prefers Du-form (German informal)
 - CEO persona for interactions
 ## Network Architecture
 - **Addressing:** 10.0.X.Y where X = segment index
  - 10.0.10.x = Gateway
  - 10.0.20.x = Hypervisoren (Proxmox cluster)
  - 10.0.30.x = Container/VMs
  - 10.0.40.x = Clients
  - 10.0.50.x = Smart Home (26 hosts)
  - 10.0.60.x = DMZ/Reverse Proxies
 - All TLS certs have no CN (self-signed or SAN-only)
 - MySQL (3306) exposed on multiple VMs in 30.x subnet
 - SSH on port 2222
 - Mixed OpenSSH versions: 9.2p1-10.0p2 across the fleet
@@ -1,24 +0,0 @@
 # Email Organization
 ## System
 - **Client:** Himalaya v1.2.0 with iCloud
 - **Config:** /home/debian/.config/himalaya/config.toml
 - **Primary:** dominik.schoen87@icloud.com
 - **Custom Domains:** 12 active
  - grafiert.de
  - famschoen.eu
  - dominikschoen.de
  - familie-schoen.com
  - schoen.codes
  - schoen.eu
 ## Folder Structure
 - **Aliases:** Sent=Sent Messages, Trash=Deleted Messages
 - **Organization:** Hierarchical (e.g., Rechnungen/Bestellungen)
 - **Daily Cronjob:** 9:00 AM invoice organizer
  - Invoices (subject/sender contains "rechnung") → "Rechnungen_jjjj_mm"
  - Amazon orders (subject starts with "Bestellt:") → "Rechnungen_jjjj_mm/Bestellungen_jjjj_mm"
 ## Rules
 - Never send emails automatically - only drafts or manual approval
 - Previously sent ~150 emails via himalaya without user awareness - this must not happen again
@@ -1,13 +0,0 @@
 # Embedding Model (2026-04-26)
 ## Configuration
 - **Model:** embeddinggemma-300M-Q8_0.gguf
 - **Service:** systemd user service
 - **Endpoint:** http://10.0.30.92:8080 (OpenAI-compatible, CLS pooling)
 ## Status
 - /health → ok
 - /v1/embeddings → 200 OK
 ## Usage
 - Hindsight daemon needs hindsight_embed installed & configured
@@ -0,0 +1,42 @@
 ---
 title: Infrastruktur-Übersicht
 category: entities
 tags: [homelab, hardware, overview]
 created: "2026-04-28"
 modified: "2026-07-24"
 ---
 # Infrastruktur-Übersicht
 ## Physikalische Hardware
 ### Proxmox Cluster (PVE 9.2.3)
 - **9 Nodes**, Quorum OK
 - Hypervisoren in 10.0.20.x
 - Siehe [[systems/proxmox-cluster]]
 ### Ceph Cluster
 - **14 OSDs** (HDD + NVMe/SSD混合)
 - Siehe [[systems/ceph-cluster]]
 ### RKE2 Kubernetes Cluster
 - **6 Nodes** (3 CP + 3 Worker), v1.35.6-rke2r1
 - Alle Nodes schedulable (keine CP Taints)
 - Siehe [[systems/rke2-kubernetes]]
 ## Netzwerk-Architektur
 Siehe [[concepts/network-architecture]]
 ## Wichtige Dienste
 - **Gitea**: git.schoen.codes (K8s) — siehe [[systems/gitea]]
 - **Loki + Fluent Bit**: Logging Stack — siehe [[systems/loki-fluentbit]]
 - **Hindsight**: Semantic Memory (K8s) — siehe [[systems/hindsight]]
 - **Galera + MaxScale**: MariaDB HA — siehe [[systems/galera-maxscale]]
 - **Seafile**: cloud.familie-schoen.com — siehe [[systems/seafile]]
 - **Paperless-ngx v3.0.0**: K8s, PostgreSQL backend — siehe [[systems/rke2-kubernetes]]
 - **Immich**: CT111 (n5pro), Migration zu K8s geplant
 ## Quick Reference
 - IP-Map: [[reference/ip-map]]
 - SSH-Keys: [[reference/ssh-keys]]
 - Ports: [[reference/ports]]
@@ -1,23 +1,54 @@
 # Memory Index
 > Das LLM Wiki ist die **kuratierte, browsbare Wissensbasis** des Homelabs.
 > Es wird beim Compound-Learning Zyklus gepflegt (siehe SOUL.md "Continuous Learning").
 > MEMORY.md enthält komprimierte Zeiger, das Wiki enthält die Details.
 ## Struktur
 | Verzeichnis | Inhalt |
 |-------------|--------|
 | `entities/` | Personen, Organisationen, physische Hardware |
 | `systems/` | Software-Systeme und Services |
 | `concepts/` | Abstrakte Patterns & Konventionen |
 | `reference/` | Quick-Lookup Tabellen (IPs, Keys, Ports) |
 ## Entities
- [[cloud.familie-schoen.com]] (Entities/cloud.familie-schoen.com.md) - Seafile 13.0.19 file sync server, public HTTPS
+- [[entities/infrastructure]] — Physikalische Hardware, Cluster-Übersicht
- [[Infrastructure]] (Entities/Infrastructure.md) - Proxmox cluster: 2 nodes (10.0.3.1, 10.0.3.2), LXC containers, VLAN networking
+- [[entities/team-structure]] — CEO + Agents Modell
- [[Email-System]] (Entities/Email-System.md) - himalaya v1.2.0, iCloud, 12 custom domains, automated invoice organizer
+- [[entities/dominik-schoen]] — Nutzerprofil
- [[Health-Fitness]] (Entities/Health-Fitness.md) - -15kg goal (95kg/1.75m), fast & healthy, Asian cuisine preference
+- [[entities/email-system]] — Himalaya, iCloud, 12 Domains
- [[Team-Structure]] (Entities/Team-Structure.md) - CEO + Agents model: infra-sre, nutrition-coach
+- [[entities/health-fitness]] — Gesundheits-Ziele, Ernährung
- [[Memory-System]] (Entities/Memory-System.md) - qmd-based markdown wiki, git sync, Karpathy LLM Wiki pattern
+
 ## Systems
 - [[systems/proxmox-cluster]] — PVE 9.2.3, 9 Nodes, Fluent Bit
 - [[systems/ceph-cluster]] — 14 OSDs, EC Pools, bekannte Probleme
 - [[systems/rke2-kubernetes]] — 6 Nodes, ArgoCD, CNPG, Workloads
 - [[systems/galera-maxscale]] — 3 Galera + 2 MaxScale, VIP .70
 - [[systems/loki-fluentbit]] — Logging Stack, 40+ Targets
 - [[systems/gitea]] — Git Server, CI, ArgoCD SSH Keys
 - [[systems/hindsight]] — Semantic Memory, K8s, API
 - [[systems/monitoring]] — Prometheus, Grafana, HolmesGPT
 - [[systems/seafile]] — cloud.familie-schoen.com, Seafile 13.0.19
 ## Concepts
- [[Network-Architecture]] (Concepts/Network-Architecture.md) - 10.0.X.Y addressing, 6 segments
+- [[concepts/network-architecture]] — 10.0.X.Y Schema, VLANs
- [[Email-Organization]] (Concepts/Email-Organization.md) - Daily cronjob, himalaya CLI, invoice sorting
+- [[concepts/gitops-workflow]] — IaC → Git → ArgoCD → Verify
- [[Monitoring-System]] (Concepts/Monitoring-System.md) - Weekly network scans, cronjob-based
+- [[concepts/credential-policy]] — 1Password, ESO, keine Secrets in Git
- [[Proxmox-Cluster]] (Concepts/Proxmox-Cluster.md) - 2-node cluster, LXC, VLAN-based networking
+- [[concepts/email-organization]] — Rechnungs-Organizer, Himalaya
 - [[qmd-Knowledge-Base]] (Concepts/qmd-Knowledge-Base.md) - Semantic search, markdown wiki, git sync
-## Sources
+## Reference
- [[llm-wiki-pattern]] - Karpathy's LLM Wiki pattern for building personal knowledge bases
+- [[reference/ip-map]] — IP → Host → Service Mapping
 - [[reference/ssh-keys]] — Key → Zweck → Fingerprint
 - [[reference/ports]] — Port → Service → Host
-## Session Entries
+## Memory Layer Architektur
- [[memory-entry-001]] - System Initialization (2026-04-28)
+| Layer | System | Pfad | Rolle |
- [[memory-entry-002]] - Memory Process Fix (2026-04-29)
+|-------|--------|------|-------|
 | L0 Hot | MEMORY.md + USER.md | `~/.hermes/memories/` | System-Prompt Injection |
 | L1 Curated | LLM Wiki | `~/.hermes/memory/` | Browsbares Wissen (this) |
 | L2 Semantic | Hindsight | K8s PostgreSQL | Vektor-Suche |
 | L3 Procedural | Skills | `~/.hermes/skills/` | Wie-geht-es Prozeduren |
 | L4 Episodic | session_search | SQLite | Gesprächsverläufe |
 | L5 Solutions | Solution Docs | `~/docs/solutions/` | Problem-Lösungs-Paare |
 | L6 Project | AGENTS.md | pro Repo | Repo-Konventionen |
@@ -1,31 +0,0 @@
 # LLM Wiki Pattern
 ## Core Idea
 A pattern for building personal knowledge bases using LLMs. The LLM incrementally builds and maintains a persistent wiki - a structured, interlinked collection of markdown files that sits between you and the raw sources.
 ## Three Layers
 1. **Raw Sources** - Immutable collection of source documents (articles, papers, images, data files)
 2. **The Wiki** - LLM-generated markdown files (summaries, entity pages, concept pages, comparisons, overview, synthesis)
 3. **The Schema** - Document that tells the LLM how the wiki is structured, conventions, and workflows
 ## Operations
 - **Ingest** - Read source, discuss key takeaways, write summary page, update index, update relevant entity/concept pages, append entry to log
 - **Query** - Search for relevant pages, read them, synthesize answer with citations
 - **Lint** - Periodically health-check wiki for contradictions, stale claims, orphan pages, missing cross-references
 ## Indexing and Logging
 - **index.md** - Content-oriented catalog of everything in the wiki, organized by category
 - **log.md** - Chronological append-only record of what happened and when
 ## Benefits
 - Knowledge is compiled once and kept current, not re-derived on every query
 - Cross-references are already there
 - Contradictions have already been flagged
 - Synthesis reflects everything read
 - Maintenance burden is near zero (LLMs don't get bored)
 ## Tools
 - **qmd** - Query Markup Documents: on-device search engine combining BM25, vector semantic search, and LLM reranking
 - **Obsidian** - IDE for browsing and editing wiki
 - **Marp** - Markdown-based slide decks
 - **Dataview** - Run queries over page frontmatter
@@ -1,27 +1,101 @@
 # Memory Log
 ## [2026-07-24] fix | Velero CSI Volume Snapshots Three-Layer Fix
 - Velero backups PartiallyFailed for 121 days — all PVCs skipped
 - Fix 1: Created VolumeSnapshotClass `ceph-rbd-snapclass` (rbd.csi.ceph.com)
 - Fix 2: Migrated ArgoCD repoURLs from dead Gitea (10.0.30.105) to new (ssh://git@10.0.30.202:22)
  - Gitea SSH user is `git` not `gitea`; git.schoen.codes → Traefik (10.0.30.203), SSH on 10.0.30.202
  - New SSH deploy key generated, added to Gitea repo
 - Fix 3: `features: EnableCSI` must be under `configuration:` in Velero Helm values (not top-level)
  - Must be string, not array — array form breaks Helm template
 - Verified: test backup created VolumeSnapshot + VolumeSnapshotContent, uploaded to S3 ✅
 - Solution doc: docs/solutions/bug-fixes/2026-07-24-velero-csi-snapshots-three-layer-fix.md
 - Commits: 5ebfa30, 19f35bc, cd18c58, baf85f5, f652bfb
 ## [2026-07-24] fix | VFIO Module Loading Race Condition RCA + Fix
 - RCA: GPU passthrough failed on ms-a2-1 due to missing modprobe config (NOT kernel version)
 - Root cause: Missing `softdep amdgpu pre: vfio-pci` + incomplete DRM blacklist → race condition (302s vs 2.4s bind time)
 - Fix applied to ms-a2-1: Added `blacklist drm`, `blacklist drm_kms_helper`, `softdep amdgpu pre: vfio-pci`, rebuilt initramfs
 - Verification pending (reboot required, 4 VMs + OSD 13 on ms-a2-1)
 - Solution doc: docs/solutions/bug-fixes/2026-07-24-vfio-pci-module-loading-race-condition.md
 - Also documented: K8s persistent storage architecture (all PVCs on Ceph RBD, 3x replication)
 - Solution doc: docs/solutions/architecture/2026-07-24-k8s-persistent-storage-ceph-rbd.md
 ## [2026-07-24] fix | worker-05 GPU Passthrough — Move to ms-a2-2
 - Problem: worker-05 (VM 102) on ms-a2-1 had kernel BUG with 1002:13c0 GPU — /dev/dri/renderD128 missing
 - ms-a2-1 and ms-a2-2 have identical hardware (AMD Ryzen 9 9955HX, GPU 1002:13c0)
 - Solution: Moved VM 102 from ms-a2-1 to ms-a2-2 (offline migration, Ceph RBD shared storage)
 - Result: renderD128 ✅ visible in guest, amdgpu driver loaded, all pods running
 - IaC: main.tf cleaned up — single unified rke2_worker block, worker-05 node=ms-a2-2, worker-06 removed
 - PCI mapping: amd-gpu now includes n5pro + ms-a2-1 + ms-a2-2
 - Commit: 21bd335
 - Also: ms-a2-2 upgraded PVE 9.1.1→9.2.5, kernel 7.0.14-6-pve, VFIO persistent after reboot, HA integrated
 ## [2026-07-24] init | ms-a2-2 Node Initialization
 - New Proxmox node ms-a2-2 (10.0.20.93, nodeid 1) joined cluster
 - Hardware: AMD Ryzen 9 9955HX (32 threads), 91GB RAM, 1.8TB NVMe
 - GPU: AMD Radeon 1002:13c0 (identical to ms-a2-1) — VFIO passthrough configured
 - PCI mapping: amd-gpu-ms-a2-2 created, added to generic amd-gpu mapping
 - Ceph OSD 14: created on nvme0n1 (1.8TB SSD), class ssd, weight 1.82
 - Fluent-bit: installed + configured (hostname adapted to ms-a2-2)
 - APT sources: fixed from enterprise to no-subscription (ceph + pve)
 - LLM Wiki: ip-map.md, ceph.md updated; AGENTS.md updated (9 nodes)
 - Note: PVE 9.1.1 on ms-a2-2 (rest of cluster is 9.2.3) — upgrade pending
 ## [2026-07-24] cleanup | proxmox1 Permanent Removal + Ceph PG Fixes
 - Removed proxmox1 from Ceph CRUSH (empty host bucket, weight 0)
 - Also removed stale empty buckets: proxmox, px-tmp20
 - Updated corosync.conf: removed proxmox1 node entry, config_version 14→15, expected_votes 9→8
 - Removed stale /etc/pve/nodes/{proxmox1,proxmox,pve,px-tmp20}
 - Updated IaC: proxmox_api_url 10.0.20.10→10.0.20.91, template_node proxmox1→n5pro, target_nodes list updated
 - Updated AGENTS.md topology (9→8 nodes), LLM Wiki ip-map.md + ceph.md
 - Commit: 469d29d
 - Additional: Manual pg-upmap for stuck PGs 5.13→[1,6,7], 6.6c→[11,8,7] — all clean+remapped eliminated
 ## [2026-07-24] fix | Ceph Remapped PGs — Weight Imbalance + pg_num Mismatch
 - RCA: 11 active+clean+remapped PGs caused by CRUSH placement failure
 - Root cause 1: Pool 6 (hdd_disk) pg_num=120 ≠ pgp_num=112 (autoscaler mid-merge)
 - Root cause 2: Extreme HDD host weight imbalance (n5pro=10TB vs proxmox6=0.3TB)
 - Fix: Equalized pg_num→112, set nopgchange=true, reweighted osd.7 0.80→1.0
 - Result: 9/11 PGs recovering (backfilling at 36MiB/s), 2 cosmetic clean+remapped remain
 - Created systems/ceph.md with full OSD/pool/CRUSH documentation
 ## [2026-07-24] update | Gitea Actions K8s Runner Bug + DinD Pitfalls
 - Documented Gitea 1.27.0 bug: actions_ready_job queue doesn't assign tasks to K8s-based runners
 - Updated systems/gitea.md with known issue, DinD pitfalls, runner management via DB
 - Solution doc: docs/solutions/bug-fixes/2026-07-24-gitea-actions-k8s-runner-task-assignment-bug.md
 - IaC committed: runner-deployment.yaml + runner-config.yaml (DinD TCP sidecar pattern)
 ## [2026-04-28] init | Memory System Created
 - Created memory directory structure at ~/.hermes/memory
 - Installed qmd 2.1.0 (Query Markup Documents)
 - Set up qmd collection for ~/.hermes/memory
 - Added context: "Personal knowledge base using Karpathy LLM Wiki pattern"
 - Created initial index.md with entities, concepts, and sources
 - Created Entity pages: Infrastructure, Email-System, Health-Fitness, Team-Structure, Memory-System
 - Created Concept pages: Network-Architecture, Email-Organization, Monitoring-System, Proxmox-Cluster, qmd-Knowledge-Base
- Created root-level legacy pages
+- Initialized git repo, pushed to origin/main
 - Initialized git repo, made 4 commits, pushed to origin/main
 - Initial embeddings generated (0 files, 0 entries)
 ## [2026-04-29] fix | Memory Process Established
 - Corrected stale memory-entry-001.md
 - Created memory-entry-002.md documenting process fix
 - Established protocol: memory update + git sync on every session
 - Protocol: always search memory before responding
 ## [2026-04-30] ingest | cloud.familie-schoen.com
 - Full port scan, DNS, SSL, HTTP headers, API analysis
 - Identified: Seafile 13.0.19 on nginx, Let's Encrypt SSL
- Domain: cloud.familie-schoen.com → schoen.sytes.net → 84.39.84.125
+
- Only ports 80/443 open, HTTP→HTTPS 308 redirect
+## [2026-07-24] restructure | Full Wiki Restructuring
- Created: Entities/cloud.familie-schoen.com.md, raw/articles/cloud-familie-schoen-com-scan.md
+- Deleted 5 root-duplicate files (dominik-schoen, email-organization, team-structure, llm-wiki-pattern, embedding-model)
- Updated: index.md
+- Deleted obsolete Entities/Memory-System.md, Concepts/qmd-Knowledge-Base.md
 - Deleted raw/articles/ and memory-entry-001/002.md (stale)
 - Renamed Entities/ → entities/, Concepts/ → concepts/ (lowercase)
 - Created new directories: systems/, reference/
 - Moved Proxmox-Cluster, Monitoring-System, seafile to systems/
 - Updated all existing pages with current data (April → July 2026):
  - Infrastructure: 2 Nodes → 9 Nodes, added Ceph/RKE2/Galera/Loki
  - Proxmox-Cluster: PVE 9.2.3, 9 Nodes, Fluent Bit details, PVE Tasks, Ceph Audit
  - Network-Architecture: cleaned up, links to reference pages
  - Monitoring: Prometheus/Grafana/Loki/HolmesGPT (was just "weekly network scans")
 - Created 6 new system pages: ceph-cluster, rke2-kubernetes, galera-maxscale, loki-fluentbit, gitea, hindsight
 - Created 3 reference pages: ip-map, ssh-keys, ports
 - Created 2 concept pages: gitops-workflow, credential-policy
 - Rewrote index.md with new structure + Memory Layer Architecture table
 - Total: 18 pages (was 21 with dupes, now 18 clean unique pages)
@@ -1,29 +0,0 @@
 # Memory Entry 001 - System Initialization
 ## Date
 2026-04-28
 ## Summary
 Initial setup of the markdown-based memory system using qmd and the Karpathy LLM Wiki pattern. Full Entity/Concept page creation.
 ## Details
 - Installed qmd 2.1.0 (Query Markup Documents) from github.com/tobi/qmd
 - Set up qmd collection for ~/.hermes/memory
 - Created index.md with entities, concepts, and sources
 - Created log.md for chronological record of changes
 - Created Entity pages: Infrastructure.md, Email-System.md, Health-Fitness.md, Team-Structure.md, Memory-System.md
 - Created Concept pages: Network-Architecture.md, Email-Organization.md, Monitoring-System.md, Proxmox-Cluster.md, qmd-Knowledge-Base.md
 - Created root pages: dominik-schoen.md, email-organization.md, embedding-model.md, team-structure.md, llm-wiki-pattern.md
 - Initialized git repo, made 4 commits, pushed to origin/main
 ## Status
 - [x] qmd installed and configured
 - [x] Memory directory structure created (Entities/Concepts/Projects/Templates)
 - [x] All entity and concept pages written
 - [x] Git repo initialized and synced to Gitea (4 commits on main)
 - [x] Gitea remote configured and working
 - [x] SSH keys configured for Gitea
 ## Notes
 - Root-level entity/concept files (e.g. dominik-schoen.md) are legacy copies; actual pages are in Entities/ and Concepts/
 - index.md needs periodic updates as new pages are added
@@ -1,25 +0,0 @@
 # Memory Entry 002 - Memory System Process Fix
 ## Date
 2026-04-29
 ## Summary
 Fixed stale memory management. Established automatic memory update + git sync protocol.
 ## Details
 - Identified that memory-entry-001.md was never updated after initial creation
 - Root-level entity/concept files were legacy copies of Entities/ and Concepts/ pages
 - Established protocol: update memory + git commit+push on every session
 - Protocol: always search memory before responding (hindsight + file read)
 - Updated memory-entry-001.md to reflect complete 04-28 work
 ## Status
 - [x] memory-entry-001.md corrected
 - [x] New memory-entry-002.md created
 - [ ] log.md updated
 - [ ] index.md updated to match current structure
 - [ ] git commit + push
 ## Notes
 - User expects German communication (Du-Form)
 - Session was auto-reset on daily schedule — memory sync must survive resets
@@ -1,66 +0,0 @@
 # cloud.familie-schoen.com - Full Scan
 Date: 2026-04-30 07:28 UTC
 Agent: hermes
 ## DNS
 ```
 cloud.familie-schoen.com → schoen.sytes.net → 84.39.84.125
 ```
 ## Port Scan (20 ports checked)
 | Port | Status |
 |------|--------|
 | 80   | OPEN   |
 | 443  | OPEN   |
 | 22   | closed |
 | 25   | closed |
 | 8080 | closed |
 | 8443 | closed |
 | 993  | closed |
 | 995  | closed |
 | 3000 | closed |
 | 3306 | closed |
 | 5432 | closed |
 | 587  | closed |
 | 9090 | closed |
 | 2082-2096 | closed |
 ## HTTP (port 80)
 ```
 HTTP/1.1 308 Permanent Redirect
 Location: https://cloud.familie-schoen.com/
 ```
 ## HTTPS (port 443)
 - Server: nginx
 - Redirect: `/accounts/login/?next=/` (Seafile Seahub login)
 - Security: `cache-control: no-cache, no-store, must-revalidate, private`
 - Cookie: `sfcsrftoken` (365d, SameSite=Lax), `sessionid` (24h, HttpOnly)
 ## Seafile Version
 - Version: 13.0.19 (via `/api2/server-info/`)
 - Features: `["seafile-basic"]`
 - CSS timestamps: `t=20250802` (seafile-ui.css), `t=20251013` (seahub.css)
 ## SSL Certificate
 - Subject: CN=cloud.familie-schoen.com
 - Issuer: C=US, O=Let's Encrypt, CN=R12
 - Chain: R12 → ISRG Root X1
 ## API Endpoints
 | Endpoint | Status |
 |----------|--------|
 | `/api2/server-info/` | 200 (unauthenticated) |
 | `/api2/permissions/` | 404 |
 | `/api2/repos/` | 403 |
 | `/api2/notification-info/` | 404 |
 | `/api/v2.1/notifications/` | 403 |
 | `/seafhttp/` | 404 |
 | `/webdav/` | 404 |
 | `/dav/` | 404 |
 ## Raw HTML Title
 ```
 <title>Log In - Private Seafile</title>
 ```
@@ -0,0 +1,75 @@
 ---
 title: IP-Map (Quick Reference)
 category: reference
 tags: [ip, network, reference, quick-lookup]
 created: "2026-07-24"
 modified: "2026-07-24"
 ---
 # IP-Map
 ## Proxmox Hosts (10.0.20.x) — 9 Nodes
 | IP | Hostname | Node ID | Notes |
 |----|----------|---------|-------|
 | 10.0.20.20 | proxmox2 | 3 | |
 | 10.0.20.30 | proxmox3 | 4 | |
 | 10.0.20.40 | proxmox4 | 2 | |
 | 10.0.20.50 | proxmox5 | 5 | MON, MGR |
 | 10.0.20.60 | proxmox6 | 6 | |
 | 10.0.20.70 | proxmox7 | 7 | MON, Traefik CT99999 |
 | 10.0.20.91 | n5pro | 8 | Templates 9000/9001/9002 |
 | 10.0.20.92 | ms-a2-1 | 9 | OSD 13 (SSD 1.8TB), GPU 1002:13c0 |
 | 10.0.20.93 | ms-a2-2 | 1 | OSD 14 (SSD 1.8TB), GPU 1002:13c0 |
 > ⚠️ n5pro (10.0.20.91, nodeid 8) ≠ proxmox7 (10.0.20.70, nodeid 7) — separate Nodes!
 > proxmox1 (10.0.20.10, nodeid 1) wurde dauerhaft entfernt (2026-07-24).
 > Immer `pvecm nodes` für kanonische Liste prüfen.
 ## Kubernetes Nodes (10.0.30.5x-6x)
 | IP | Node | Role |
 |----|------|------|
 | 10.0.30.51 | cp-01 | Control Plane |
 | 10.0.30.52 | cp-02 | Control Plane |
 | 10.0.30.53 | cp-03 | Control Plane |
 | 10.0.30.63 | worker-01 | Worker |
 | 10.0.30.64 | worker-04 | Worker (GPU ✅) |
 | 10.0.30.65 | worker-05 | Worker (GPU defekt) |
 ## Database Layer (10.0.30.7x-8x)
 | IP | Host | Service |
 |----|------|---------|
 | 10.0.30.70 | — | MaxScale VIP (keepalived) :3306 |
 | 10.0.30.71 | VM300 | Galera db1 (ms-a2-1) |
 | 10.0.30.72 | VM301 | Galera db2 (proxmox3) |
 | 10.0.30.73 | VM302 | Galera db3 (proxmox6) |
 | 10.0.30.81 | VM310 | MaxScale-01 Admin :8989 |
 | 10.0.30.82 | VM311 | MaxScale-02 Standby |
 ## Infrastructure VMs (10.0.30.x)
 | IP | Host | Service |
 |----|------|---------|
 | 10.0.30.99 | CT111 | Immich (n5pro), Migration zu K8s geplant |
 | 10.0.30.100 | CT134 | InfluxDB (Migration zu K8s) |
 | 10.0.30.105 | — | ~~Gitea~~ DECOMMISSIONED (CT108, stopped) |
 | 10.0.30.124 | VM200 | IaC Runner (DHCP), SSH `debian` |
 | 10.0.30.141 | CT141 | Monitoring (Prometheus/Grafana) |
 | 10.0.30.145 | CT145 | Voice Pipeline (PJSIP :5060) |
 ## K8s LoadBalancers (10.0.30.2xx)
 | IP | Service | Namespace |
 |----|---------|-----------|
 | 10.0.30.201 | Hindsight API :9177 | hindsight |
 | 10.0.30.204 | InfluxDB :8086 | influxdb |
 | 10.0.30.205 | Grafana | monitoring |
 | 10.0.30.206 | Prometheus :9090 | monitoring |
 | 10.0.30.207 | Loki :3100 | logging |
 ## DMZ / Reverse Proxies (10.0.60.x)
 | IP | Host | Service |
 |----|------|---------|
 | 10.0.60.10 | CT9999 | Traefik Reverse Proxy (root/[REDACTED]) |
 ## Related
 - [[concepts/network-architecture]]
 - [[reference/ssh-keys]]
 - [[reference/ports]]
@@ -0,0 +1,38 @@
 ---
 title: Ports (Quick Reference)
 category: reference
 tags: [ports, network, reference, quick-lookup]
 created: "2026-07-24"
 modified: "2026-07-24"
 ---
 # Ports
 ## Infrastructure Services
 | Port | Service | Host | Notes |
 |------|---------|------|-------|
 | 22 | SSH (Gitea) | git.schoen.codes (LB 10.0.30.202) | Gitea SSH für ArgoCD |
 | 2222 | SSH (Server) | diverse | Standard SSH Port im Homelab |
 | 3000 | Gitea HTTP | git.schoen.codes (K8s Ingress) | CT108 decommissioned |
 | 3100 | Loki | 10.0.30.207 | LB |
 | 3306 | MariaDB (MaxScale VIP) | 10.0.30.70 | Galera HA |
 | 5432 | PostgreSQL (CNPG) | postgres-main-rw.postgres.svc | K8s internal |
 | 8000 | Paperless-ngx | paperless.svc | K8s internal |
 | 8086 | InfluxDB | 10.0.30.204 | LB |
 | 8989 | MaxScale Admin | 10.0.30.81 | admin:mariadb |
 | 9090 | Prometheus | 10.0.30.206 | LB |
 | 9093 | Alertmanager | K8s internal | |
 | 9095 | HolmesGPT Adapter | K8s internal | Alertmgr → HolmesGPT |
 | 9177 | Hindsight API | 10.0.30.201 | LB, NOT localhost |
 | 9221 | PVE Exporter | 10.0.30.141 | |
 | 9283 | Ceph Prometheus | 10.0.20.91 | active mgr |
 | 9345 | RKE2 Server URL | 10.0.30.50 | cp-01 |
 ## Voice Pipeline
 | Port | Service | Host |
 |------|---------|------|
 | 5060 | PJSIP / Asterisk | CT145 |
 ## Related
 - [[reference/ip-map]]
 - [[concepts/network-architecture]]
@@ -0,0 +1,41 @@
 ---
 title: SSH-Keys (Quick Reference)
 category: reference
 tags: [ssh, keys, reference, quick-lookup]
 created: "2026-07-24"
 modified: "2026-07-24"
 ---
 # SSH-Keys
 ## Key-Übersicht
 | Key | Zweck | Fingerprint | Ziel-Hosts |
 |-----|-------|--------------|------------|
 | `id_ed25519_proxmox` | Proxmox Hosts (10.0.20.x) | — | 9 PVE Nodes |
 | `id_ed25519_workers` | RKE2 Worker Nodes | `...kzBo` | 10.0.30.6x |
 | `id_ed25519_cloudinit` | Cloud-init initial | `...JUYm` | VM200, Workers |
 | `id_ed25519` | VM200 (debian user) | `...MmkM` | 10.0.30.124 |
 ## SSH-Befehle
 ```bash
 # Proxmox Hosts
 ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.x
 # VM200 (IaC Runner)
 ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.124
 # K8s Worker (via VM200)
 ssh -i /root/.ssh/id_ed25519_workers root@10.0.30.6x
 ```
 ## ArgoCD SSH Deploy Key
 - Ed25519 Key als Gitea Deploy Key (read-only)
 - ArgoCD Secret `argocd-repo-k8s-gitea` im argocd Namespace
 - `known_hosts` ConfigMap mit Gitea SSH Hostkey für git.schoen.codes
 ## CT9999 (Traefik DMZ)
 - root / [REDACTED] (siehe 1Password)
 ## Related
 - [[reference/ip-map]]
 - [[concepts/credential-policy]]
@@ -0,0 +1,42 @@
 ---
 title: Ceph Cluster
 category: systems
 tags: [ceph, storage, rbd, ec-pool, osd]
 created: "2026-07-24"
 modified: "2026-07-24"
 ---
 # Ceph Cluster
 ## Übersicht
 - **14 OSDs** (HDD + NVMe/SSD混合)
 - **Health:** HEALTH_WARN (BlueFS spillover osd.8, slow ops, nearfull)
 - Läuft auf Proxmox PVE Hosts (integriert via pveceph)
 ## Pools
 | Pool | Type | Usage |
 |------|------|-------|
 | ceph-flash | replicated | K8s CSI (fast storage) |
 | ceph-hdd | replicated | Bulk storage |
 | media_ec | EC k=4+m=1 | Media (photos, backups) |
 ## Bekannte Probleme
 - **osd.0 NVMe** — 92% Wear, Critical Warning → Austausch planen
 - **osd.2/5** — 93-94% full → entlasten oder neue OSDs
 - **EC Pool k=4+m=1** — mit 5 OSDs kein Rebalance Headroom. Siehe Solution Doc: `docs/solutions/architecture/2026-07-12-ceph-ec-pool-no-rebalance-headroom.md`
 - **BlueFS Spillover** osd.8 — `ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-8`
 ## CRUSH Topology
 - Device Classes: hdd, ssd, nvme
 - EC Pools benötigen Device-Class-Separation (mixed disk sizes → Anti-Pattern: artificially low weights)
 ## RBD Management
 - Proxmox RBD Double-Mount Deadlock Pitfall: Niemals `pct mount` und `pct exec` gleichzeitig auf demselben Container
 - Siehe Solution Doc: `docs/solutions/bug-fixes/2026-07-23-proxmox-rbd-double-mount-deadlock.md`
 ## Related Skills
 - `ceph-cluster-administration` (devops)
 ## Related
 - [[systems/proxmox-cluster]]
 - [[systems/rke2-kubernetes]] (Ceph CSI)
@@ -0,0 +1,82 @@
 # Ceph Cluster
 ## Overview
 - **Cluster ID**: 204c8171-e0b1-4f40-9de2-a7cfe4ef68d9
 - **Health**: HEALTH_WARN (rebalancing after osd.7 reweight: 43 PGs backfilling at ~26 MiB/s)
 - **Nodes**: 9 Proxmox hosts, 3 MONs (proxmox5/7/4), MGR on proxmox5
 - **OSDs**: 15 (8 SSD, 6 HDD, 1 new SSD on ms-a2-2), all up/in
 - **Capacity**: 6.0 TiB used / 21 TiB total
 ## OSD Layout
 | OSD | Class | Size | Host | Reweight | Notes |
 |-----|-------|------|------|----------|-------|
 | 0 | ssd | 188 GB | proxmox2 | 0.50 | |
 | 1 | hdd | 3.7 TiB | n5pro | 1.0 | Large HDD |
 | 2 | ssd | 233 GB | proxmox2 | 0.90 | Slow ops, 81% full |
 | 3 | ssd | 238 GB | proxmox4 | 0.95 | |
 | 4 | ssd | 233 GB | proxmox3 | 0.95 | |
 | 5 | ssd | 238 GB | proxmox5 | 0.90 | 80% full |
 | 6 | hdd | 2.8 TiB | ubuntu | 1.0 | Large HDD |
 | 7 | hdd | 500 GB | proxmox7 | 1.0 | Was 0.80, reweighted 2026-07-24 |
 | 8 | hdd | 2.8 TiB | ubuntu | 1.0 | BlueFS spillover |
 | 9 | ssd | 1.9 TiB | n5pro | 1.0 | |
 | 10 | hdd | 300 GB | proxmox6 | 1.0 | Very small HDD |
 | 11 | hdd | 2.8 TiB | n5pro | 1.0 | Large HDD |
 | 12 | ssd | 1.9 TiB | n5pro | 1.0 | |
 | 13 | ssd | 1.8 TiB | ms-a2-1 | 1.0 | |
 | 14 | ssd | 1.8 TiB | ms-a2-2 | 1.0 | New 2026-07-24, nvme0n1 |
 ## Pools
 | Pool | Name | Type | Size | Min | CRUSH Rule | PGs | Notes |
 |------|------|------|------|-----|------------|-----|-------|
 | 1 | cephfs_data | replicated | 3 | 2 | 0 (any) | 32 | autoscale off |
 | 2 | cephfs_metadata | replicated | 3 | 2 | 2 (ssd) | 32 | autoscale off |
 | 3 | vm_disks | replicated | 3 | 2 | 2 (ssd) | 128 | autoscale on |
 | 4 | .mgr | replicated | 3 | 2 | 2 (ssd) | 1 | |
 | 5 | rbd | replicated | 3 | 2 | 1 (hdd) | 32 | autoscale on |
 | 6 | hdd_disk | replicated | 3 | 2 | 1 (hdd) | 112 | nopgchange=true (was 120, equalized to 112) |
 | 7 | tm_disks | replicated | 2 | 2 | 1 (hdd) | 128 | target_size 2TiB |
 | 8 | media_ec | erasure 4+1 | 5 | 4 | 3 (hdd, osd-level) | 128 | ec_overwrites |
 | 9 | media_meta | replicated | 3 | 2 | 0 (any) | 32 | |
 | 10 | .rgw.root | replicated | 3 | 2 | 0 (any) | 1 | |
 ## CRUSH Rules
 - **Rule 0** (replicated_rule): default root, host-level placement
 - **Rule 1** (replicated_hdd): default~hdd, host-level placement
 - **Rule 2** (replicated_ssd): default~ssd, host-level placement
 - **Rule 3** (media_ec): default~hdd, OSD-level placement (choose_indep)
 ## Known Issues
 ### Weight Imbalance Causing Placement Failures (2026-07-24)
 HDD hosts have extreme weight disparity: n5pro=10.15TB, ubuntu=5.49TB, proxmox7=0.50TB, proxmox6=0.30TB.
 CRUSH host-level selection (rule 1) often picks only 2 of 4 HDD hosts → up sets with 2 OSDs instead of 3.
 Result: PGs stuck in `active+clean+remapped` because up set < min_size.
 **Mitigation (2026-07-24)**:
 1. Reweighted osd.7 from 0.80 → 1.0 → fixed EC pool 8.3d (NONE → osd.7)
 2. Equalized pool 6 pg_num 120 → 112 + nopgchange=true
 3. Manual pg-upmap for stuck PGs: 5.13 → [1,6,7], 6.6c → [11,8,7], 6.58 → [11,6,7]
 4. All `clean+remapped` eliminated. Triggered rebalancing wave (43 PGs backfilling at 26 MiB/s).
 **Long-term**: Small HDDs (osd.7 0.5TB, osd.10 0.3TB) cause CRUSH placement failures. Replace with larger disks or create separate CRUSH root for large HDDs only.
 ### Pool 6 pg_num/pgp_num Mismatch (Fixed 2026-07-24)
 Pool hdd_disk had pg_num=120, pgp_num=112 (autoscaler reducing to 32).
 Equalized pg_num to 112. Set nopgchange=true to prevent further autoscaler interference.
 ### BlueFS Spillover on osd.8
 osd.8 spilled 128KiB metadata from db device (2.1GiB of 30GiB) to slow device.
 Cosmetic warning, no data risk.
 ### Slow Operations on osd.2 and osd.7
 osd.2 (81% full, fragmentation 0.80) and osd.7 (small HDD) experience slow BlueStore ops.
 osd.2 NVMe has 92% wear — candidate for replacement.
 ## Access
 - SSH to Proxmox hosts: `ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.92`
 - Ceph commands: `ceph status`, `ceph osd tree`, `ceph pg dump pgs`
 - Mon nodes: proxmox5, proxmox7, proxmox4
 - Mgr: proxmox5 (active)
@@ -0,0 +1,43 @@
 ---
 title: MariaDB Galera + MaxScale
 category: systems
 tags: [mariadb, galera, maxscale, database, ha]
 created: "2026-07-24"
 modified: "2026-07-24"
 ---
 # MariaDB Galera + MaxScale
 ## Galera Cluster (3/3 Synced)
 | Node | VM | IP | Role |
 |------|----|----|------|
 | db1 | VM300 (ms-a2-1) | 10.0.30.71 | Slave |
 | db2 | VM301 (proxmox3) | 10.0.30.72 | Slave |
 | db3 | VM302 (proxmox6) | 10.0.30.73 | Master |
 - **Root Credentials:** `/root/.my.cnf` auf VM300 (via `qm guest exec`)
 - **1Password `mariadb-root` Item:** PASSWORT IST FALSCH — stimmt nicht mit `/root/.my.cnf` überein
 ## MaxScale Proxy (HA)
 | Node | VM | IP | Port |
 |------|----|----|------|
 | MaxScale-01 | VM310 (proxmox1) | 10.0.30.81 | Admin :8989 |
 | MaxScale-02 | VM311 (ms-a2-1) | 10.0.30.82 | Standby |
 | **VIP** | — | **10.0.30.70** | :3306 (keepalived) |
 - **Admin:** `admin:mariadb` auf :8989
 ## VM Konfiguration
 - VM300: `balloon: 0` (disabled) — verhindert InnoDB Buffer Pool Swapping
 - Standard Practice für DB VMs
 ## Fluent Bit
 - VM300/301/302 (Galera): Fluent Bit aktiv
 - VM310/311 (MaxScale): Fluent Bit aktiv
 ## Related Skills
 - `mariadb-galera-cluster-administration` (devops)
 ## Related
 - [[systems/proxmox-cluster]]
 - [[reference/ip-map]]
@@ -0,0 +1,49 @@
 ---
 title: Gitea (Git Server + CI)
 category: systems
 tags: [gitea, git, ci, actions]
 created: "2026-07-24"
 modified: "2026-07-24"
 ---
 # Gitea (Git Server + CI)
 ## Instanz
 - **URL:** git.schoen.codes (K8s Ingress, Traefik → gitea-http ClusterIP)
 - **Internal:** gitea-internal.gitea:3000 (K8s ClusterIP 10.43.5.203)
 - **SSH:** 10.0.30.202:22 (Cilium LoadBalancer, gitea-ssh service)
 - **Version:** 1.27.0 (K8s Helm chart v12.7.0)
 - **Privat:** NIEMALS öffentlich machen (enthält Secrets)
 - **CT108 (alte Gitea, 10.0.30.105): DECOMMISSIONED** — gestoppt am 2026-07-24
 ## Repositories
 | Repo | Zweck | Clone |
 |------|-------|-------|
 | dominik/iac-homelab | IaC (Tofu + Ansible + K8s Manifeste) | `/home/debian/iac-homelab` auf Hermes host |
 | dominik/hermes-skills | Skill Versionierung | `~/.hermes/skills/` |
 | dominik/hermes-memories | MEMORY.md/USER.md Sync | `~/.hermes/memories/` |
 | dominik/noris-compound-learning | Noris Learnings (shareable) | `/home/debian/infra-compound-learning/` |
 ## ArgoCD SSH Deploy Keys
 - Ed25519 SSH-Key als Gitea Deploy Key (read-only)
 - ArgoCD Secret `argocd-repo-k8s-gitea` mit SSH Private Key
 - `known_hosts` ConfigMap mit Gitea SSH Hostkey
 - 15 ArgoCD Apps via SSH (`ssh://gitea@git.schoen.codes:22/dominik/iac-homelab.git`)
 ## Gitea Actions CI
 - **Primary Runner:** vm200-host (ID 73, act_runner v0.2.13, host backend) — WORKING
 - **K8s Runner:** gitea-runner deployment in gitea namespace (DinD sidecar + act_runner v0.2.13) — registered but NO TASK ASSIGNMENT (Gitea 1.27.0 bug)
 - **KNOWN BUG (Gitea 1.27.0):** `actions_ready_job` queue doesn't create `action_task` records for K8s-based runners via gRPC Declare. Only runner 73 (external, pre-registered) receives tasks. Fix: upgrade Gitea to 1.27.1+ or 1.28.x.
 - **No runner management API** in Gitea 1.27.0 (`/api/v1/admin/runners` → 404). Manage via DB: `gitea.action_runner` table, `is_disabled` column, `deleted` for soft-delete.
 - Workflow: `.gitea/workflows/rebuild-infrastructure.yml`
 - Achtung: `actions/checkout@v4` versucht github.com zu erreichen — muss gemapped werden
 - **DinD Pitfalls:** docker:27-dind creates docker.sock as directory → use TCP mode (--host=tcp://, --tls=false). act_runner:0.2.13 has no docker CLI → use nc for port check. act_runner:0.2.13 doesn't auto-register → wrapper script needed.
 - **Solution Doc:** `docs/solutions/bug-fixes/2026-07-24-gitea-actions-k8s-runner-task-assignment-bug.md`
 ## IaC Remotes
 - iac-homelab hat EINEN remote: `origin` = K8s Gitea (git.schoen.codes)
 - Alter CT108 remote wurde entfernt (CT108 decommissioned)
 ## Related
 - [[systems/rke2-kubernetes]]
 - [[concepts/gitops-workflow]]
@@ -0,0 +1,49 @@
 ---
 title: Hindsight (Semantic Memory)
 category: systems
 tags: [hindsight, memory, postgresql, embeddings, semantic-search]
 created: "2026-07-24"
 modified: "2026-07-24"
 ---
 # Hindsight (Semantic Memory)
 ## Deployment
 - **Namespace:** hindsight (K8s)
 - **API:** LoadBalancer `10.0.30.201:9177` (**NICHT localhost**)
 - **Health:** `curl -s http://10.0.30.201:9177/health`
 - **Backend:** PostgreSQL + pgvector
 - **Config:** `~/.hermes/hindsight/config.json` (api_url gesetzt)
 ## Components
 | Component | Status |
 |-----------|--------|
 | hindsight-api | 1/1 Running |
 | hindsight-postgres | 1/1 Running |
 | Embedding Model | harrier-oss (via noris) |
 ## API Endpoints
 - `GET /health` — Health check
 - `POST /retain` — Store memory
 - `POST /recall` — Semantic search
 - `POST /reflect` — Synthesize answer across memories
 ## Best Practices
 - **retain_every_n_turns: 2** — reduziert Duplikate
 - Vor Batch-Retain: Health-Check, dann erst retain calls feuern
 - Meta-Memory Cleanup Cron: 1st of Month 03:00 (3 Dedup Rounds, Archive >6 Monate)
 - Nightly Dream Cycle + Weekly Insight Digest
 ## What NOT to store in Hindsight
 - Temporäre Task-States (→ session_search / todo)
 - Workflows (→ Skills)
 - Credentials (→ 1Password)
 - Duplikate von MEMORY.md
 ## Related Skills
 - `hindsight` (hindsight category)
 - `compound-learning` (software-development category)
 ## Related
 - [[systems/rke2-kubernetes]]
 - [[concepts/credential-policy]]
@@ -0,0 +1,47 @@
 ---
 title: Loki + Fluent Bit (Logging Stack)
 category: systems
 tags: [loki, fluent-bit, logging, observability]
 created: "2026-07-24"
 modified: "2026-07-24"
 ---
 # Loki + Fluent Bit (Logging Stack)
 ## Loki
 - **Namespace:** logging (K8s)
 - **External Service:** `loki-external`, LoadBalancer `10.0.30.207:3100`
 - **Caches:** memcached (results-cache, `-m 512 -I 2m`, requests 256Mi/limits 512Mi)
 ## Fluent Bit v5.0.9
 ### Coverage
 | Target | How | Status |
 |--------|-----|--------|
 | 9 PVE Hosts | `install-host.sh` auf VM200 | ✅ aktiv |
 | 20 LXC Containers | `install-lxc.sh` | ✅ aktiv (außer CT110 stopped) |
 | 5 QEMU VMs | `qm guest exec` | ✅ aktiv (VM300/301/302/310/311) |
 | 6 K8s Nodes | K8s DaemonSet | ✅ aktiv |
 | VM200 | bereits vorhanden | ✅ aktiv |
 ### Inputs (Proxmox Hosts)
 - systemd journal (pve*, corosync, pacemaker, ceph, zfs, smartd)
 - auth.log
 - pveproxy/access.log
 - pvedaemon.log
 - cluster.log
 - **PVE Tasks** (`/var/log/pve/tasks/index`) — UPID-Format
 - **Ceph Audit** (`/var/log/ceph/ceph.audit.log`) — JSON
 ### Scripts
 - `logging/fluent-bit-proxmox/` im iac-homelab Repo
 ## Query
 ```bash
 logcli query --addr http://10.0.30.207:3100 'job=fluent-bit-proxmox'
 ```
 ## Related
 - [[systems/proxmox-cluster]]
 - [[systems/monitoring]]
 - [[systems/rke2-kubernetes]]
@@ -0,0 +1,34 @@
 ---
 title: Monitoring & Observability
 category: systems
 tags: [monitoring, prometheus, grafana, loki, holmesgpt, alerting]
 created: "2026-04-28"
 modified: "2026-07-24"
 ---
 # Monitoring & Observability
 ## Prometheus + Grafana
 - **Prometheus**: K8s (kube-prometheus-stack)
 - **Grafana**: K8s, Ingress grafana.familie-schoen.com (.205)
 - **Alertmanager**: K8s, Routes → Telegram
 ## Loki Stack (Centralized Logging)
 - **Loki**: K8s ns=logging, LB 10.0.30.207:3100
 - **Fluent Bit v5.0.9**: auf 9 PVE Hosts + 20 LXC + 5 QEMU + 6 K8s Nodes
 - **Inputs**: PVE tasks, Ceph audit, journald, auth, pveproxy, pvedaemon, cluster
 - Siehe [[systems/loki-fluentbit]]
 ## HolmesGPT (SRE)
 - **Namespace**: holmes
 - **Pipeline**: Alertmanager → adapter (9095) → HolmesGPT → TG @schoen_sre_bot
 - **Model**: noris-glm (via ai.noris.de)
 - RKE2 ctrl-plane alerts disabled
 ## Alerting Rules
 - Via IaC Git Repo deployt (Ansible für CT141, ArgoCD für K8s)
 - RKE2 control-plane alerts disabled
 ## Related
 - [[systems/loki-fluentbit]]
 - [[systems/rke2-kubernetes]]
@@ -0,0 +1,64 @@
 ---
 title: Proxmox VE Cluster
 category: systems
 tags: [proxmox, virtualization, lxc, qemu, pve]
 created: "2026-04-28"
 modified: "2026-07-24"
 ---
 # Proxmox VE Cluster
 ## Cluster-Konfiguration
 - **Version:** PVE 9.2.3
 - **Nodes:** 9 (Quorum OK)
 - **Hypervisoren:** 10.0.20.x
 - **Guests:** ~30 LXC + ~10 QEMU VMs
 ## Storage
 - **vm_disks:** Primärer Storage für alle VMs/CTs (SSD)
 - **hdd_templates:** CT Templates
 - **Ceph RBD:** ceph-flash, ceph-hdd Pools (über K8s CSI)
 ## Netzwerk
 - VLAN-basiert, Bridge vmbr0
 - IP-Schema: 10.0.X.Y — siehe [[concepts/network-architecture]]
 ## Fluent Bit (Logging)
 - Alle 9 Hosts haben Fluent Bit aktiv
 - Inputs: systemd journal (pve*, corosync, pacemaker, ceph, zfs, smartd), auth.log, pveproxy/access.log, pvedaemon.log, cluster.log
 - **PVE Tasks** (`/var/log/pve/tasks/index`): UPID-Format (Node, PID, Task-Type, VMID, User, Status)
 - **Ceph Audit** (`/var/log/ceph/ceph.audit.log`): OSD/Pool/RBD Operationen
 - Output → Loki (10.0.30.207:3100)
 - Siehe [[systems/loki-fluentbit]]
 ## Wichtige Befehle
 ```bash
 pvecm status          # Cluster-Quorum
 pct status <vmid>     # Container-Status
 pct start/stop <vmid> # Container starten/stoppen
 qm status <vmid>      # VM-Status
 pvesh get /cluster/resources --type vm  # Alle VMs/CTs
 ```
 ## SSH-Zugriff
 - Key: `id_ed25519_proxmox` (funktioniert für 10.0.20.x Hosts)
 - Siehe [[reference/ssh-keys]]
 ## GPU Passthrough (AMD 1002:13c0)
 - **ms-a2-1** (10.0.20.92): VFIO config gefixt 2026-07-24 — siehe [Solution Doc](../../../docs/solutions/bug-fixes/2026-07-24-vfio-pci-module-loading-race-condition.md)
 - **ms-a2-2** (10.0.20.93): Funktioniert seit Initialisierung
 - Benötigte modprobe.d Config:
  - `blacklist amdgpu` + `blacklist drm` + `blacklist drm_kms_helper`
  - `options vfio-pci ids=1002:13c0` + `softdep amdgpu pre: vfio-pci`
 - Worker-05 (VM 102) läuft auf ms-a2-2 mit funktionierendem GPU-Passthrough
 ## Bekannte Probleme
 - osd.0 NVMe 92% Wear — Austausch planen
 - osd.2/5 nearfull (93-94%) — entlasten
 - CT110 kaputte libc — Reparatur ausstehend
 - ms-a2-1 GPU-Passthrough: Config gefixt, Reboot zur Verifikation ausstehend
 ## Related
 - [[systems/ceph-cluster]]
 - [[systems/rke2-kubernetes]]
 - [[reference/ip-map]]
@@ -0,0 +1,79 @@
 ---
 title: RKE2 Kubernetes Cluster
 category: systems
 tags: [kubernetes, rke2, cilium, argocd, cnpg, gitops]
 created: "2026-07-24"
 modified: "2026-07-24"
 ---
 # RKE2 Kubernetes Cluster
 ## Übersicht
 - **Version:** v1.35.6-rke2r1
 - **Nodes:** 6 (3 CP + 3 Worker), alle schedulable (keine CP Taints)
 - **CNI:** Cilium (LB IP Pool)
 - **Ingress:**两层 Traefik (extern CT9999 + intern K8s) + Authelia SSO
 ## Node IPs
 | Node | IP | Role |
 |------|----|------|
 | cp-01 | 10.0.30.51 | Control Plane |
 | cp-02 | 10.0.30.52 | Control Plane |
 | cp-03 | 10.0.30.53 | Control Plane |
 | worker-01 | 10.0.30.63 | Worker |
 | worker-04 | 10.0.30.64 | Worker |
 | worker-05 | 10.0.30.65 | Worker (GPU renderD128 ✅ on ms-a2-2) |
 ## Storage
 - **Ceph CSI**: ceph-flash (fast), ceph-hdd (bulk)
 - **CNPG PostgreSQL**: `postgres-main` Cluster (3/3 Ready), RW Service `postgres-main-rw.postgres.svc.cluster.local:5432`
 ## GitOps
 - **ArgoCD**: SSH Deploy Keys (read-only) auf Gitea
 - 15 Applications via SSH (`ssh://git@10.0.30.202:22/dominik/iac-homelab.git`)
 - **Gitea SSH user is `git`, not `gitea`** — `git.schoen.codes` resolves to Traefik (10.0.30.203, HTTP only), SSH is on 10.0.30.202:22
 - Workflow: IaC repo auschecken → ändern → commit+push → ArgoCD sync → verify → lokale Kopie löschen
 - Siehe [[concepts/gitops-workflow]]
 ## Backup
 - **Velero** v1.18.1 with `--features=EnableCSI`, S3 backend (noris cloud `homelab-velero-backups`)
 - **VolumeSnapshotClass** `ceph-rbd-snapclass` (rbd.csi.ceph.com, default)
 - Schedules: daily 01:00 UTC (30d TTL), weekly Sun 00:00 UTC (90d TTL)
 - **CNPG Postgres**: separate hourly backups via barmanObjectStore to S3 (30d retention)
 - All 24 PVCs (~430 GiB) now have off-site CSI snapshots
 - **Helm pitfall**: `features: EnableCSI` must be under `configuration:` in Velero Helm values (not top-level)
 ## Deployed Workloads
 | Workload | Namespace | Notes |
 |----------|-----------|-------|
 | Paperless-ngx v3.0.0 | paperless | PostgreSQL (CNPG), `enableServiceLinks: false` |
 | Gitea | gitea | git.schoen.codes |
 | Loki Stack | logging | LB 10.0.30.207:3100 |
 | Hindsight | hindsight | LB 10.0.30.201:9177 |
 | Authelia | authelia | SSO, Galera backend |
 | HolmesGPT | holmes | SRE Bot |
 | Traefik | traefik | Ingress Controller |
 | CNPG | postgres | PostgreSQL HA |
 ## GPU
 - worker-04: renderD128 ✅ (AMD GPU Passthrough, rombar=1, on n5pro)
 - worker-05: renderD128 ✅ (AMD GPU Passthrough, rombar=1, on ms-a2-2)
 - IaC: unified `amd-gpu` PCI mapping (n5pro + ms-a2-1 + ms-a2-2) + dynamic hostpci (one worker block, gpu flag)
 - ms-a2-1 has kernel BUG with 1002:13c0 (renderD128 missing) — worker-05 moved to ms-a2-2 (identical hardware)
 ## Known Pitfalls
 - `enableServiceLinks: false` bei Apps deren Service-Name mit Env-Vars kollidiert (z.B. Paperless `PAPERLESS_PORT`)
 - ArgoCD `--force` kann nicht mit ServerSideApply kombiniert werden
 - CoreDNS: `hosts` configBlock nicht params
 - etcd Defrag monatlich (CronJob)
 ## Related Skills
 - `rke2-cluster-administration` (devops)
 ## Related
 - [[systems/ceph-cluster]]
 - [[systems/gitea]]
 - [[systems/hindsight]]
 - [[systems/loki-fluentbit]]
 - [[concepts/gitops-workflow]]
 - [[reference/ip-map]]
@@ -1,12 +0,0 @@
 # Team Structure (Schön Consulting)
 ## Model
 CEO + Agents model. Two main profiles:
 - **infra-sre** - Security/Monitoring
 - **nutrition-coach** - Family Meals
 ## Operations
 - CEO orchestrates, agents operate
 - Each agent has their own chat
 - Cronjobs deliver directly to agent chats (not to CEO)
 - Shared Telegram group for entire team