diff --git a/Concepts/Monitoring-System.md b/Concepts/Monitoring-System.md deleted file mode 100644 index b45002f..0000000 --- a/Concepts/Monitoring-System.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -title: Monitoring-System -category: Infrastructure -tags: [monitoring, network-scans, security, cronjob] -created: "2026-04-28" -modified: "2026-04-28" ---- - -# Monitoring-System - -## Netzwerk-Scans -- **Häufigkeit:** Wöchentlich -- **Cronjob:** SRE-Agent (infra-sre) -- **Scan-Bereiche:** - - 10.0.10.x/24 (Gateway) - - 10.0.20.x/24 (Hypervisoren) - - 10.0.30.x/24 (Container/VMs) - - 10.0.40.x/24 (Clients) - - 10.0.50.x/24 (Smart Home, 26 hosts) - - 10.0.60.x/24 (DMZ/Reverse Proxies) - -## Smart Home Monitoring -- **Hosts:** 26 in 10.0.50.x -- **Scan-Methode:** SEGMENTS Dictionary -- **Ziel:** Geräte-Status und Verfügbarkeit - -## Security-Checks -- **SSH-Versionen:** 9.2p1-10.0p2 (gemischt) -- **TLS-Zertifikate:** Keine CN, self-signed oder SAN-only -- **Ports:** 2222 (SSH), 3306 (MySQL) - -## Wichtige Befehle -```bash -# Netzwerk-Scan durchführen -nmap -sn 10.0.20.0/24 - -# SSH-Version prüfen -ssh -V - -# TLS-Zertifikat prüfen -openssl s_client -connect : -``` - -## Cronjobs -- **Netzwerk-Scan:** Wöchentlich -- **Smart Home Scan:** Täglich -- **Rechnungen-Organizer:** Täglich 09:00 Uhr -- **Memory-Sync:** Täglich 06:00, 12:00, 18:00 Uhr diff --git a/Concepts/Proxmox-Cluster.md b/Concepts/Proxmox-Cluster.md deleted file mode 100644 index 58816e5..0000000 --- a/Concepts/Proxmox-Cluster.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Proxmox Cluster -category: Infrastructure -tags: [proxmox, virtualization, containers, lxc] -created: "2026-04-28" -modified: "2026-04-28" ---- - -# Proxmox Cluster - -## Cluster-Konfiguration -- **Hypervisoren:** In 10.0.20.x -- **Container:** In 10.0.30.x -- **VLAN-basiertes Netzwerk** mit Bridges -- **Shared Storage:** ZFS/ISCSI - -## Wichtige Befehle -```bash -# Container-Status prüfen -pct status - -# Container starten/stoppen -pct start -pct stop - -# Container konfigurieren -pct setcfg - -# Cluster-Status -pvecm status -``` - -## VLAN-Konfiguration -- **Bridge:** vmbr0 für Netzwerk -- **VLAN Tagging:** Für Segmentierung -- **IP-Adressierung:** 10.0.X.Y (siehe Netzwerk-Architektur) - -## Monitoring -- **Netzwerk-Scans:** Wöchentlich via cronjob -- **Smart Home:** 26 hosts in 10.0.50.x -- **SEGMENTS Dictionary:** Zentrale Konfiguration für Scan-Intervalle diff --git a/Concepts/qmd-Knowledge-Base.md b/Concepts/qmd-Knowledge-Base.md deleted file mode 100644 index 57c0294..0000000 --- a/Concepts/qmd-Knowledge-Base.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -title: qmd Wissenssystem -category: Tools -tags: [qmd, knowledge-base, embeddings, search] -created: "2026-04-28" -modified: "2026-04-28" ---- - -# qmd Wissenssystem - -## Überblick -qmd ist ein markdown-basiertes Wissenssystem mit: -- **Embeddings:** Semantische Suche -- **Suche:** qmd search -- **Index:** ~/.cache/qmd/ -- **MCP Server:** qmd mcp --http - -## Installation -```bash -npm install -g @tobilu/qmd -``` - -## Verwendung -```bash -# Embeddings generieren -qmd embed - -# Semantische Suche -qmd search "query" - -# MCP Server starten -qmd mcp --http - -# Server als Daemon starten -qmd mcp --http --daemon -``` - -## Memory-Integration -- **Verzeichnis:** ~/.hermes/memory/ -- **Struktur:** Entities/, Concepts/, Projects/, Templates/ -- **Sync:** Git + Cronjobs -- **Index:** qmd embed für neue/änderte Einträge - -## Arbeitsweise -1. **Manuell:** Markdown-Dateien werden direkt geschrieben -2. **Automatisch:** Cronjobs syncen und indexieren -3. **Suche:** qmd search für semantische Suche -4. **Index:** qmd embed für neue/änderte Einträge diff --git a/Entities/Infrastructure.md b/Entities/Infrastructure.md deleted file mode 100644 index ba50129..0000000 --- a/Entities/Infrastructure.md +++ /dev/null @@ -1,43 +0,0 @@ ---- -title: Infrastruktur-Übersicht -category: Infrastructure -tags: [networking, proxmox, monitoring, gitea] -created: "2026-04-28" -modified: "2026-04-28" ---- - -# Infrastruktur-Übersicht - -## Netzwerk-Architektur (Schön Consulting) -- **Addressing:** 10.0.X.Y (nicht standard /8-Notation) - - 10.0.10.x = Gateway - - 10.0.20.x = Hypervisoren (Proxmox Cluster) - - 10.0.30.x = Container/VMs - - 10.0.40.x = Clients - - 10.0.50.x = Smart Home (26 hosts) - - 10.0.60.x = DMZ/Reverse Proxies -- **TLS:** Alle Zertifikate haben kein CN (self-signed oder SAN-only), keine reverse-DNS -- **MySQL:** 3306 auf mehreren VMs in 30.x Subnetz exponiert -- **SSH:** Port 2222 auf allen Servern -- **SSH Versionen:** Gemischt (9.2p1-10.0p2) - -## Proxmox Cluster -- **Hypervisoren:** In 10.0.20.x -- **Container:** In 10.0.30.x -- **VLAN-basiertes Netzwerk** mit Bridges - -## Gitea Instance -- **URL:** https://git.familie-schoen.com -- **Auth:** SSH Key-basiert -- **Git User:** Dominik Schön -- **Repository:** memory.git (für Memory-Sync) - -## Monitoring -- **Netzwerk-Scans:** Wöchentlich via cronjob -- **Smart Home:** 26 hosts in 10.0.50.x -- **SEGMENTS Dictionary:** Zentrale Konfiguration für Scan-Intervalle - -## Wichtige Dienste -- **Embedding Model:** embeddinggemma-300M-Q8_0.gguf auf 10.0.30.92:8080 (OpenAI-kompatibel, CLS pooling) -- **Himalaya v1.2.0:** iCloud Mail (Email-Client) -- **qmd:** Markdown-basiertes Wissenssystem mit Embeddings diff --git a/Entities/Memory-System.md b/Entities/Memory-System.md deleted file mode 100644 index 8aadc00..0000000 --- a/Entities/Memory-System.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -title: Memory-System -category: Tools -tags: [memory, hindsight, qmd, markdown] -created: "2026-04-28" -modified: "2026-04-28" ---- - -# Memory-System - -## Hindsight Memory System -- **Version:** 1.0 -- **Provider:** OpenAI (via custom endpoint) -- **Model:** embeddinggemma-300M-Q8_0.gguf -- **Endpoint:** http://10.0.30.92:8080 -- **OpenAI-kompatibel:** Ja -- **CLS Pooling:** Ja - -## Hindsight Daemon -- **Status:** Läuft auf 10.0.30.92 -- **Config:** ~/.hindsight/profiles/hermes.env -- **API Key:** In hermes.env gespeichert -- **Log:** ~/.hindsight/daemon.log - -## qmd Integration -- **qmd:** Markdown-basiertes Wissenssystem -- **Embeddings:** Via qmd embed -- **Suche:** Via qmd search -- **MCP Server:** qmd mcp --http -- **Index:** ~/.cache/qmd/ - -## Memory-Verzeichnis -- **Pfad:** ~/.hermes/memory/ -- **Struktur:** - - Entities/ (Konkrete Dinge, Personen, Systeme) - - Concepts/ (Abstrakte Ideen, Prinzipien, Methoden) - - Projects/ (Aktuelle und abgeschlossene Projekte) - - Templates/ (Vorlagen für neue Einträge) - -## Git-Sync -- **Remote:** https://git.familie-schoen.com/dominik/memory.git -- **Branch:** main -- **User:** Dominik Schön -- **Commit-Messages:** "Auto-sync: YYYY-MM-DD HH:MM" - -## Arbeitsweise -1. **Manuell:** Memory-Einträge werden direkt in Markdown geschrieben -2. **Automatisch:** Cronjobs syncen und indexieren -3. **Suche:** qmd search für semantische Suche -4. **Index:** qmd embed für neue/änderte Einträge diff --git a/concepts/credential-policy.md b/concepts/credential-policy.md new file mode 100644 index 0000000..c6982cf --- /dev/null +++ b/concepts/credential-policy.md @@ -0,0 +1,47 @@ +--- +title: Credential Policy +category: concepts +tags: [credentials, security, 1password, secrets] +created: "2026-07-24" +modified: "2026-07-24" +--- + +# Credential Policy + +## Regel +Credentials, API-Keys und Secrets gehören in den 1Password Vault "Hermes" und werden von dort abgerufen (via `op` CLI). Passwörter in .env-Dateien sind ebenfalls akzeptabel. **Keine Secrets in Git-Repositories.** + +## 1Password Vaults +| Vault | Zweck | Used By | +|-------|-------|---------| +| Hermes | General secrets | Hermes Agent | +| Kubernetes ESO | K8s ExternalSecrets Operator | ESO → K8s Secrets | + +## ESO (External Secrets Operator) +- Liest aus Vault "Kubernetes ESO" +- Failing ExternalSecrets burn 1000 requests/hr +- Bei Problemen: ESO scale 0, warten, fixen, wieder scale 1 + +## Bekannte Issues +- **`mariadb-root` Item in 1Password**: PASSWORT IST FALSCH — echtes PW in `/root/.my.cnf` auf VM300 +- **1Password CLI v2.35**: assignment statements (`username=val`), NOT `--username` flags +- **op CLI**: `--reveal` für blocked fields +- **Account-Level Rate Limit**: 1Password rate-limited auf ACCOUNT-Ebene, nicht per SA. ESO (25 ExternalSecrets × 1h) kann das gemeinsame Kontingent erschöpfen → op item get auf Hermes Host schlägt fehl. Fix: ESO scale 0, 1h warten, refresh intervals auf 6h erhöhen. Siehe Solution Doc: `docs/solutions/architecture/2026-07-24-1password-account-level-rate-limit.md` + +## K8s Secret Pattern +```yaml +# ExternalSecret referenziert 1Password Item +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +spec: + data: + - secretKey: password + remoteRef: + key: "Paperless DB Pass" # 1Password Item Title + property: password +``` + +## Related +- [[reference/ssh-keys]] +- [[systems/gitea]] (NIEMALS öffentlich — enthält Secrets) +- [[systems/hindsight]] diff --git a/Concepts/Email-Organization.md b/concepts/email-organization.md similarity index 100% rename from Concepts/Email-Organization.md rename to concepts/email-organization.md diff --git a/concepts/gitops-workflow.md b/concepts/gitops-workflow.md new file mode 100644 index 0000000..c0af704 --- /dev/null +++ b/concepts/gitops-workflow.md @@ -0,0 +1,41 @@ +--- +title: GitOps Workflow +category: concepts +tags: [gitops, argocd, iac, terraform, ansible, workflow] +created: "2026-07-24" +modified: "2026-07-24" +--- + +# GitOps Workflow + +## Prinzip +Infra-Changes **bevorzugt** über GitOps. Geht nicht immer. Ausnahmen müssen von Dominik freigegeben werden. + +## Workflow +1. IaC Repo lokal auschecken (`/root/iac-homelab` auf VM200) +2. Ändern (Tofu/Ansible/K8s Manifeste) +3. Commit + Push (zu BEIDEN Gitea Remotes: origin + k8s) +4. ArgoCD sync (oder auto-sync) +5. Verify (kubectl get, curl, etc.) +6. Lokale Kopie löschen + +## ArgoCD +- 15 Applications via SSH Deploy Keys +- `ssh://gitea@git.schoen.codes:22/dominik/iac-homelab.git` +- `--force` kann nicht mit ServerSideApply kombiniert werden +- Bei hängendem Operation-State: App terminieren und neu anstoßen + +## IaC Tools +- **OpenTofu**: VM/CT Provisioning, GPU PCI Mapping +- **Ansible**: RKE2 Installation, Node Taints, Fluent Bit +- **K8s Manifeste**: clusters/main/apps/*.yaml + +## Was NICHT über GitOps geht +- Direkte DB-Operationen (SQL) +- Ceph `ceph` CLI Befehle +- Proxmox `pct`/`qm` Live-Operationen +- Notfall-Reparaturen + +## Related +- [[systems/rke2-kubernetes]] +- [[systems/gitea]] diff --git a/Concepts/Network-Architecture.md b/concepts/network-architecture.md similarity index 53% rename from Concepts/Network-Architecture.md rename to concepts/network-architecture.md index 5fb881c..6ee658d 100644 --- a/Concepts/Network-Architecture.md +++ b/concepts/network-architecture.md @@ -1,25 +1,24 @@ --- title: Netzwerk-Architektur -category: Infrastructure +category: concepts tags: [networking, ip-addressing, subnets, vlan] created: "2026-04-28" -modified: "2026-04-28" +modified: "2026-07-24" --- # Netzwerk-Architektur ## IP-Adressierung (10.0.X.Y) -Nicht-standard /8-Notation, aber mit klarem Schema: - 10.0.10.x = Gateway -- 10.0.20.x = Hypervisoren (Proxmox Cluster) -- 10.0.30.x = Container/VMs +- 10.0.20.x = Hypervisoren (Proxmox Cluster, 9 Nodes) +- 10.0.30.x = Container/VMs (Services) - 10.0.40.x = Clients - 10.0.50.x = Smart Home (26 hosts) - 10.0.60.x = DMZ/Reverse Proxies ## VLAN-Struktur - Jedes Segment hat eigenes VLAN -- Proxmox Bridge-Konfiguration +- Proxmox Bridge-Konfiguration (vmbr0) - Inter-Segment-Kommunikation via Gateway ## TLS-Zertifikate @@ -28,6 +27,13 @@ Nicht-standard /8-Notation, aber mit klarem Schema: - Keine reverse-DNS Einträge ## Wichtige Ports -- **MySQL:** 3306 (auf mehreren VMs in 30.x) -- **SSH:** 2222 (alle Server) -- **Embedding Model:** 8080 (10.0.30.92) +Siehe [[reference/ports]] für vollständige Liste. + +## DNS +- externe Domains: familie-schoen.com, schoen.codes, essen.schoen.eu +- interne Auflösung über K8s CoreDNS + +## Related +- [[reference/ip-map]] +- [[reference/ports]] +- [[systems/proxmox-cluster]] diff --git a/dominik-schoen.md b/dominik-schoen.md deleted file mode 100644 index eacc9f2..0000000 --- a/dominik-schoen.md +++ /dev/null @@ -1,33 +0,0 @@ -# Dominik Schön - -## Overview -CEO Schön Consulting Inc. PhD (Dr.). Former CodeCamp:N, Noris Network. - -## Personal Details -- **Family:** Sarah (Wife, loves carbs), Cleo (4y, loves pasta) -- **Health Goals:** -15kg target (currently 95kg, 1.75m) -- **Food Preferences:** Asian cuisine preferred. No allergies. -- **Shopping:** Aldi (Fridays). HelloFresh occasionally. - -## Work Rhythm -- Home Office: Mon/Wed/Fri -- Meal Prep: Tue/Thu -- Meals: Daily dinner. Mon/Wed/Fri. Weekend: 2 warm meals. -- Cleo Kitabox: Mon-Thu - -## Communication -- Prefers Du-form (German informal) -- CEO persona for interactions - -## Network Architecture -- **Addressing:** 10.0.X.Y where X = segment index - - 10.0.10.x = Gateway - - 10.0.20.x = Hypervisoren (Proxmox cluster) - - 10.0.30.x = Container/VMs - - 10.0.40.x = Clients - - 10.0.50.x = Smart Home (26 hosts) - - 10.0.60.x = DMZ/Reverse Proxies -- All TLS certs have no CN (self-signed or SAN-only) -- MySQL (3306) exposed on multiple VMs in 30.x subnet -- SSH on port 2222 -- Mixed OpenSSH versions: 9.2p1-10.0p2 across the fleet diff --git a/email-organization.md b/email-organization.md deleted file mode 100644 index 4adb653..0000000 --- a/email-organization.md +++ /dev/null @@ -1,24 +0,0 @@ -# Email Organization - -## System -- **Client:** Himalaya v1.2.0 with iCloud -- **Config:** /home/debian/.config/himalaya/config.toml -- **Primary:** dominik.schoen87@icloud.com -- **Custom Domains:** 12 active - - grafiert.de - - famschoen.eu - - dominikschoen.de - - familie-schoen.com - - schoen.codes - - schoen.eu - -## Folder Structure -- **Aliases:** Sent=Sent Messages, Trash=Deleted Messages -- **Organization:** Hierarchical (e.g., Rechnungen/Bestellungen) -- **Daily Cronjob:** 9:00 AM invoice organizer - - Invoices (subject/sender contains "rechnung") → "Rechnungen_jjjj_mm" - - Amazon orders (subject starts with "Bestellt:") → "Rechnungen_jjjj_mm/Bestellungen_jjjj_mm" - -## Rules -- Never send emails automatically - only drafts or manual approval -- Previously sent ~150 emails via himalaya without user awareness - this must not happen again diff --git a/embedding-model.md b/embedding-model.md deleted file mode 100644 index 286e00c..0000000 --- a/embedding-model.md +++ /dev/null @@ -1,13 +0,0 @@ -# Embedding Model (2026-04-26) - -## Configuration -- **Model:** embeddinggemma-300M-Q8_0.gguf -- **Service:** systemd user service -- **Endpoint:** http://10.0.30.92:8080 (OpenAI-compatible, CLS pooling) - -## Status -- /health → ok -- /v1/embeddings → 200 OK - -## Usage -- Hindsight daemon needs hindsight_embed installed & configured diff --git a/Entities/Email-System.md b/entities/email-system.md similarity index 100% rename from Entities/Email-System.md rename to entities/email-system.md diff --git a/Entities/Health-Fitness.md b/entities/health-fitness.md similarity index 100% rename from Entities/Health-Fitness.md rename to entities/health-fitness.md diff --git a/entities/infrastructure.md b/entities/infrastructure.md new file mode 100644 index 0000000..93e7afb --- /dev/null +++ b/entities/infrastructure.md @@ -0,0 +1,42 @@ +--- +title: Infrastruktur-Übersicht +category: entities +tags: [homelab, hardware, overview] +created: "2026-04-28" +modified: "2026-07-24" +--- + +# Infrastruktur-Übersicht + +## Physikalische Hardware + +### Proxmox Cluster (PVE 9.2.3) +- **9 Nodes**, Quorum OK +- Hypervisoren in 10.0.20.x +- Siehe [[systems/proxmox-cluster]] + +### Ceph Cluster +- **14 OSDs** (HDD + NVMe/SSD混合) +- Siehe [[systems/ceph-cluster]] + +### RKE2 Kubernetes Cluster +- **6 Nodes** (3 CP + 3 Worker), v1.35.6-rke2r1 +- Alle Nodes schedulable (keine CP Taints) +- Siehe [[systems/rke2-kubernetes]] + +## Netzwerk-Architektur +Siehe [[concepts/network-architecture]] + +## Wichtige Dienste +- **Gitea**: git.schoen.codes (K8s) — siehe [[systems/gitea]] +- **Loki + Fluent Bit**: Logging Stack — siehe [[systems/loki-fluentbit]] +- **Hindsight**: Semantic Memory (K8s) — siehe [[systems/hindsight]] +- **Galera + MaxScale**: MariaDB HA — siehe [[systems/galera-maxscale]] +- **Seafile**: cloud.familie-schoen.com — siehe [[systems/seafile]] +- **Paperless-ngx v3.0.0**: K8s, PostgreSQL backend — siehe [[systems/rke2-kubernetes]] +- **Immich**: CT111 (n5pro), Migration zu K8s geplant + +## Quick Reference +- IP-Map: [[reference/ip-map]] +- SSH-Keys: [[reference/ssh-keys]] +- Ports: [[reference/ports]] diff --git a/Entities/Team-Structure.md b/entities/team-structure.md similarity index 100% rename from Entities/Team-Structure.md rename to entities/team-structure.md diff --git a/index.md b/index.md index 7482ac1..46963ce 100644 --- a/index.md +++ b/index.md @@ -1,23 +1,54 @@ # Memory Index +> Das LLM Wiki ist die **kuratierte, browsbare Wissensbasis** des Homelabs. +> Es wird beim Compound-Learning Zyklus gepflegt (siehe SOUL.md "Continuous Learning"). +> MEMORY.md enthält komprimierte Zeiger, das Wiki enthält die Details. + +## Struktur + +| Verzeichnis | Inhalt | +|-------------|--------| +| `entities/` | Personen, Organisationen, physische Hardware | +| `systems/` | Software-Systeme und Services | +| `concepts/` | Abstrakte Patterns & Konventionen | +| `reference/` | Quick-Lookup Tabellen (IPs, Keys, Ports) | + ## Entities -- [[cloud.familie-schoen.com]] (Entities/cloud.familie-schoen.com.md) - Seafile 13.0.19 file sync server, public HTTPS -- [[Infrastructure]] (Entities/Infrastructure.md) - Proxmox cluster: 2 nodes (10.0.3.1, 10.0.3.2), LXC containers, VLAN networking -- [[Email-System]] (Entities/Email-System.md) - himalaya v1.2.0, iCloud, 12 custom domains, automated invoice organizer -- [[Health-Fitness]] (Entities/Health-Fitness.md) - -15kg goal (95kg/1.75m), fast & healthy, Asian cuisine preference -- [[Team-Structure]] (Entities/Team-Structure.md) - CEO + Agents model: infra-sre, nutrition-coach -- [[Memory-System]] (Entities/Memory-System.md) - qmd-based markdown wiki, git sync, Karpathy LLM Wiki pattern +- [[entities/infrastructure]] — Physikalische Hardware, Cluster-Übersicht +- [[entities/team-structure]] — CEO + Agents Modell +- [[entities/dominik-schoen]] — Nutzerprofil +- [[entities/email-system]] — Himalaya, iCloud, 12 Domains +- [[entities/health-fitness]] — Gesundheits-Ziele, Ernährung + +## Systems +- [[systems/proxmox-cluster]] — PVE 9.2.3, 9 Nodes, Fluent Bit +- [[systems/ceph-cluster]] — 14 OSDs, EC Pools, bekannte Probleme +- [[systems/rke2-kubernetes]] — 6 Nodes, ArgoCD, CNPG, Workloads +- [[systems/galera-maxscale]] — 3 Galera + 2 MaxScale, VIP .70 +- [[systems/loki-fluentbit]] — Logging Stack, 40+ Targets +- [[systems/gitea]] — Git Server, CI, ArgoCD SSH Keys +- [[systems/hindsight]] — Semantic Memory, K8s, API +- [[systems/monitoring]] — Prometheus, Grafana, HolmesGPT +- [[systems/seafile]] — cloud.familie-schoen.com, Seafile 13.0.19 ## Concepts -- [[Network-Architecture]] (Concepts/Network-Architecture.md) - 10.0.X.Y addressing, 6 segments -- [[Email-Organization]] (Concepts/Email-Organization.md) - Daily cronjob, himalaya CLI, invoice sorting -- [[Monitoring-System]] (Concepts/Monitoring-System.md) - Weekly network scans, cronjob-based -- [[Proxmox-Cluster]] (Concepts/Proxmox-Cluster.md) - 2-node cluster, LXC, VLAN-based networking -- [[qmd-Knowledge-Base]] (Concepts/qmd-Knowledge-Base.md) - Semantic search, markdown wiki, git sync +- [[concepts/network-architecture]] — 10.0.X.Y Schema, VLANs +- [[concepts/gitops-workflow]] — IaC → Git → ArgoCD → Verify +- [[concepts/credential-policy]] — 1Password, ESO, keine Secrets in Git +- [[concepts/email-organization]] — Rechnungs-Organizer, Himalaya -## Sources -- [[llm-wiki-pattern]] - Karpathy's LLM Wiki pattern for building personal knowledge bases +## Reference +- [[reference/ip-map]] — IP → Host → Service Mapping +- [[reference/ssh-keys]] — Key → Zweck → Fingerprint +- [[reference/ports]] — Port → Service → Host -## Session Entries -- [[memory-entry-001]] - System Initialization (2026-04-28) -- [[memory-entry-002]] - Memory Process Fix (2026-04-29) +## Memory Layer Architektur +| Layer | System | Pfad | Rolle | +|-------|--------|------|-------| +| L0 Hot | MEMORY.md + USER.md | `~/.hermes/memories/` | System-Prompt Injection | +| L1 Curated | LLM Wiki | `~/.hermes/memory/` | Browsbares Wissen (this) | +| L2 Semantic | Hindsight | K8s PostgreSQL | Vektor-Suche | +| L3 Procedural | Skills | `~/.hermes/skills/` | Wie-geht-es Prozeduren | +| L4 Episodic | session_search | SQLite | Gesprächsverläufe | +| L5 Solutions | Solution Docs | `~/docs/solutions/` | Problem-Lösungs-Paare | +| L6 Project | AGENTS.md | pro Repo | Repo-Konventionen | diff --git a/llm-wiki-pattern.md b/llm-wiki-pattern.md deleted file mode 100644 index c8f10f1..0000000 --- a/llm-wiki-pattern.md +++ /dev/null @@ -1,31 +0,0 @@ -# LLM Wiki Pattern - -## Core Idea -A pattern for building personal knowledge bases using LLMs. The LLM incrementally builds and maintains a persistent wiki - a structured, interlinked collection of markdown files that sits between you and the raw sources. - -## Three Layers -1. **Raw Sources** - Immutable collection of source documents (articles, papers, images, data files) -2. **The Wiki** - LLM-generated markdown files (summaries, entity pages, concept pages, comparisons, overview, synthesis) -3. **The Schema** - Document that tells the LLM how the wiki is structured, conventions, and workflows - -## Operations -- **Ingest** - Read source, discuss key takeaways, write summary page, update index, update relevant entity/concept pages, append entry to log -- **Query** - Search for relevant pages, read them, synthesize answer with citations -- **Lint** - Periodically health-check wiki for contradictions, stale claims, orphan pages, missing cross-references - -## Indexing and Logging -- **index.md** - Content-oriented catalog of everything in the wiki, organized by category -- **log.md** - Chronological append-only record of what happened and when - -## Benefits -- Knowledge is compiled once and kept current, not re-derived on every query -- Cross-references are already there -- Contradictions have already been flagged -- Synthesis reflects everything read -- Maintenance burden is near zero (LLMs don't get bored) - -## Tools -- **qmd** - Query Markup Documents: on-device search engine combining BM25, vector semantic search, and LLM reranking -- **Obsidian** - IDE for browsing and editing wiki -- **Marp** - Markdown-based slide decks -- **Dataview** - Run queries over page frontmatter diff --git a/log.md b/log.md index 3606c52..52d778f 100644 --- a/log.md +++ b/log.md @@ -1,27 +1,101 @@ # Memory Log +## [2026-07-24] fix | Velero CSI Volume Snapshots Three-Layer Fix +- Velero backups PartiallyFailed for 121 days — all PVCs skipped +- Fix 1: Created VolumeSnapshotClass `ceph-rbd-snapclass` (rbd.csi.ceph.com) +- Fix 2: Migrated ArgoCD repoURLs from dead Gitea (10.0.30.105) to new (ssh://git@10.0.30.202:22) + - Gitea SSH user is `git` not `gitea`; git.schoen.codes → Traefik (10.0.30.203), SSH on 10.0.30.202 + - New SSH deploy key generated, added to Gitea repo +- Fix 3: `features: EnableCSI` must be under `configuration:` in Velero Helm values (not top-level) + - Must be string, not array — array form breaks Helm template +- Verified: test backup created VolumeSnapshot + VolumeSnapshotContent, uploaded to S3 ✅ +- Solution doc: docs/solutions/bug-fixes/2026-07-24-velero-csi-snapshots-three-layer-fix.md +- Commits: 5ebfa30, 19f35bc, cd18c58, baf85f5, f652bfb + +## [2026-07-24] fix | VFIO Module Loading Race Condition RCA + Fix +- RCA: GPU passthrough failed on ms-a2-1 due to missing modprobe config (NOT kernel version) +- Root cause: Missing `softdep amdgpu pre: vfio-pci` + incomplete DRM blacklist → race condition (302s vs 2.4s bind time) +- Fix applied to ms-a2-1: Added `blacklist drm`, `blacklist drm_kms_helper`, `softdep amdgpu pre: vfio-pci`, rebuilt initramfs +- Verification pending (reboot required, 4 VMs + OSD 13 on ms-a2-1) +- Solution doc: docs/solutions/bug-fixes/2026-07-24-vfio-pci-module-loading-race-condition.md +- Also documented: K8s persistent storage architecture (all PVCs on Ceph RBD, 3x replication) +- Solution doc: docs/solutions/architecture/2026-07-24-k8s-persistent-storage-ceph-rbd.md + +## [2026-07-24] fix | worker-05 GPU Passthrough — Move to ms-a2-2 +- Problem: worker-05 (VM 102) on ms-a2-1 had kernel BUG with 1002:13c0 GPU — /dev/dri/renderD128 missing +- ms-a2-1 and ms-a2-2 have identical hardware (AMD Ryzen 9 9955HX, GPU 1002:13c0) +- Solution: Moved VM 102 from ms-a2-1 to ms-a2-2 (offline migration, Ceph RBD shared storage) +- Result: renderD128 ✅ visible in guest, amdgpu driver loaded, all pods running +- IaC: main.tf cleaned up — single unified rke2_worker block, worker-05 node=ms-a2-2, worker-06 removed +- PCI mapping: amd-gpu now includes n5pro + ms-a2-1 + ms-a2-2 +- Commit: 21bd335 +- Also: ms-a2-2 upgraded PVE 9.1.1→9.2.5, kernel 7.0.14-6-pve, VFIO persistent after reboot, HA integrated + +## [2026-07-24] init | ms-a2-2 Node Initialization +- New Proxmox node ms-a2-2 (10.0.20.93, nodeid 1) joined cluster +- Hardware: AMD Ryzen 9 9955HX (32 threads), 91GB RAM, 1.8TB NVMe +- GPU: AMD Radeon 1002:13c0 (identical to ms-a2-1) — VFIO passthrough configured +- PCI mapping: amd-gpu-ms-a2-2 created, added to generic amd-gpu mapping +- Ceph OSD 14: created on nvme0n1 (1.8TB SSD), class ssd, weight 1.82 +- Fluent-bit: installed + configured (hostname adapted to ms-a2-2) +- APT sources: fixed from enterprise to no-subscription (ceph + pve) +- LLM Wiki: ip-map.md, ceph.md updated; AGENTS.md updated (9 nodes) +- Note: PVE 9.1.1 on ms-a2-2 (rest of cluster is 9.2.3) — upgrade pending + +## [2026-07-24] cleanup | proxmox1 Permanent Removal + Ceph PG Fixes +- Removed proxmox1 from Ceph CRUSH (empty host bucket, weight 0) +- Also removed stale empty buckets: proxmox, px-tmp20 +- Updated corosync.conf: removed proxmox1 node entry, config_version 14→15, expected_votes 9→8 +- Removed stale /etc/pve/nodes/{proxmox1,proxmox,pve,px-tmp20} +- Updated IaC: proxmox_api_url 10.0.20.10→10.0.20.91, template_node proxmox1→n5pro, target_nodes list updated +- Updated AGENTS.md topology (9→8 nodes), LLM Wiki ip-map.md + ceph.md +- Commit: 469d29d +- Additional: Manual pg-upmap for stuck PGs 5.13→[1,6,7], 6.6c→[11,8,7] — all clean+remapped eliminated + +## [2026-07-24] fix | Ceph Remapped PGs — Weight Imbalance + pg_num Mismatch +- RCA: 11 active+clean+remapped PGs caused by CRUSH placement failure +- Root cause 1: Pool 6 (hdd_disk) pg_num=120 ≠ pgp_num=112 (autoscaler mid-merge) +- Root cause 2: Extreme HDD host weight imbalance (n5pro=10TB vs proxmox6=0.3TB) +- Fix: Equalized pg_num→112, set nopgchange=true, reweighted osd.7 0.80→1.0 +- Result: 9/11 PGs recovering (backfilling at 36MiB/s), 2 cosmetic clean+remapped remain +- Created systems/ceph.md with full OSD/pool/CRUSH documentation + +## [2026-07-24] update | Gitea Actions K8s Runner Bug + DinD Pitfalls +- Documented Gitea 1.27.0 bug: actions_ready_job queue doesn't assign tasks to K8s-based runners +- Updated systems/gitea.md with known issue, DinD pitfalls, runner management via DB +- Solution doc: docs/solutions/bug-fixes/2026-07-24-gitea-actions-k8s-runner-task-assignment-bug.md +- IaC committed: runner-deployment.yaml + runner-config.yaml (DinD TCP sidecar pattern) + ## [2026-04-28] init | Memory System Created - Created memory directory structure at ~/.hermes/memory - Installed qmd 2.1.0 (Query Markup Documents) -- Set up qmd collection for ~/.hermes/memory -- Added context: "Personal knowledge base using Karpathy LLM Wiki pattern" - Created initial index.md with entities, concepts, and sources - Created Entity pages: Infrastructure, Email-System, Health-Fitness, Team-Structure, Memory-System - Created Concept pages: Network-Architecture, Email-Organization, Monitoring-System, Proxmox-Cluster, qmd-Knowledge-Base -- Created root-level legacy pages -- Initialized git repo, made 4 commits, pushed to origin/main -- Initial embeddings generated (0 files, 0 entries) +- Initialized git repo, pushed to origin/main ## [2026-04-29] fix | Memory Process Established - Corrected stale memory-entry-001.md -- Created memory-entry-002.md documenting process fix - Established protocol: memory update + git sync on every session -- Protocol: always search memory before responding ## [2026-04-30] ingest | cloud.familie-schoen.com - Full port scan, DNS, SSL, HTTP headers, API analysis - Identified: Seafile 13.0.19 on nginx, Let's Encrypt SSL -- Domain: cloud.familie-schoen.com → schoen.sytes.net → 84.39.84.125 -- Only ports 80/443 open, HTTP→HTTPS 308 redirect -- Created: Entities/cloud.familie-schoen.com.md, raw/articles/cloud-familie-schoen-com-scan.md -- Updated: index.md + +## [2026-07-24] restructure | Full Wiki Restructuring +- Deleted 5 root-duplicate files (dominik-schoen, email-organization, team-structure, llm-wiki-pattern, embedding-model) +- Deleted obsolete Entities/Memory-System.md, Concepts/qmd-Knowledge-Base.md +- Deleted raw/articles/ and memory-entry-001/002.md (stale) +- Renamed Entities/ → entities/, Concepts/ → concepts/ (lowercase) +- Created new directories: systems/, reference/ +- Moved Proxmox-Cluster, Monitoring-System, seafile to systems/ +- Updated all existing pages with current data (April → July 2026): + - Infrastructure: 2 Nodes → 9 Nodes, added Ceph/RKE2/Galera/Loki + - Proxmox-Cluster: PVE 9.2.3, 9 Nodes, Fluent Bit details, PVE Tasks, Ceph Audit + - Network-Architecture: cleaned up, links to reference pages + - Monitoring: Prometheus/Grafana/Loki/HolmesGPT (was just "weekly network scans") +- Created 6 new system pages: ceph-cluster, rke2-kubernetes, galera-maxscale, loki-fluentbit, gitea, hindsight +- Created 3 reference pages: ip-map, ssh-keys, ports +- Created 2 concept pages: gitops-workflow, credential-policy +- Rewrote index.md with new structure + Memory Layer Architecture table +- Total: 18 pages (was 21 with dupes, now 18 clean unique pages) diff --git a/memory-entry-001.md b/memory-entry-001.md deleted file mode 100644 index cd09669..0000000 --- a/memory-entry-001.md +++ /dev/null @@ -1,29 +0,0 @@ -# Memory Entry 001 - System Initialization - -## Date -2026-04-28 - -## Summary -Initial setup of the markdown-based memory system using qmd and the Karpathy LLM Wiki pattern. Full Entity/Concept page creation. - -## Details -- Installed qmd 2.1.0 (Query Markup Documents) from github.com/tobi/qmd -- Set up qmd collection for ~/.hermes/memory -- Created index.md with entities, concepts, and sources -- Created log.md for chronological record of changes -- Created Entity pages: Infrastructure.md, Email-System.md, Health-Fitness.md, Team-Structure.md, Memory-System.md -- Created Concept pages: Network-Architecture.md, Email-Organization.md, Monitoring-System.md, Proxmox-Cluster.md, qmd-Knowledge-Base.md -- Created root pages: dominik-schoen.md, email-organization.md, embedding-model.md, team-structure.md, llm-wiki-pattern.md -- Initialized git repo, made 4 commits, pushed to origin/main - -## Status -- [x] qmd installed and configured -- [x] Memory directory structure created (Entities/Concepts/Projects/Templates) -- [x] All entity and concept pages written -- [x] Git repo initialized and synced to Gitea (4 commits on main) -- [x] Gitea remote configured and working -- [x] SSH keys configured for Gitea - -## Notes -- Root-level entity/concept files (e.g. dominik-schoen.md) are legacy copies; actual pages are in Entities/ and Concepts/ -- index.md needs periodic updates as new pages are added diff --git a/memory-entry-002.md b/memory-entry-002.md deleted file mode 100644 index 7d7a15c..0000000 --- a/memory-entry-002.md +++ /dev/null @@ -1,25 +0,0 @@ -# Memory Entry 002 - Memory System Process Fix - -## Date -2026-04-29 - -## Summary -Fixed stale memory management. Established automatic memory update + git sync protocol. - -## Details -- Identified that memory-entry-001.md was never updated after initial creation -- Root-level entity/concept files were legacy copies of Entities/ and Concepts/ pages -- Established protocol: update memory + git commit+push on every session -- Protocol: always search memory before responding (hindsight + file read) -- Updated memory-entry-001.md to reflect complete 04-28 work - -## Status -- [x] memory-entry-001.md corrected -- [x] New memory-entry-002.md created -- [ ] log.md updated -- [ ] index.md updated to match current structure -- [ ] git commit + push - -## Notes -- User expects German communication (Du-Form) -- Session was auto-reset on daily schedule — memory sync must survive resets diff --git a/raw/articles/cloud-familie-schoen-com-scan.md b/raw/articles/cloud-familie-schoen-com-scan.md deleted file mode 100644 index 9a47097..0000000 --- a/raw/articles/cloud-familie-schoen-com-scan.md +++ /dev/null @@ -1,66 +0,0 @@ -# cloud.familie-schoen.com - Full Scan - -Date: 2026-04-30 07:28 UTC -Agent: hermes - -## DNS -``` -cloud.familie-schoen.com → schoen.sytes.net → 84.39.84.125 -``` - -## Port Scan (20 ports checked) -| Port | Status | -|------|--------| -| 80 | OPEN | -| 443 | OPEN | -| 22 | closed | -| 25 | closed | -| 8080 | closed | -| 8443 | closed | -| 993 | closed | -| 995 | closed | -| 3000 | closed | -| 3306 | closed | -| 5432 | closed | -| 587 | closed | -| 9090 | closed | -| 2082-2096 | closed | - -## HTTP (port 80) -``` -HTTP/1.1 308 Permanent Redirect -Location: https://cloud.familie-schoen.com/ -``` - -## HTTPS (port 443) -- Server: nginx -- Redirect: `/accounts/login/?next=/` (Seafile Seahub login) -- Security: `cache-control: no-cache, no-store, must-revalidate, private` -- Cookie: `sfcsrftoken` (365d, SameSite=Lax), `sessionid` (24h, HttpOnly) - -## Seafile Version -- Version: 13.0.19 (via `/api2/server-info/`) -- Features: `["seafile-basic"]` -- CSS timestamps: `t=20250802` (seafile-ui.css), `t=20251013` (seahub.css) - -## SSL Certificate -- Subject: CN=cloud.familie-schoen.com -- Issuer: C=US, O=Let's Encrypt, CN=R12 -- Chain: R12 → ISRG Root X1 - -## API Endpoints -| Endpoint | Status | -|----------|--------| -| `/api2/server-info/` | 200 (unauthenticated) | -| `/api2/permissions/` | 404 | -| `/api2/repos/` | 403 | -| `/api2/notification-info/` | 404 | -| `/api/v2.1/notifications/` | 403 | -| `/seafhttp/` | 404 | -| `/webdav/` | 404 | -| `/dav/` | 404 | - -## Raw HTML Title -``` -Log In - Private Seafile -``` diff --git a/reference/ip-map.md b/reference/ip-map.md new file mode 100644 index 0000000..eac63d3 --- /dev/null +++ b/reference/ip-map.md @@ -0,0 +1,75 @@ +--- +title: IP-Map (Quick Reference) +category: reference +tags: [ip, network, reference, quick-lookup] +created: "2026-07-24" +modified: "2026-07-24" +--- + +# IP-Map + +## Proxmox Hosts (10.0.20.x) — 9 Nodes +| IP | Hostname | Node ID | Notes | +|----|----------|---------|-------| +| 10.0.20.20 | proxmox2 | 3 | | +| 10.0.20.30 | proxmox3 | 4 | | +| 10.0.20.40 | proxmox4 | 2 | | +| 10.0.20.50 | proxmox5 | 5 | MON, MGR | +| 10.0.20.60 | proxmox6 | 6 | | +| 10.0.20.70 | proxmox7 | 7 | MON, Traefik CT99999 | +| 10.0.20.91 | n5pro | 8 | Templates 9000/9001/9002 | +| 10.0.20.92 | ms-a2-1 | 9 | OSD 13 (SSD 1.8TB), GPU 1002:13c0 | +| 10.0.20.93 | ms-a2-2 | 1 | OSD 14 (SSD 1.8TB), GPU 1002:13c0 | + +> ⚠️ n5pro (10.0.20.91, nodeid 8) ≠ proxmox7 (10.0.20.70, nodeid 7) — separate Nodes! +> proxmox1 (10.0.20.10, nodeid 1) wurde dauerhaft entfernt (2026-07-24). +> Immer `pvecm nodes` für kanonische Liste prüfen. + +## Kubernetes Nodes (10.0.30.5x-6x) +| IP | Node | Role | +|----|------|------| +| 10.0.30.51 | cp-01 | Control Plane | +| 10.0.30.52 | cp-02 | Control Plane | +| 10.0.30.53 | cp-03 | Control Plane | +| 10.0.30.63 | worker-01 | Worker | +| 10.0.30.64 | worker-04 | Worker (GPU ✅) | +| 10.0.30.65 | worker-05 | Worker (GPU defekt) | + +## Database Layer (10.0.30.7x-8x) +| IP | Host | Service | +|----|------|---------| +| 10.0.30.70 | — | MaxScale VIP (keepalived) :3306 | +| 10.0.30.71 | VM300 | Galera db1 (ms-a2-1) | +| 10.0.30.72 | VM301 | Galera db2 (proxmox3) | +| 10.0.30.73 | VM302 | Galera db3 (proxmox6) | +| 10.0.30.81 | VM310 | MaxScale-01 Admin :8989 | +| 10.0.30.82 | VM311 | MaxScale-02 Standby | + +## Infrastructure VMs (10.0.30.x) +| IP | Host | Service | +|----|------|---------| +| 10.0.30.99 | CT111 | Immich (n5pro), Migration zu K8s geplant | +| 10.0.30.100 | CT134 | InfluxDB (Migration zu K8s) | +| 10.0.30.105 | — | ~~Gitea~~ DECOMMISSIONED (CT108, stopped) | +| 10.0.30.124 | VM200 | IaC Runner (DHCP), SSH `debian` | +| 10.0.30.141 | CT141 | Monitoring (Prometheus/Grafana) | +| 10.0.30.145 | CT145 | Voice Pipeline (PJSIP :5060) | + +## K8s LoadBalancers (10.0.30.2xx) +| IP | Service | Namespace | +|----|---------|-----------| +| 10.0.30.201 | Hindsight API :9177 | hindsight | +| 10.0.30.204 | InfluxDB :8086 | influxdb | +| 10.0.30.205 | Grafana | monitoring | +| 10.0.30.206 | Prometheus :9090 | monitoring | +| 10.0.30.207 | Loki :3100 | logging | + +## DMZ / Reverse Proxies (10.0.60.x) +| IP | Host | Service | +|----|------|---------| +| 10.0.60.10 | CT9999 | Traefik Reverse Proxy (root/[REDACTED]) | + +## Related +- [[concepts/network-architecture]] +- [[reference/ssh-keys]] +- [[reference/ports]] diff --git a/reference/ports.md b/reference/ports.md new file mode 100644 index 0000000..7c88942 --- /dev/null +++ b/reference/ports.md @@ -0,0 +1,38 @@ +--- +title: Ports (Quick Reference) +category: reference +tags: [ports, network, reference, quick-lookup] +created: "2026-07-24" +modified: "2026-07-24" +--- + +# Ports + +## Infrastructure Services +| Port | Service | Host | Notes | +|------|---------|------|-------| +| 22 | SSH (Gitea) | git.schoen.codes (LB 10.0.30.202) | Gitea SSH für ArgoCD | +| 2222 | SSH (Server) | diverse | Standard SSH Port im Homelab | +| 3000 | Gitea HTTP | git.schoen.codes (K8s Ingress) | CT108 decommissioned | +| 3100 | Loki | 10.0.30.207 | LB | +| 3306 | MariaDB (MaxScale VIP) | 10.0.30.70 | Galera HA | +| 5432 | PostgreSQL (CNPG) | postgres-main-rw.postgres.svc | K8s internal | +| 8000 | Paperless-ngx | paperless.svc | K8s internal | +| 8086 | InfluxDB | 10.0.30.204 | LB | +| 8989 | MaxScale Admin | 10.0.30.81 | admin:mariadb | +| 9090 | Prometheus | 10.0.30.206 | LB | +| 9093 | Alertmanager | K8s internal | | +| 9095 | HolmesGPT Adapter | K8s internal | Alertmgr → HolmesGPT | +| 9177 | Hindsight API | 10.0.30.201 | LB, NOT localhost | +| 9221 | PVE Exporter | 10.0.30.141 | | +| 9283 | Ceph Prometheus | 10.0.20.91 | active mgr | +| 9345 | RKE2 Server URL | 10.0.30.50 | cp-01 | + +## Voice Pipeline +| Port | Service | Host | +|------|---------|------| +| 5060 | PJSIP / Asterisk | CT145 | + +## Related +- [[reference/ip-map]] +- [[concepts/network-architecture]] diff --git a/reference/ssh-keys.md b/reference/ssh-keys.md new file mode 100644 index 0000000..dcb9c12 --- /dev/null +++ b/reference/ssh-keys.md @@ -0,0 +1,41 @@ +--- +title: SSH-Keys (Quick Reference) +category: reference +tags: [ssh, keys, reference, quick-lookup] +created: "2026-07-24" +modified: "2026-07-24" +--- + +# SSH-Keys + +## Key-Übersicht +| Key | Zweck | Fingerprint | Ziel-Hosts | +|-----|-------|--------------|------------| +| `id_ed25519_proxmox` | Proxmox Hosts (10.0.20.x) | — | 9 PVE Nodes | +| `id_ed25519_workers` | RKE2 Worker Nodes | `...kzBo` | 10.0.30.6x | +| `id_ed25519_cloudinit` | Cloud-init initial | `...JUYm` | VM200, Workers | +| `id_ed25519` | VM200 (debian user) | `...MmkM` | 10.0.30.124 | + +## SSH-Befehle +```bash +# Proxmox Hosts +ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.x + +# VM200 (IaC Runner) +ssh -i ~/.ssh/id_ed25519_proxmox debian@10.0.30.124 + +# K8s Worker (via VM200) +ssh -i /root/.ssh/id_ed25519_workers root@10.0.30.6x +``` + +## ArgoCD SSH Deploy Key +- Ed25519 Key als Gitea Deploy Key (read-only) +- ArgoCD Secret `argocd-repo-k8s-gitea` im argocd Namespace +- `known_hosts` ConfigMap mit Gitea SSH Hostkey für git.schoen.codes + +## CT9999 (Traefik DMZ) +- root / [REDACTED] (siehe 1Password) + +## Related +- [[reference/ip-map]] +- [[concepts/credential-policy]] diff --git a/systems/ceph-cluster.md b/systems/ceph-cluster.md new file mode 100644 index 0000000..abfdf56 --- /dev/null +++ b/systems/ceph-cluster.md @@ -0,0 +1,42 @@ +--- +title: Ceph Cluster +category: systems +tags: [ceph, storage, rbd, ec-pool, osd] +created: "2026-07-24" +modified: "2026-07-24" +--- + +# Ceph Cluster + +## Übersicht +- **14 OSDs** (HDD + NVMe/SSD混合) +- **Health:** HEALTH_WARN (BlueFS spillover osd.8, slow ops, nearfull) +- Läuft auf Proxmox PVE Hosts (integriert via pveceph) + +## Pools +| Pool | Type | Usage | +|------|------|-------| +| ceph-flash | replicated | K8s CSI (fast storage) | +| ceph-hdd | replicated | Bulk storage | +| media_ec | EC k=4+m=1 | Media (photos, backups) | + +## Bekannte Probleme +- **osd.0 NVMe** — 92% Wear, Critical Warning → Austausch planen +- **osd.2/5** — 93-94% full → entlasten oder neue OSDs +- **EC Pool k=4+m=1** — mit 5 OSDs kein Rebalance Headroom. Siehe Solution Doc: `docs/solutions/architecture/2026-07-12-ceph-ec-pool-no-rebalance-headroom.md` +- **BlueFS Spillover** osd.8 — `ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-8` + +## CRUSH Topology +- Device Classes: hdd, ssd, nvme +- EC Pools benötigen Device-Class-Separation (mixed disk sizes → Anti-Pattern: artificially low weights) + +## RBD Management +- Proxmox RBD Double-Mount Deadlock Pitfall: Niemals `pct mount` und `pct exec` gleichzeitig auf demselben Container +- Siehe Solution Doc: `docs/solutions/bug-fixes/2026-07-23-proxmox-rbd-double-mount-deadlock.md` + +## Related Skills +- `ceph-cluster-administration` (devops) + +## Related +- [[systems/proxmox-cluster]] +- [[systems/rke2-kubernetes]] (Ceph CSI) diff --git a/systems/ceph.md b/systems/ceph.md new file mode 100644 index 0000000..8d7261f --- /dev/null +++ b/systems/ceph.md @@ -0,0 +1,82 @@ +# Ceph Cluster + +## Overview +- **Cluster ID**: 204c8171-e0b1-4f40-9de2-a7cfe4ef68d9 +- **Health**: HEALTH_WARN (rebalancing after osd.7 reweight: 43 PGs backfilling at ~26 MiB/s) +- **Nodes**: 9 Proxmox hosts, 3 MONs (proxmox5/7/4), MGR on proxmox5 +- **OSDs**: 15 (8 SSD, 6 HDD, 1 new SSD on ms-a2-2), all up/in +- **Capacity**: 6.0 TiB used / 21 TiB total + +## OSD Layout + +| OSD | Class | Size | Host | Reweight | Notes | +|-----|-------|------|------|----------|-------| +| 0 | ssd | 188 GB | proxmox2 | 0.50 | | +| 1 | hdd | 3.7 TiB | n5pro | 1.0 | Large HDD | +| 2 | ssd | 233 GB | proxmox2 | 0.90 | Slow ops, 81% full | +| 3 | ssd | 238 GB | proxmox4 | 0.95 | | +| 4 | ssd | 233 GB | proxmox3 | 0.95 | | +| 5 | ssd | 238 GB | proxmox5 | 0.90 | 80% full | +| 6 | hdd | 2.8 TiB | ubuntu | 1.0 | Large HDD | +| 7 | hdd | 500 GB | proxmox7 | 1.0 | Was 0.80, reweighted 2026-07-24 | +| 8 | hdd | 2.8 TiB | ubuntu | 1.0 | BlueFS spillover | +| 9 | ssd | 1.9 TiB | n5pro | 1.0 | | +| 10 | hdd | 300 GB | proxmox6 | 1.0 | Very small HDD | +| 11 | hdd | 2.8 TiB | n5pro | 1.0 | Large HDD | +| 12 | ssd | 1.9 TiB | n5pro | 1.0 | | +| 13 | ssd | 1.8 TiB | ms-a2-1 | 1.0 | | +| 14 | ssd | 1.8 TiB | ms-a2-2 | 1.0 | New 2026-07-24, nvme0n1 | + +## Pools + +| Pool | Name | Type | Size | Min | CRUSH Rule | PGs | Notes | +|------|------|------|------|-----|------------|-----|-------| +| 1 | cephfs_data | replicated | 3 | 2 | 0 (any) | 32 | autoscale off | +| 2 | cephfs_metadata | replicated | 3 | 2 | 2 (ssd) | 32 | autoscale off | +| 3 | vm_disks | replicated | 3 | 2 | 2 (ssd) | 128 | autoscale on | +| 4 | .mgr | replicated | 3 | 2 | 2 (ssd) | 1 | | +| 5 | rbd | replicated | 3 | 2 | 1 (hdd) | 32 | autoscale on | +| 6 | hdd_disk | replicated | 3 | 2 | 1 (hdd) | 112 | nopgchange=true (was 120, equalized to 112) | +| 7 | tm_disks | replicated | 2 | 2 | 1 (hdd) | 128 | target_size 2TiB | +| 8 | media_ec | erasure 4+1 | 5 | 4 | 3 (hdd, osd-level) | 128 | ec_overwrites | +| 9 | media_meta | replicated | 3 | 2 | 0 (any) | 32 | | +| 10 | .rgw.root | replicated | 3 | 2 | 0 (any) | 1 | | + +## CRUSH Rules +- **Rule 0** (replicated_rule): default root, host-level placement +- **Rule 1** (replicated_hdd): default~hdd, host-level placement +- **Rule 2** (replicated_ssd): default~ssd, host-level placement +- **Rule 3** (media_ec): default~hdd, OSD-level placement (choose_indep) + +## Known Issues + +### Weight Imbalance Causing Placement Failures (2026-07-24) +HDD hosts have extreme weight disparity: n5pro=10.15TB, ubuntu=5.49TB, proxmox7=0.50TB, proxmox6=0.30TB. +CRUSH host-level selection (rule 1) often picks only 2 of 4 HDD hosts → up sets with 2 OSDs instead of 3. +Result: PGs stuck in `active+clean+remapped` because up set < min_size. + +**Mitigation (2026-07-24)**: +1. Reweighted osd.7 from 0.80 → 1.0 → fixed EC pool 8.3d (NONE → osd.7) +2. Equalized pool 6 pg_num 120 → 112 + nopgchange=true +3. Manual pg-upmap for stuck PGs: 5.13 → [1,6,7], 6.6c → [11,8,7], 6.58 → [11,6,7] +4. All `clean+remapped` eliminated. Triggered rebalancing wave (43 PGs backfilling at 26 MiB/s). + +**Long-term**: Small HDDs (osd.7 0.5TB, osd.10 0.3TB) cause CRUSH placement failures. Replace with larger disks or create separate CRUSH root for large HDDs only. + +### Pool 6 pg_num/pgp_num Mismatch (Fixed 2026-07-24) +Pool hdd_disk had pg_num=120, pgp_num=112 (autoscaler reducing to 32). +Equalized pg_num to 112. Set nopgchange=true to prevent further autoscaler interference. + +### BlueFS Spillover on osd.8 +osd.8 spilled 128KiB metadata from db device (2.1GiB of 30GiB) to slow device. +Cosmetic warning, no data risk. + +### Slow Operations on osd.2 and osd.7 +osd.2 (81% full, fragmentation 0.80) and osd.7 (small HDD) experience slow BlueStore ops. +osd.2 NVMe has 92% wear — candidate for replacement. + +## Access +- SSH to Proxmox hosts: `ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.92` +- Ceph commands: `ceph status`, `ceph osd tree`, `ceph pg dump pgs` +- Mon nodes: proxmox5, proxmox7, proxmox4 +- Mgr: proxmox5 (active) diff --git a/systems/galera-maxscale.md b/systems/galera-maxscale.md new file mode 100644 index 0000000..758ed49 --- /dev/null +++ b/systems/galera-maxscale.md @@ -0,0 +1,43 @@ +--- +title: MariaDB Galera + MaxScale +category: systems +tags: [mariadb, galera, maxscale, database, ha] +created: "2026-07-24" +modified: "2026-07-24" +--- + +# MariaDB Galera + MaxScale + +## Galera Cluster (3/3 Synced) +| Node | VM | IP | Role | +|------|----|----|------| +| db1 | VM300 (ms-a2-1) | 10.0.30.71 | Slave | +| db2 | VM301 (proxmox3) | 10.0.30.72 | Slave | +| db3 | VM302 (proxmox6) | 10.0.30.73 | Master | + +- **Root Credentials:** `/root/.my.cnf` auf VM300 (via `qm guest exec`) +- **1Password `mariadb-root` Item:** PASSWORT IST FALSCH — stimmt nicht mit `/root/.my.cnf` überein + +## MaxScale Proxy (HA) +| Node | VM | IP | Port | +|------|----|----|------| +| MaxScale-01 | VM310 (proxmox1) | 10.0.30.81 | Admin :8989 | +| MaxScale-02 | VM311 (ms-a2-1) | 10.0.30.82 | Standby | +| **VIP** | — | **10.0.30.70** | :3306 (keepalived) | + +- **Admin:** `admin:mariadb` auf :8989 + +## VM Konfiguration +- VM300: `balloon: 0` (disabled) — verhindert InnoDB Buffer Pool Swapping +- Standard Practice für DB VMs + +## Fluent Bit +- VM300/301/302 (Galera): Fluent Bit aktiv +- VM310/311 (MaxScale): Fluent Bit aktiv + +## Related Skills +- `mariadb-galera-cluster-administration` (devops) + +## Related +- [[systems/proxmox-cluster]] +- [[reference/ip-map]] diff --git a/systems/gitea.md b/systems/gitea.md new file mode 100644 index 0000000..65e52de --- /dev/null +++ b/systems/gitea.md @@ -0,0 +1,49 @@ +--- +title: Gitea (Git Server + CI) +category: systems +tags: [gitea, git, ci, actions] +created: "2026-07-24" +modified: "2026-07-24" +--- + +# Gitea (Git Server + CI) + +## Instanz +- **URL:** git.schoen.codes (K8s Ingress, Traefik → gitea-http ClusterIP) +- **Internal:** gitea-internal.gitea:3000 (K8s ClusterIP 10.43.5.203) +- **SSH:** 10.0.30.202:22 (Cilium LoadBalancer, gitea-ssh service) +- **Version:** 1.27.0 (K8s Helm chart v12.7.0) +- **Privat:** NIEMALS öffentlich machen (enthält Secrets) +- **CT108 (alte Gitea, 10.0.30.105): DECOMMISSIONED** — gestoppt am 2026-07-24 + +## Repositories +| Repo | Zweck | Clone | +|------|-------|-------| +| dominik/iac-homelab | IaC (Tofu + Ansible + K8s Manifeste) | `/home/debian/iac-homelab` auf Hermes host | +| dominik/hermes-skills | Skill Versionierung | `~/.hermes/skills/` | +| dominik/hermes-memories | MEMORY.md/USER.md Sync | `~/.hermes/memories/` | +| dominik/noris-compound-learning | Noris Learnings (shareable) | `/home/debian/infra-compound-learning/` | + +## ArgoCD SSH Deploy Keys +- Ed25519 SSH-Key als Gitea Deploy Key (read-only) +- ArgoCD Secret `argocd-repo-k8s-gitea` mit SSH Private Key +- `known_hosts` ConfigMap mit Gitea SSH Hostkey +- 15 ArgoCD Apps via SSH (`ssh://gitea@git.schoen.codes:22/dominik/iac-homelab.git`) + +## Gitea Actions CI +- **Primary Runner:** vm200-host (ID 73, act_runner v0.2.13, host backend) — WORKING +- **K8s Runner:** gitea-runner deployment in gitea namespace (DinD sidecar + act_runner v0.2.13) — registered but NO TASK ASSIGNMENT (Gitea 1.27.0 bug) +- **KNOWN BUG (Gitea 1.27.0):** `actions_ready_job` queue doesn't create `action_task` records for K8s-based runners via gRPC Declare. Only runner 73 (external, pre-registered) receives tasks. Fix: upgrade Gitea to 1.27.1+ or 1.28.x. +- **No runner management API** in Gitea 1.27.0 (`/api/v1/admin/runners` → 404). Manage via DB: `gitea.action_runner` table, `is_disabled` column, `deleted` for soft-delete. +- Workflow: `.gitea/workflows/rebuild-infrastructure.yml` +- Achtung: `actions/checkout@v4` versucht github.com zu erreichen — muss gemapped werden +- **DinD Pitfalls:** docker:27-dind creates docker.sock as directory → use TCP mode (--host=tcp://, --tls=false). act_runner:0.2.13 has no docker CLI → use nc for port check. act_runner:0.2.13 doesn't auto-register → wrapper script needed. +- **Solution Doc:** `docs/solutions/bug-fixes/2026-07-24-gitea-actions-k8s-runner-task-assignment-bug.md` + +## IaC Remotes +- iac-homelab hat EINEN remote: `origin` = K8s Gitea (git.schoen.codes) +- Alter CT108 remote wurde entfernt (CT108 decommissioned) + +## Related +- [[systems/rke2-kubernetes]] +- [[concepts/gitops-workflow]] diff --git a/systems/hindsight.md b/systems/hindsight.md new file mode 100644 index 0000000..30cbc3e --- /dev/null +++ b/systems/hindsight.md @@ -0,0 +1,49 @@ +--- +title: Hindsight (Semantic Memory) +category: systems +tags: [hindsight, memory, postgresql, embeddings, semantic-search] +created: "2026-07-24" +modified: "2026-07-24" +--- + +# Hindsight (Semantic Memory) + +## Deployment +- **Namespace:** hindsight (K8s) +- **API:** LoadBalancer `10.0.30.201:9177` (**NICHT localhost**) +- **Health:** `curl -s http://10.0.30.201:9177/health` +- **Backend:** PostgreSQL + pgvector +- **Config:** `~/.hermes/hindsight/config.json` (api_url gesetzt) + +## Components +| Component | Status | +|-----------|--------| +| hindsight-api | 1/1 Running | +| hindsight-postgres | 1/1 Running | +| Embedding Model | harrier-oss (via noris) | + +## API Endpoints +- `GET /health` — Health check +- `POST /retain` — Store memory +- `POST /recall` — Semantic search +- `POST /reflect` — Synthesize answer across memories + +## Best Practices +- **retain_every_n_turns: 2** — reduziert Duplikate +- Vor Batch-Retain: Health-Check, dann erst retain calls feuern +- Meta-Memory Cleanup Cron: 1st of Month 03:00 (3 Dedup Rounds, Archive >6 Monate) +- Nightly Dream Cycle + Weekly Insight Digest + +## What NOT to store in Hindsight +- Temporäre Task-States (→ session_search / todo) +- Workflows (→ Skills) +- Credentials (→ 1Password) +- Duplikate von MEMORY.md + +## Related Skills +- `hindsight` (hindsight category) +- `compound-learning` (software-development category) + +## Related +- [[systems/rke2-kubernetes]] +- [[concepts/credential-policy]] diff --git a/systems/loki-fluentbit.md b/systems/loki-fluentbit.md new file mode 100644 index 0000000..a155ab3 --- /dev/null +++ b/systems/loki-fluentbit.md @@ -0,0 +1,47 @@ +--- +title: Loki + Fluent Bit (Logging Stack) +category: systems +tags: [loki, fluent-bit, logging, observability] +created: "2026-07-24" +modified: "2026-07-24" +--- + +# Loki + Fluent Bit (Logging Stack) + +## Loki +- **Namespace:** logging (K8s) +- **External Service:** `loki-external`, LoadBalancer `10.0.30.207:3100` +- **Caches:** memcached (results-cache, `-m 512 -I 2m`, requests 256Mi/limits 512Mi) + +## Fluent Bit v5.0.9 + +### Coverage +| Target | How | Status | +|--------|-----|--------| +| 9 PVE Hosts | `install-host.sh` auf VM200 | ✅ aktiv | +| 20 LXC Containers | `install-lxc.sh` | ✅ aktiv (außer CT110 stopped) | +| 5 QEMU VMs | `qm guest exec` | ✅ aktiv (VM300/301/302/310/311) | +| 6 K8s Nodes | K8s DaemonSet | ✅ aktiv | +| VM200 | bereits vorhanden | ✅ aktiv | + +### Inputs (Proxmox Hosts) +- systemd journal (pve*, corosync, pacemaker, ceph, zfs, smartd) +- auth.log +- pveproxy/access.log +- pvedaemon.log +- cluster.log +- **PVE Tasks** (`/var/log/pve/tasks/index`) — UPID-Format +- **Ceph Audit** (`/var/log/ceph/ceph.audit.log`) — JSON + +### Scripts +- `logging/fluent-bit-proxmox/` im iac-homelab Repo + +## Query +```bash +logcli query --addr http://10.0.30.207:3100 'job=fluent-bit-proxmox' +``` + +## Related +- [[systems/proxmox-cluster]] +- [[systems/monitoring]] +- [[systems/rke2-kubernetes]] diff --git a/systems/monitoring.md b/systems/monitoring.md new file mode 100644 index 0000000..170f61b --- /dev/null +++ b/systems/monitoring.md @@ -0,0 +1,34 @@ +--- +title: Monitoring & Observability +category: systems +tags: [monitoring, prometheus, grafana, loki, holmesgpt, alerting] +created: "2026-04-28" +modified: "2026-07-24" +--- + +# Monitoring & Observability + +## Prometheus + Grafana +- **Prometheus**: K8s (kube-prometheus-stack) +- **Grafana**: K8s, Ingress grafana.familie-schoen.com (.205) +- **Alertmanager**: K8s, Routes → Telegram + +## Loki Stack (Centralized Logging) +- **Loki**: K8s ns=logging, LB 10.0.30.207:3100 +- **Fluent Bit v5.0.9**: auf 9 PVE Hosts + 20 LXC + 5 QEMU + 6 K8s Nodes +- **Inputs**: PVE tasks, Ceph audit, journald, auth, pveproxy, pvedaemon, cluster +- Siehe [[systems/loki-fluentbit]] + +## HolmesGPT (SRE) +- **Namespace**: holmes +- **Pipeline**: Alertmanager → adapter (9095) → HolmesGPT → TG @schoen_sre_bot +- **Model**: noris-glm (via ai.noris.de) +- RKE2 ctrl-plane alerts disabled + +## Alerting Rules +- Via IaC Git Repo deployt (Ansible für CT141, ArgoCD für K8s) +- RKE2 control-plane alerts disabled + +## Related +- [[systems/loki-fluentbit]] +- [[systems/rke2-kubernetes]] diff --git a/systems/proxmox-cluster.md b/systems/proxmox-cluster.md new file mode 100644 index 0000000..92d1a1c --- /dev/null +++ b/systems/proxmox-cluster.md @@ -0,0 +1,64 @@ +--- +title: Proxmox VE Cluster +category: systems +tags: [proxmox, virtualization, lxc, qemu, pve] +created: "2026-04-28" +modified: "2026-07-24" +--- + +# Proxmox VE Cluster + +## Cluster-Konfiguration +- **Version:** PVE 9.2.3 +- **Nodes:** 9 (Quorum OK) +- **Hypervisoren:** 10.0.20.x +- **Guests:** ~30 LXC + ~10 QEMU VMs + +## Storage +- **vm_disks:** Primärer Storage für alle VMs/CTs (SSD) +- **hdd_templates:** CT Templates +- **Ceph RBD:** ceph-flash, ceph-hdd Pools (über K8s CSI) + +## Netzwerk +- VLAN-basiert, Bridge vmbr0 +- IP-Schema: 10.0.X.Y — siehe [[concepts/network-architecture]] + +## Fluent Bit (Logging) +- Alle 9 Hosts haben Fluent Bit aktiv +- Inputs: systemd journal (pve*, corosync, pacemaker, ceph, zfs, smartd), auth.log, pveproxy/access.log, pvedaemon.log, cluster.log +- **PVE Tasks** (`/var/log/pve/tasks/index`): UPID-Format (Node, PID, Task-Type, VMID, User, Status) +- **Ceph Audit** (`/var/log/ceph/ceph.audit.log`): OSD/Pool/RBD Operationen +- Output → Loki (10.0.30.207:3100) +- Siehe [[systems/loki-fluentbit]] + +## Wichtige Befehle +```bash +pvecm status # Cluster-Quorum +pct status # Container-Status +pct start/stop # Container starten/stoppen +qm status # VM-Status +pvesh get /cluster/resources --type vm # Alle VMs/CTs +``` + +## SSH-Zugriff +- Key: `id_ed25519_proxmox` (funktioniert für 10.0.20.x Hosts) +- Siehe [[reference/ssh-keys]] + +## GPU Passthrough (AMD 1002:13c0) +- **ms-a2-1** (10.0.20.92): VFIO config gefixt 2026-07-24 — siehe [Solution Doc](../../../docs/solutions/bug-fixes/2026-07-24-vfio-pci-module-loading-race-condition.md) +- **ms-a2-2** (10.0.20.93): Funktioniert seit Initialisierung +- Benötigte modprobe.d Config: + - `blacklist amdgpu` + `blacklist drm` + `blacklist drm_kms_helper` + - `options vfio-pci ids=1002:13c0` + `softdep amdgpu pre: vfio-pci` +- Worker-05 (VM 102) läuft auf ms-a2-2 mit funktionierendem GPU-Passthrough + +## Bekannte Probleme +- osd.0 NVMe 92% Wear — Austausch planen +- osd.2/5 nearfull (93-94%) — entlasten +- CT110 kaputte libc — Reparatur ausstehend +- ms-a2-1 GPU-Passthrough: Config gefixt, Reboot zur Verifikation ausstehend + +## Related +- [[systems/ceph-cluster]] +- [[systems/rke2-kubernetes]] +- [[reference/ip-map]] diff --git a/systems/rke2-kubernetes.md b/systems/rke2-kubernetes.md new file mode 100644 index 0000000..98a098e --- /dev/null +++ b/systems/rke2-kubernetes.md @@ -0,0 +1,79 @@ +--- +title: RKE2 Kubernetes Cluster +category: systems +tags: [kubernetes, rke2, cilium, argocd, cnpg, gitops] +created: "2026-07-24" +modified: "2026-07-24" +--- + +# RKE2 Kubernetes Cluster + +## Übersicht +- **Version:** v1.35.6-rke2r1 +- **Nodes:** 6 (3 CP + 3 Worker), alle schedulable (keine CP Taints) +- **CNI:** Cilium (LB IP Pool) +- **Ingress:**两层 Traefik (extern CT9999 + intern K8s) + Authelia SSO + +## Node IPs +| Node | IP | Role | +|------|----|------| +| cp-01 | 10.0.30.51 | Control Plane | +| cp-02 | 10.0.30.52 | Control Plane | +| cp-03 | 10.0.30.53 | Control Plane | +| worker-01 | 10.0.30.63 | Worker | +| worker-04 | 10.0.30.64 | Worker | +| worker-05 | 10.0.30.65 | Worker (GPU renderD128 ✅ on ms-a2-2) | + +## Storage +- **Ceph CSI**: ceph-flash (fast), ceph-hdd (bulk) +- **CNPG PostgreSQL**: `postgres-main` Cluster (3/3 Ready), RW Service `postgres-main-rw.postgres.svc.cluster.local:5432` + +## GitOps +- **ArgoCD**: SSH Deploy Keys (read-only) auf Gitea +- 15 Applications via SSH (`ssh://git@10.0.30.202:22/dominik/iac-homelab.git`) +- **Gitea SSH user is `git`, not `gitea`** — `git.schoen.codes` resolves to Traefik (10.0.30.203, HTTP only), SSH is on 10.0.30.202:22 +- Workflow: IaC repo auschecken → ändern → commit+push → ArgoCD sync → verify → lokale Kopie löschen +- Siehe [[concepts/gitops-workflow]] + +## Backup +- **Velero** v1.18.1 with `--features=EnableCSI`, S3 backend (noris cloud `homelab-velero-backups`) +- **VolumeSnapshotClass** `ceph-rbd-snapclass` (rbd.csi.ceph.com, default) +- Schedules: daily 01:00 UTC (30d TTL), weekly Sun 00:00 UTC (90d TTL) +- **CNPG Postgres**: separate hourly backups via barmanObjectStore to S3 (30d retention) +- All 24 PVCs (~430 GiB) now have off-site CSI snapshots +- **Helm pitfall**: `features: EnableCSI` must be under `configuration:` in Velero Helm values (not top-level) + +## Deployed Workloads +| Workload | Namespace | Notes | +|----------|-----------|-------| +| Paperless-ngx v3.0.0 | paperless | PostgreSQL (CNPG), `enableServiceLinks: false` | +| Gitea | gitea | git.schoen.codes | +| Loki Stack | logging | LB 10.0.30.207:3100 | +| Hindsight | hindsight | LB 10.0.30.201:9177 | +| Authelia | authelia | SSO, Galera backend | +| HolmesGPT | holmes | SRE Bot | +| Traefik | traefik | Ingress Controller | +| CNPG | postgres | PostgreSQL HA | + +## GPU +- worker-04: renderD128 ✅ (AMD GPU Passthrough, rombar=1, on n5pro) +- worker-05: renderD128 ✅ (AMD GPU Passthrough, rombar=1, on ms-a2-2) +- IaC: unified `amd-gpu` PCI mapping (n5pro + ms-a2-1 + ms-a2-2) + dynamic hostpci (one worker block, gpu flag) +- ms-a2-1 has kernel BUG with 1002:13c0 (renderD128 missing) — worker-05 moved to ms-a2-2 (identical hardware) + +## Known Pitfalls +- `enableServiceLinks: false` bei Apps deren Service-Name mit Env-Vars kollidiert (z.B. Paperless `PAPERLESS_PORT`) +- ArgoCD `--force` kann nicht mit ServerSideApply kombiniert werden +- CoreDNS: `hosts` configBlock nicht params +- etcd Defrag monatlich (CronJob) + +## Related Skills +- `rke2-cluster-administration` (devops) + +## Related +- [[systems/ceph-cluster]] +- [[systems/gitea]] +- [[systems/hindsight]] +- [[systems/loki-fluentbit]] +- [[concepts/gitops-workflow]] +- [[reference/ip-map]] diff --git a/Entities/cloud.familie-schoen.com.md b/systems/seafile.md similarity index 100% rename from Entities/cloud.familie-schoen.com.md rename to systems/seafile.md diff --git a/team-structure.md b/team-structure.md deleted file mode 100644 index fe9b15b..0000000 --- a/team-structure.md +++ /dev/null @@ -1,12 +0,0 @@ -# Team Structure (Schön Consulting) - -## Model -CEO + Agents model. Two main profiles: -- **infra-sre** - Security/Monitoring -- **nutrition-coach** - Family Meals - -## Operations -- CEO orchestrates, agents operate -- Each agent has their own chat -- Cronjobs deliver directly to agent chats (not to CEO) -- Shared Telegram group for entire team