Auto-sync: 2026-07-24
This commit is contained in:
@@ -0,0 +1,42 @@
|
||||
---
|
||||
title: Ceph Cluster
|
||||
category: systems
|
||||
tags: [ceph, storage, rbd, ec-pool, osd]
|
||||
created: "2026-07-24"
|
||||
modified: "2026-07-24"
|
||||
---
|
||||
|
||||
# Ceph Cluster
|
||||
|
||||
## Übersicht
|
||||
- **14 OSDs** (HDD + NVMe/SSD混合)
|
||||
- **Health:** HEALTH_WARN (BlueFS spillover osd.8, slow ops, nearfull)
|
||||
- Läuft auf Proxmox PVE Hosts (integriert via pveceph)
|
||||
|
||||
## Pools
|
||||
| Pool | Type | Usage |
|
||||
|------|------|-------|
|
||||
| ceph-flash | replicated | K8s CSI (fast storage) |
|
||||
| ceph-hdd | replicated | Bulk storage |
|
||||
| media_ec | EC k=4+m=1 | Media (photos, backups) |
|
||||
|
||||
## Bekannte Probleme
|
||||
- **osd.0 NVMe** — 92% Wear, Critical Warning → Austausch planen
|
||||
- **osd.2/5** — 93-94% full → entlasten oder neue OSDs
|
||||
- **EC Pool k=4+m=1** — mit 5 OSDs kein Rebalance Headroom. Siehe Solution Doc: `docs/solutions/architecture/2026-07-12-ceph-ec-pool-no-rebalance-headroom.md`
|
||||
- **BlueFS Spillover** osd.8 — `ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-8`
|
||||
|
||||
## CRUSH Topology
|
||||
- Device Classes: hdd, ssd, nvme
|
||||
- EC Pools benötigen Device-Class-Separation (mixed disk sizes → Anti-Pattern: artificially low weights)
|
||||
|
||||
## RBD Management
|
||||
- Proxmox RBD Double-Mount Deadlock Pitfall: Niemals `pct mount` und `pct exec` gleichzeitig auf demselben Container
|
||||
- Siehe Solution Doc: `docs/solutions/bug-fixes/2026-07-23-proxmox-rbd-double-mount-deadlock.md`
|
||||
|
||||
## Related Skills
|
||||
- `ceph-cluster-administration` (devops)
|
||||
|
||||
## Related
|
||||
- [[systems/proxmox-cluster]]
|
||||
- [[systems/rke2-kubernetes]] (Ceph CSI)
|
||||
@@ -0,0 +1,82 @@
|
||||
# Ceph Cluster
|
||||
|
||||
## Overview
|
||||
- **Cluster ID**: 204c8171-e0b1-4f40-9de2-a7cfe4ef68d9
|
||||
- **Health**: HEALTH_WARN (rebalancing after osd.7 reweight: 43 PGs backfilling at ~26 MiB/s)
|
||||
- **Nodes**: 9 Proxmox hosts, 3 MONs (proxmox5/7/4), MGR on proxmox5
|
||||
- **OSDs**: 15 (8 SSD, 6 HDD, 1 new SSD on ms-a2-2), all up/in
|
||||
- **Capacity**: 6.0 TiB used / 21 TiB total
|
||||
|
||||
## OSD Layout
|
||||
|
||||
| OSD | Class | Size | Host | Reweight | Notes |
|
||||
|-----|-------|------|------|----------|-------|
|
||||
| 0 | ssd | 188 GB | proxmox2 | 0.50 | |
|
||||
| 1 | hdd | 3.7 TiB | n5pro | 1.0 | Large HDD |
|
||||
| 2 | ssd | 233 GB | proxmox2 | 0.90 | Slow ops, 81% full |
|
||||
| 3 | ssd | 238 GB | proxmox4 | 0.95 | |
|
||||
| 4 | ssd | 233 GB | proxmox3 | 0.95 | |
|
||||
| 5 | ssd | 238 GB | proxmox5 | 0.90 | 80% full |
|
||||
| 6 | hdd | 2.8 TiB | ubuntu | 1.0 | Large HDD |
|
||||
| 7 | hdd | 500 GB | proxmox7 | 1.0 | Was 0.80, reweighted 2026-07-24 |
|
||||
| 8 | hdd | 2.8 TiB | ubuntu | 1.0 | BlueFS spillover |
|
||||
| 9 | ssd | 1.9 TiB | n5pro | 1.0 | |
|
||||
| 10 | hdd | 300 GB | proxmox6 | 1.0 | Very small HDD |
|
||||
| 11 | hdd | 2.8 TiB | n5pro | 1.0 | Large HDD |
|
||||
| 12 | ssd | 1.9 TiB | n5pro | 1.0 | |
|
||||
| 13 | ssd | 1.8 TiB | ms-a2-1 | 1.0 | |
|
||||
| 14 | ssd | 1.8 TiB | ms-a2-2 | 1.0 | New 2026-07-24, nvme0n1 |
|
||||
|
||||
## Pools
|
||||
|
||||
| Pool | Name | Type | Size | Min | CRUSH Rule | PGs | Notes |
|
||||
|------|------|------|------|-----|------------|-----|-------|
|
||||
| 1 | cephfs_data | replicated | 3 | 2 | 0 (any) | 32 | autoscale off |
|
||||
| 2 | cephfs_metadata | replicated | 3 | 2 | 2 (ssd) | 32 | autoscale off |
|
||||
| 3 | vm_disks | replicated | 3 | 2 | 2 (ssd) | 128 | autoscale on |
|
||||
| 4 | .mgr | replicated | 3 | 2 | 2 (ssd) | 1 | |
|
||||
| 5 | rbd | replicated | 3 | 2 | 1 (hdd) | 32 | autoscale on |
|
||||
| 6 | hdd_disk | replicated | 3 | 2 | 1 (hdd) | 112 | nopgchange=true (was 120, equalized to 112) |
|
||||
| 7 | tm_disks | replicated | 2 | 2 | 1 (hdd) | 128 | target_size 2TiB |
|
||||
| 8 | media_ec | erasure 4+1 | 5 | 4 | 3 (hdd, osd-level) | 128 | ec_overwrites |
|
||||
| 9 | media_meta | replicated | 3 | 2 | 0 (any) | 32 | |
|
||||
| 10 | .rgw.root | replicated | 3 | 2 | 0 (any) | 1 | |
|
||||
|
||||
## CRUSH Rules
|
||||
- **Rule 0** (replicated_rule): default root, host-level placement
|
||||
- **Rule 1** (replicated_hdd): default~hdd, host-level placement
|
||||
- **Rule 2** (replicated_ssd): default~ssd, host-level placement
|
||||
- **Rule 3** (media_ec): default~hdd, OSD-level placement (choose_indep)
|
||||
|
||||
## Known Issues
|
||||
|
||||
### Weight Imbalance Causing Placement Failures (2026-07-24)
|
||||
HDD hosts have extreme weight disparity: n5pro=10.15TB, ubuntu=5.49TB, proxmox7=0.50TB, proxmox6=0.30TB.
|
||||
CRUSH host-level selection (rule 1) often picks only 2 of 4 HDD hosts → up sets with 2 OSDs instead of 3.
|
||||
Result: PGs stuck in `active+clean+remapped` because up set < min_size.
|
||||
|
||||
**Mitigation (2026-07-24)**:
|
||||
1. Reweighted osd.7 from 0.80 → 1.0 → fixed EC pool 8.3d (NONE → osd.7)
|
||||
2. Equalized pool 6 pg_num 120 → 112 + nopgchange=true
|
||||
3. Manual pg-upmap for stuck PGs: 5.13 → [1,6,7], 6.6c → [11,8,7], 6.58 → [11,6,7]
|
||||
4. All `clean+remapped` eliminated. Triggered rebalancing wave (43 PGs backfilling at 26 MiB/s).
|
||||
|
||||
**Long-term**: Small HDDs (osd.7 0.5TB, osd.10 0.3TB) cause CRUSH placement failures. Replace with larger disks or create separate CRUSH root for large HDDs only.
|
||||
|
||||
### Pool 6 pg_num/pgp_num Mismatch (Fixed 2026-07-24)
|
||||
Pool hdd_disk had pg_num=120, pgp_num=112 (autoscaler reducing to 32).
|
||||
Equalized pg_num to 112. Set nopgchange=true to prevent further autoscaler interference.
|
||||
|
||||
### BlueFS Spillover on osd.8
|
||||
osd.8 spilled 128KiB metadata from db device (2.1GiB of 30GiB) to slow device.
|
||||
Cosmetic warning, no data risk.
|
||||
|
||||
### Slow Operations on osd.2 and osd.7
|
||||
osd.2 (81% full, fragmentation 0.80) and osd.7 (small HDD) experience slow BlueStore ops.
|
||||
osd.2 NVMe has 92% wear — candidate for replacement.
|
||||
|
||||
## Access
|
||||
- SSH to Proxmox hosts: `ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.92`
|
||||
- Ceph commands: `ceph status`, `ceph osd tree`, `ceph pg dump pgs`
|
||||
- Mon nodes: proxmox5, proxmox7, proxmox4
|
||||
- Mgr: proxmox5 (active)
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
title: MariaDB Galera + MaxScale
|
||||
category: systems
|
||||
tags: [mariadb, galera, maxscale, database, ha]
|
||||
created: "2026-07-24"
|
||||
modified: "2026-07-24"
|
||||
---
|
||||
|
||||
# MariaDB Galera + MaxScale
|
||||
|
||||
## Galera Cluster (3/3 Synced)
|
||||
| Node | VM | IP | Role |
|
||||
|------|----|----|------|
|
||||
| db1 | VM300 (ms-a2-1) | 10.0.30.71 | Slave |
|
||||
| db2 | VM301 (proxmox3) | 10.0.30.72 | Slave |
|
||||
| db3 | VM302 (proxmox6) | 10.0.30.73 | Master |
|
||||
|
||||
- **Root Credentials:** `/root/.my.cnf` auf VM300 (via `qm guest exec`)
|
||||
- **1Password `mariadb-root` Item:** PASSWORT IST FALSCH — stimmt nicht mit `/root/.my.cnf` überein
|
||||
|
||||
## MaxScale Proxy (HA)
|
||||
| Node | VM | IP | Port |
|
||||
|------|----|----|------|
|
||||
| MaxScale-01 | VM310 (proxmox1) | 10.0.30.81 | Admin :8989 |
|
||||
| MaxScale-02 | VM311 (ms-a2-1) | 10.0.30.82 | Standby |
|
||||
| **VIP** | — | **10.0.30.70** | :3306 (keepalived) |
|
||||
|
||||
- **Admin:** `admin:mariadb` auf :8989
|
||||
|
||||
## VM Konfiguration
|
||||
- VM300: `balloon: 0` (disabled) — verhindert InnoDB Buffer Pool Swapping
|
||||
- Standard Practice für DB VMs
|
||||
|
||||
## Fluent Bit
|
||||
- VM300/301/302 (Galera): Fluent Bit aktiv
|
||||
- VM310/311 (MaxScale): Fluent Bit aktiv
|
||||
|
||||
## Related Skills
|
||||
- `mariadb-galera-cluster-administration` (devops)
|
||||
|
||||
## Related
|
||||
- [[systems/proxmox-cluster]]
|
||||
- [[reference/ip-map]]
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
title: Gitea (Git Server + CI)
|
||||
category: systems
|
||||
tags: [gitea, git, ci, actions]
|
||||
created: "2026-07-24"
|
||||
modified: "2026-07-24"
|
||||
---
|
||||
|
||||
# Gitea (Git Server + CI)
|
||||
|
||||
## Instanz
|
||||
- **URL:** git.schoen.codes (K8s Ingress, Traefik → gitea-http ClusterIP)
|
||||
- **Internal:** gitea-internal.gitea:3000 (K8s ClusterIP 10.43.5.203)
|
||||
- **SSH:** 10.0.30.202:22 (Cilium LoadBalancer, gitea-ssh service)
|
||||
- **Version:** 1.27.0 (K8s Helm chart v12.7.0)
|
||||
- **Privat:** NIEMALS öffentlich machen (enthält Secrets)
|
||||
- **CT108 (alte Gitea, 10.0.30.105): DECOMMISSIONED** — gestoppt am 2026-07-24
|
||||
|
||||
## Repositories
|
||||
| Repo | Zweck | Clone |
|
||||
|------|-------|-------|
|
||||
| dominik/iac-homelab | IaC (Tofu + Ansible + K8s Manifeste) | `/home/debian/iac-homelab` auf Hermes host |
|
||||
| dominik/hermes-skills | Skill Versionierung | `~/.hermes/skills/` |
|
||||
| dominik/hermes-memories | MEMORY.md/USER.md Sync | `~/.hermes/memories/` |
|
||||
| dominik/noris-compound-learning | Noris Learnings (shareable) | `/home/debian/infra-compound-learning/` |
|
||||
|
||||
## ArgoCD SSH Deploy Keys
|
||||
- Ed25519 SSH-Key als Gitea Deploy Key (read-only)
|
||||
- ArgoCD Secret `argocd-repo-k8s-gitea` mit SSH Private Key
|
||||
- `known_hosts` ConfigMap mit Gitea SSH Hostkey
|
||||
- 15 ArgoCD Apps via SSH (`ssh://gitea@git.schoen.codes:22/dominik/iac-homelab.git`)
|
||||
|
||||
## Gitea Actions CI
|
||||
- **Primary Runner:** vm200-host (ID 73, act_runner v0.2.13, host backend) — WORKING
|
||||
- **K8s Runner:** gitea-runner deployment in gitea namespace (DinD sidecar + act_runner v0.2.13) — registered but NO TASK ASSIGNMENT (Gitea 1.27.0 bug)
|
||||
- **KNOWN BUG (Gitea 1.27.0):** `actions_ready_job` queue doesn't create `action_task` records for K8s-based runners via gRPC Declare. Only runner 73 (external, pre-registered) receives tasks. Fix: upgrade Gitea to 1.27.1+ or 1.28.x.
|
||||
- **No runner management API** in Gitea 1.27.0 (`/api/v1/admin/runners` → 404). Manage via DB: `gitea.action_runner` table, `is_disabled` column, `deleted` for soft-delete.
|
||||
- Workflow: `.gitea/workflows/rebuild-infrastructure.yml`
|
||||
- Achtung: `actions/checkout@v4` versucht github.com zu erreichen — muss gemapped werden
|
||||
- **DinD Pitfalls:** docker:27-dind creates docker.sock as directory → use TCP mode (--host=tcp://, --tls=false). act_runner:0.2.13 has no docker CLI → use nc for port check. act_runner:0.2.13 doesn't auto-register → wrapper script needed.
|
||||
- **Solution Doc:** `docs/solutions/bug-fixes/2026-07-24-gitea-actions-k8s-runner-task-assignment-bug.md`
|
||||
|
||||
## IaC Remotes
|
||||
- iac-homelab hat EINEN remote: `origin` = K8s Gitea (git.schoen.codes)
|
||||
- Alter CT108 remote wurde entfernt (CT108 decommissioned)
|
||||
|
||||
## Related
|
||||
- [[systems/rke2-kubernetes]]
|
||||
- [[concepts/gitops-workflow]]
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
title: Hindsight (Semantic Memory)
|
||||
category: systems
|
||||
tags: [hindsight, memory, postgresql, embeddings, semantic-search]
|
||||
created: "2026-07-24"
|
||||
modified: "2026-07-24"
|
||||
---
|
||||
|
||||
# Hindsight (Semantic Memory)
|
||||
|
||||
## Deployment
|
||||
- **Namespace:** hindsight (K8s)
|
||||
- **API:** LoadBalancer `10.0.30.201:9177` (**NICHT localhost**)
|
||||
- **Health:** `curl -s http://10.0.30.201:9177/health`
|
||||
- **Backend:** PostgreSQL + pgvector
|
||||
- **Config:** `~/.hermes/hindsight/config.json` (api_url gesetzt)
|
||||
|
||||
## Components
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| hindsight-api | 1/1 Running |
|
||||
| hindsight-postgres | 1/1 Running |
|
||||
| Embedding Model | harrier-oss (via noris) |
|
||||
|
||||
## API Endpoints
|
||||
- `GET /health` — Health check
|
||||
- `POST /retain` — Store memory
|
||||
- `POST /recall` — Semantic search
|
||||
- `POST /reflect` — Synthesize answer across memories
|
||||
|
||||
## Best Practices
|
||||
- **retain_every_n_turns: 2** — reduziert Duplikate
|
||||
- Vor Batch-Retain: Health-Check, dann erst retain calls feuern
|
||||
- Meta-Memory Cleanup Cron: 1st of Month 03:00 (3 Dedup Rounds, Archive >6 Monate)
|
||||
- Nightly Dream Cycle + Weekly Insight Digest
|
||||
|
||||
## What NOT to store in Hindsight
|
||||
- Temporäre Task-States (→ session_search / todo)
|
||||
- Workflows (→ Skills)
|
||||
- Credentials (→ 1Password)
|
||||
- Duplikate von MEMORY.md
|
||||
|
||||
## Related Skills
|
||||
- `hindsight` (hindsight category)
|
||||
- `compound-learning` (software-development category)
|
||||
|
||||
## Related
|
||||
- [[systems/rke2-kubernetes]]
|
||||
- [[concepts/credential-policy]]
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: Loki + Fluent Bit (Logging Stack)
|
||||
category: systems
|
||||
tags: [loki, fluent-bit, logging, observability]
|
||||
created: "2026-07-24"
|
||||
modified: "2026-07-24"
|
||||
---
|
||||
|
||||
# Loki + Fluent Bit (Logging Stack)
|
||||
|
||||
## Loki
|
||||
- **Namespace:** logging (K8s)
|
||||
- **External Service:** `loki-external`, LoadBalancer `10.0.30.207:3100`
|
||||
- **Caches:** memcached (results-cache, `-m 512 -I 2m`, requests 256Mi/limits 512Mi)
|
||||
|
||||
## Fluent Bit v5.0.9
|
||||
|
||||
### Coverage
|
||||
| Target | How | Status |
|
||||
|--------|-----|--------|
|
||||
| 9 PVE Hosts | `install-host.sh` auf VM200 | ✅ aktiv |
|
||||
| 20 LXC Containers | `install-lxc.sh` | ✅ aktiv (außer CT110 stopped) |
|
||||
| 5 QEMU VMs | `qm guest exec` | ✅ aktiv (VM300/301/302/310/311) |
|
||||
| 6 K8s Nodes | K8s DaemonSet | ✅ aktiv |
|
||||
| VM200 | bereits vorhanden | ✅ aktiv |
|
||||
|
||||
### Inputs (Proxmox Hosts)
|
||||
- systemd journal (pve*, corosync, pacemaker, ceph, zfs, smartd)
|
||||
- auth.log
|
||||
- pveproxy/access.log
|
||||
- pvedaemon.log
|
||||
- cluster.log
|
||||
- **PVE Tasks** (`/var/log/pve/tasks/index`) — UPID-Format
|
||||
- **Ceph Audit** (`/var/log/ceph/ceph.audit.log`) — JSON
|
||||
|
||||
### Scripts
|
||||
- `logging/fluent-bit-proxmox/` im iac-homelab Repo
|
||||
|
||||
## Query
|
||||
```bash
|
||||
logcli query --addr http://10.0.30.207:3100 'job=fluent-bit-proxmox'
|
||||
```
|
||||
|
||||
## Related
|
||||
- [[systems/proxmox-cluster]]
|
||||
- [[systems/monitoring]]
|
||||
- [[systems/rke2-kubernetes]]
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
title: Monitoring & Observability
|
||||
category: systems
|
||||
tags: [monitoring, prometheus, grafana, loki, holmesgpt, alerting]
|
||||
created: "2026-04-28"
|
||||
modified: "2026-07-24"
|
||||
---
|
||||
|
||||
# Monitoring & Observability
|
||||
|
||||
## Prometheus + Grafana
|
||||
- **Prometheus**: K8s (kube-prometheus-stack)
|
||||
- **Grafana**: K8s, Ingress grafana.familie-schoen.com (.205)
|
||||
- **Alertmanager**: K8s, Routes → Telegram
|
||||
|
||||
## Loki Stack (Centralized Logging)
|
||||
- **Loki**: K8s ns=logging, LB 10.0.30.207:3100
|
||||
- **Fluent Bit v5.0.9**: auf 9 PVE Hosts + 20 LXC + 5 QEMU + 6 K8s Nodes
|
||||
- **Inputs**: PVE tasks, Ceph audit, journald, auth, pveproxy, pvedaemon, cluster
|
||||
- Siehe [[systems/loki-fluentbit]]
|
||||
|
||||
## HolmesGPT (SRE)
|
||||
- **Namespace**: holmes
|
||||
- **Pipeline**: Alertmanager → adapter (9095) → HolmesGPT → TG @schoen_sre_bot
|
||||
- **Model**: noris-glm (via ai.noris.de)
|
||||
- RKE2 ctrl-plane alerts disabled
|
||||
|
||||
## Alerting Rules
|
||||
- Via IaC Git Repo deployt (Ansible für CT141, ArgoCD für K8s)
|
||||
- RKE2 control-plane alerts disabled
|
||||
|
||||
## Related
|
||||
- [[systems/loki-fluentbit]]
|
||||
- [[systems/rke2-kubernetes]]
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
title: Proxmox VE Cluster
|
||||
category: systems
|
||||
tags: [proxmox, virtualization, lxc, qemu, pve]
|
||||
created: "2026-04-28"
|
||||
modified: "2026-07-24"
|
||||
---
|
||||
|
||||
# Proxmox VE Cluster
|
||||
|
||||
## Cluster-Konfiguration
|
||||
- **Version:** PVE 9.2.3
|
||||
- **Nodes:** 9 (Quorum OK)
|
||||
- **Hypervisoren:** 10.0.20.x
|
||||
- **Guests:** ~30 LXC + ~10 QEMU VMs
|
||||
|
||||
## Storage
|
||||
- **vm_disks:** Primärer Storage für alle VMs/CTs (SSD)
|
||||
- **hdd_templates:** CT Templates
|
||||
- **Ceph RBD:** ceph-flash, ceph-hdd Pools (über K8s CSI)
|
||||
|
||||
## Netzwerk
|
||||
- VLAN-basiert, Bridge vmbr0
|
||||
- IP-Schema: 10.0.X.Y — siehe [[concepts/network-architecture]]
|
||||
|
||||
## Fluent Bit (Logging)
|
||||
- Alle 9 Hosts haben Fluent Bit aktiv
|
||||
- Inputs: systemd journal (pve*, corosync, pacemaker, ceph, zfs, smartd), auth.log, pveproxy/access.log, pvedaemon.log, cluster.log
|
||||
- **PVE Tasks** (`/var/log/pve/tasks/index`): UPID-Format (Node, PID, Task-Type, VMID, User, Status)
|
||||
- **Ceph Audit** (`/var/log/ceph/ceph.audit.log`): OSD/Pool/RBD Operationen
|
||||
- Output → Loki (10.0.30.207:3100)
|
||||
- Siehe [[systems/loki-fluentbit]]
|
||||
|
||||
## Wichtige Befehle
|
||||
```bash
|
||||
pvecm status # Cluster-Quorum
|
||||
pct status <vmid> # Container-Status
|
||||
pct start/stop <vmid> # Container starten/stoppen
|
||||
qm status <vmid> # VM-Status
|
||||
pvesh get /cluster/resources --type vm # Alle VMs/CTs
|
||||
```
|
||||
|
||||
## SSH-Zugriff
|
||||
- Key: `id_ed25519_proxmox` (funktioniert für 10.0.20.x Hosts)
|
||||
- Siehe [[reference/ssh-keys]]
|
||||
|
||||
## GPU Passthrough (AMD 1002:13c0)
|
||||
- **ms-a2-1** (10.0.20.92): VFIO config gefixt 2026-07-24 — siehe [Solution Doc](../../../docs/solutions/bug-fixes/2026-07-24-vfio-pci-module-loading-race-condition.md)
|
||||
- **ms-a2-2** (10.0.20.93): Funktioniert seit Initialisierung
|
||||
- Benötigte modprobe.d Config:
|
||||
- `blacklist amdgpu` + `blacklist drm` + `blacklist drm_kms_helper`
|
||||
- `options vfio-pci ids=1002:13c0` + `softdep amdgpu pre: vfio-pci`
|
||||
- Worker-05 (VM 102) läuft auf ms-a2-2 mit funktionierendem GPU-Passthrough
|
||||
|
||||
## Bekannte Probleme
|
||||
- osd.0 NVMe 92% Wear — Austausch planen
|
||||
- osd.2/5 nearfull (93-94%) — entlasten
|
||||
- CT110 kaputte libc — Reparatur ausstehend
|
||||
- ms-a2-1 GPU-Passthrough: Config gefixt, Reboot zur Verifikation ausstehend
|
||||
|
||||
## Related
|
||||
- [[systems/ceph-cluster]]
|
||||
- [[systems/rke2-kubernetes]]
|
||||
- [[reference/ip-map]]
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
title: RKE2 Kubernetes Cluster
|
||||
category: systems
|
||||
tags: [kubernetes, rke2, cilium, argocd, cnpg, gitops]
|
||||
created: "2026-07-24"
|
||||
modified: "2026-07-24"
|
||||
---
|
||||
|
||||
# RKE2 Kubernetes Cluster
|
||||
|
||||
## Übersicht
|
||||
- **Version:** v1.35.6-rke2r1
|
||||
- **Nodes:** 6 (3 CP + 3 Worker), alle schedulable (keine CP Taints)
|
||||
- **CNI:** Cilium (LB IP Pool)
|
||||
- **Ingress:**两层 Traefik (extern CT9999 + intern K8s) + Authelia SSO
|
||||
|
||||
## Node IPs
|
||||
| Node | IP | Role |
|
||||
|------|----|------|
|
||||
| cp-01 | 10.0.30.51 | Control Plane |
|
||||
| cp-02 | 10.0.30.52 | Control Plane |
|
||||
| cp-03 | 10.0.30.53 | Control Plane |
|
||||
| worker-01 | 10.0.30.63 | Worker |
|
||||
| worker-04 | 10.0.30.64 | Worker |
|
||||
| worker-05 | 10.0.30.65 | Worker (GPU renderD128 ✅ on ms-a2-2) |
|
||||
|
||||
## Storage
|
||||
- **Ceph CSI**: ceph-flash (fast), ceph-hdd (bulk)
|
||||
- **CNPG PostgreSQL**: `postgres-main` Cluster (3/3 Ready), RW Service `postgres-main-rw.postgres.svc.cluster.local:5432`
|
||||
|
||||
## GitOps
|
||||
- **ArgoCD**: SSH Deploy Keys (read-only) auf Gitea
|
||||
- 15 Applications via SSH (`ssh://git@10.0.30.202:22/dominik/iac-homelab.git`)
|
||||
- **Gitea SSH user is `git`, not `gitea`** — `git.schoen.codes` resolves to Traefik (10.0.30.203, HTTP only), SSH is on 10.0.30.202:22
|
||||
- Workflow: IaC repo auschecken → ändern → commit+push → ArgoCD sync → verify → lokale Kopie löschen
|
||||
- Siehe [[concepts/gitops-workflow]]
|
||||
|
||||
## Backup
|
||||
- **Velero** v1.18.1 with `--features=EnableCSI`, S3 backend (noris cloud `homelab-velero-backups`)
|
||||
- **VolumeSnapshotClass** `ceph-rbd-snapclass` (rbd.csi.ceph.com, default)
|
||||
- Schedules: daily 01:00 UTC (30d TTL), weekly Sun 00:00 UTC (90d TTL)
|
||||
- **CNPG Postgres**: separate hourly backups via barmanObjectStore to S3 (30d retention)
|
||||
- All 24 PVCs (~430 GiB) now have off-site CSI snapshots
|
||||
- **Helm pitfall**: `features: EnableCSI` must be under `configuration:` in Velero Helm values (not top-level)
|
||||
|
||||
## Deployed Workloads
|
||||
| Workload | Namespace | Notes |
|
||||
|----------|-----------|-------|
|
||||
| Paperless-ngx v3.0.0 | paperless | PostgreSQL (CNPG), `enableServiceLinks: false` |
|
||||
| Gitea | gitea | git.schoen.codes |
|
||||
| Loki Stack | logging | LB 10.0.30.207:3100 |
|
||||
| Hindsight | hindsight | LB 10.0.30.201:9177 |
|
||||
| Authelia | authelia | SSO, Galera backend |
|
||||
| HolmesGPT | holmes | SRE Bot |
|
||||
| Traefik | traefik | Ingress Controller |
|
||||
| CNPG | postgres | PostgreSQL HA |
|
||||
|
||||
## GPU
|
||||
- worker-04: renderD128 ✅ (AMD GPU Passthrough, rombar=1, on n5pro)
|
||||
- worker-05: renderD128 ✅ (AMD GPU Passthrough, rombar=1, on ms-a2-2)
|
||||
- IaC: unified `amd-gpu` PCI mapping (n5pro + ms-a2-1 + ms-a2-2) + dynamic hostpci (one worker block, gpu flag)
|
||||
- ms-a2-1 has kernel BUG with 1002:13c0 (renderD128 missing) — worker-05 moved to ms-a2-2 (identical hardware)
|
||||
|
||||
## Known Pitfalls
|
||||
- `enableServiceLinks: false` bei Apps deren Service-Name mit Env-Vars kollidiert (z.B. Paperless `PAPERLESS_PORT`)
|
||||
- ArgoCD `--force` kann nicht mit ServerSideApply kombiniert werden
|
||||
- CoreDNS: `hosts` configBlock nicht params
|
||||
- etcd Defrag monatlich (CronJob)
|
||||
|
||||
## Related Skills
|
||||
- `rke2-cluster-administration` (devops)
|
||||
|
||||
## Related
|
||||
- [[systems/ceph-cluster]]
|
||||
- [[systems/gitea]]
|
||||
- [[systems/hindsight]]
|
||||
- [[systems/loki-fluentbit]]
|
||||
- [[concepts/gitops-workflow]]
|
||||
- [[reference/ip-map]]
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
title: cloud.familie-schoen.com
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
type: entity
|
||||
tags: [cloud, hosting, file-sync, ssl, domains]
|
||||
sources: [raw/articles/cloud-familie-schoen-com-scan.md]
|
||||
---
|
||||
|
||||
# cloud.familie-schoen.com
|
||||
|
||||
## Overview
|
||||
Private [[Seafile 13.0.19]] file sync and collaboration server. Self-hosted alternative to Nextcloud/Owncloud.
|
||||
|
||||
## DNS & Network
|
||||
- **Domain:** `cloud.familie-schoen.com`
|
||||
- **Alias (CNAME):** `schoen.sytes.net` (DuckDNS dynamic DNS)
|
||||
- **Public IP:** `84.39.84.125`
|
||||
- **Ports:** 80 (HTTP → HTTPS redirect), 443 (HTTPS/Seafile)
|
||||
- **No internal LAN exposure** (public IP, not on 10.0.x.x)
|
||||
|
||||
## SSL/TLS
|
||||
- **Provider:** Let's Encrypt (R12 → ISRG Root X1)
|
||||
- **SAN:** `cloud.familie-schoen.com`
|
||||
- **Auto-renewal:** Managed via certbot or acme.sh on host
|
||||
|
||||
## Services
|
||||
### Seafile (Seahub Web Frontend)
|
||||
- **Version:** 13.0.19
|
||||
- **Features:** `seafile-basic` (no Pro features detected)
|
||||
- **Web UI:** `/accounts/login/?next=/` (Django-based auth)
|
||||
- **API:** `/api2/server-info/` (unauthenticated, returns version)
|
||||
- **Static Assets:** `/media/` (timestamped: `t=20250802`, `t=20251013`)
|
||||
- **Languages:** CA, CS, DE, EN, ES, FR, IT, JA, KO, NL, PL, PT, RO, RU, SV, TR, UK, ZH-CN, ZH-TW
|
||||
|
||||
### Reverse Proxy (nginx)
|
||||
- HTTP → HTTPS 308 redirect
|
||||
- Security headers: `cache-control: no-cache, no-store`, `SameSite=Lax`
|
||||
- Session cookie: `sessionid` (HttpOnly, 24h), `sfcsrftoken` (365d)
|
||||
- No CDN/proxy layer (direct to Seafile)
|
||||
|
||||
## Access
|
||||
- **URL:** `https://cloud.familie-schoen.com`
|
||||
- **Admin:** `dominik@familie-schoen.com` (1Password Vault "Hermes")
|
||||
|
||||
## Related Systems
|
||||
- Part of [[Infrastructure]]
|
||||
- Shares domain family with [[Email-System]] (familie-schoen.com)
|
||||
- Hosted on same machine as other family services
|
||||
Reference in New Issue
Block a user