81 lines
3.9 KiB
Markdown
81 lines
3.9 KiB
Markdown
---
|
|
title: RKE2 Kubernetes Cluster
|
|
category: systems
|
|
tags: [kubernetes, rke2, cilium, argocd, cnpg, gitops]
|
|
created: "2026-07-24"
|
|
modified: "2026-09-17"
|
|
---
|
|
|
|
# RKE2 Kubernetes Cluster
|
|
|
|
## Übersicht
|
|
- **Version:** v1.35.6-rke2r1
|
|
- **Nodes:** 6 (3 CP + 3 Worker), alle schedulable (keine CP Taints)
|
|
- **CNI:** Cilium (LB IP Pool)
|
|
- **Ingress:**两层 Traefik (extern CT9999 + intern K8s) + Authelia SSO
|
|
|
|
## Node IPs
|
|
| Node | IP | Role |
|
|
|------|----|------|
|
|
| cp-01 | 10.0.30.51 | Control Plane |
|
|
| cp-02 | 10.0.30.52 | Control Plane |
|
|
| cp-03 | 10.0.30.53 | Control Plane |
|
|
| worker-01 | 10.0.30.63 | Worker |
|
|
| worker-04 | 10.0.30.64 | Worker |
|
|
| worker-05 | 10.0.30.65 | Worker (GPU renderD128 ✅ on ms-a2-2) |
|
|
|
|
## Storage
|
|
- **Ceph CSI**: ceph-flash (fast), ceph-hdd (bulk)
|
|
- **⚠️ Snapshot-CRD-Flavor (seit Rebuild 2026-08-01):** `volumesnapshotclasses`-CRD (RKE2-Addon `rke2-snapshot-controller-crd`) hat **FLAT-Schema** — `driver`/`deletionPolicy`/`parameters` auf TOP-LEVEL, `spec:` existiert nicht im Schema (Upstream-CRD wäre nested!). Niemals struktur-intuitiv "reparieren" — CRD-Schema lesen. Live-VSCs: `ceph-rbd-snapclass` (default) + `cephfs-snapclass`
|
|
- **CNPG PostgreSQL**: `postgres-main` Cluster (3/3 Ready), RW Service `postgres-main-rw.postgres.svc.cluster.local:5432`; Specs **100Gi data / 20Gi WAL** (grow-only — Shrink wird von CNPG-Admission verboten, Git immer nach oben alignieren)
|
|
|
|
## GitOps
|
|
- **ArgoCD**: SSH Deploy Keys (read-only) auf Gitea
|
|
- **Feed-Quelle (Stand 2026-09-17):** `http://10.0.30.105:3000/...` = Legacy-CT108-Gitea (noch aktiv!). Geplanter Flip auf `ssh://git@10.0.30.200:22/dominik/iac-homelab.git` (end-to-end verifiziert), danach CT108-Stop (Freigabe Dominik)
|
|
- **Gitea SSH user is `git`, not `gitea`** — SSH-LoadBalancer = **10.0.30.200:22** (svc gitea-ssh, targetPort 2222, NodePort 31441). Legacy-.202 ist TOD (Timeout). Repo-Historien am 2026-09-17 konsolidiert (identische Tips auf beiden Remotes, Merge `d76d3e8` + `8c741c5`)
|
|
- Workflow: IaC repo auschecken → ändern → commit+push → ArgoCD sync → verify → lokale Kopie löschen
|
|
- Siehe [[concepts/gitops-workflow]]
|
|
|
|
## Backup
|
|
- **Velero** v1.18.1 with `--features=EnableCSI`, S3 backend (noris cloud `homelab-velero-backups`)
|
|
- **VolumeSnapshotClass** `ceph-rbd-snapclass` (rbd.csi.ceph.com, default)
|
|
- Schedules: daily 01:00 UTC (30d TTL), weekly Sun 00:00 UTC (90d TTL)
|
|
- **CNPG Postgres**: separate hourly backups via barmanObjectStore to S3 (30d retention)
|
|
- All 24 PVCs (~430 GiB) now have off-site CSI snapshots
|
|
- **Helm pitfall**: `features: EnableCSI` must be under `configuration:` in Velero Helm values (not top-level)
|
|
|
|
## Deployed Workloads
|
|
| Workload | Namespace | Notes |
|
|
|----------|-----------|-------|
|
|
| Paperless-ngx v3.0.0 | paperless | PostgreSQL (CNPG), `enableServiceLinks: false` |
|
|
| Gitea | gitea | git.schoen.codes |
|
|
| Loki Stack | logging | LB 10.0.30.207:3100 |
|
|
| Hindsight | hindsight | LB 10.0.30.201:9177 |
|
|
| Authelia | authelia | SSO, Galera backend |
|
|
| HolmesGPT | holmes | SRE Bot |
|
|
| Traefik | traefik | Ingress Controller |
|
|
| CNPG | postgres | PostgreSQL HA |
|
|
|
|
## GPU
|
|
- worker-04: renderD128 ✅ (AMD GPU Passthrough, rombar=1, on n5pro)
|
|
- worker-05: renderD128 ✅ (AMD GPU Passthrough, rombar=1, on ms-a2-2)
|
|
- IaC: unified `amd-gpu` PCI mapping (n5pro + ms-a2-1 + ms-a2-2) + dynamic hostpci (one worker block, gpu flag)
|
|
- ms-a2-1 has kernel BUG with 1002:13c0 (renderD128 missing) — worker-05 moved to ms-a2-2 (identical hardware)
|
|
|
|
## Known Pitfalls
|
|
- `enableServiceLinks: false` bei Apps deren Service-Name mit Env-Vars kollidiert (z.B. Paperless `PAPERLESS_PORT`)
|
|
- ArgoCD `--force` kann nicht mit ServerSideApply kombiniert werden
|
|
- CoreDNS: `hosts` configBlock nicht params
|
|
- etcd Defrag monatlich (CronJob)
|
|
|
|
## Related Skills
|
|
- `rke2-cluster-administration` (devops)
|
|
|
|
## Related
|
|
- [[systems/ceph-cluster]]
|
|
- [[systems/gitea]]
|
|
- [[systems/hindsight]]
|
|
- [[systems/loki-fluentbit]]
|
|
- [[concepts/gitops-workflow]]
|
|
- [[reference/ip-map]]
|