hermes-skills/devops/rke2-cluster-administration/SKILL.md at main

Files

T

Debian 01bd921ced feat: add home-assistant-dashboard-conventions skill + update multiple skills

- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards)
- Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills
- New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size

2026-07-14 18:35:16 +00:00

98 KiB

Raw Permalink Blame History

name, description, version, tags

name

description

version

Overview

This skill covers administration of an RKE2 Kubernetes cluster running on Proxmox VE QEMU VMs. The cluster uses:

RKE2 v1.35.x (CNCF-conformant K8s)
Cilium as CNI (eBPF-based, with L2 announcements + LB IP pools)
ArgoCD for GitOps (App-of-Apps pattern)
External Secrets Operator synced to 1Password
CloudNativePG for PostgreSQL
Velero for backups
Ceph CSI for RBD storage
Traefik as ingress controller (RKE2-bundled)

Note: MariaDB Galera runs as native VMs (VM300-302 + MaxScale VM310), NOT in K8s. A K8s MariaDB Operator was previously deployed but has been removed — Galera VMs are the authoritative database.

Load this skill for any RKE2 cluster lifecycle task, node diagnostics, ArgoCD/GitOps operations, Cilium LB pool changes, or K8s service migration.

User preference: Always check skills_list for relevant skills before starting any task. This skill was often the right one but wasn't loaded until late in sessions, wasting time on trial-and-error. Load it early.

CRITICAL workflow rule (user-enforced): Before starting ANY task on this infrastructure, run skills_list and search for relevant skills FIRST. This skill alone contains the mgmt-runner IP (10.0.30.124), SSH key paths, kubectl location, PVE node mappings, and dozens of pitfalls — all of which are lost time if you probe blindly without loading the skill first.

Section 1: Cluster Architecture & Access

1.1 Cluster Topology

Role	VMID	Node	IP	Specs
CP-01	118	varies (HA)	10.0.30.51	4c / 12GB / 40GB
CP-02	130	varies (HA)	10.0.30.52	4c / 12GB / 40GB
CP-03	129	varies (HA)	10.0.30.53	4c / 12GB / 40GB
Worker-01	128	varies (HA)	10.0.30.61	4c / 12GB / 80GB
Worker-02	132	varies (HA)	10.0.30.62	4c / 12GB / 80GB
Worker-03	131	varies (HA)	10.0.30.63	4c / 12GB / 80GB

RKE2 API VIP: 10.0.30.50 (Keepalived on CP nodes)

Note: PVE HA migrates VMs between nodes. Always discover which node hosts a given VM before operating on it (see Section 6.1).

1.2 Management VM (kubectl access)

VM 200 (mgmt-runner-01) has kubectl, helm, and git installed. It also serves as the self-hosted GitHub Actions runner.

Direct SSH from Hermes host: ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 — this is the fastest path for running kubectl/helm/git commands.

Kubeconfig location: /root/.kube/config on VM 200. Must set KUBECONFIG=/root/.kube/config explicitly — kubectl does not pick it up from ~/.kube/config via qm guest exec.

# Run kubectl on mgmt-runner via Proxmox
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "ssh -o ConnectTimeout=5 root@NODE_HOSTING_VM200 \
   'qm guest exec 200 -- sh -c \"KUBECONFIG=/root/.kube/config kubectl get nodes\"'"

Pitfall: kubectl without explicit KUBECONFIG= returns localhost:8080 connection refused — the default kubeconfig path isn't honored through qm guest exec.

1.3 Getting Kubeconfig from a Control Plane Node

If the mgmt-runner doesn't have kubeconfig yet:

# Get kubeconfig from CP-01 (VM 118)
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "qm guest exec 118 -- cat /etc/rancher/rke2/rke2.yaml"

# Fix the server URL from 127.0.0.1 to the CP IP
# Replace 127.0.0.1 with 10.0.30.51 (CP-01 IP)
# Base64-transfer to mgmt-runner (see docker-host-administration Section 8.4)

Section 2: Cluster Lifecycle (Start/Stop/Reboot)

2.1 Starting the Cluster from Cold Boot

All 6 VMs were stopped → start them in order:

# 1. Start Control Plane nodes first
# CPs: 118, 130, 129 — may be on different PVE nodes due to HA
# Start each individually to avoid HA placement races

# 2. Wait 15-30s for CPs to begin etcd convergence
sleep 15

# 3. Start Worker nodes
# Workers: 128, 131, 132

# 4. Wait for cluster to converge (2-5 minutes)
# Workers initially show NotReady — they need to retrieve
# serving-kubelet.crt from the API server (503 until CPs are ready)

Pitfall: qm start returns immediately ("Requesting HA start") but HA may place the VM on a different node than expected. Always scan all PVE nodes to find where each VM landed:

for node in proxmox1 proxmox2 proxmox3 proxmox4 proxmox5 proxmox6 proxmox7; do
  ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
    "ssh -o ConnectTimeout=3 root@$node 'qm list | grep rke2'" 2>/dev/null
done

2.2 Cold-Boot Recovery Timeline

After all VMs are started, expect this convergence sequence:

Time	What happens
0-30s	CP nodes boot, etcd begins forming quorum
30-60s	Workers connect to CP via proxy (`wss://CP_IP:9345/v1-rke2/connect`)
1-2min	Workers retrieve serving-kubelet.crt (was 503 during CP convergence)
2-3min	Cilium agents start on all nodes, networking comes up
3-5min	ArgoCD, External Secrets, operators begin reconciling
5-10min	Most pods Running, some Terminating (old replicas from before shutdown)
10-15min	Databases (Galera, PostgreSQL) reach quorum and become ready

Patience is key. Don't panic when pods show Unknown/Terminating early on — this is normal cold-boot behavior, not a failure.

2.3 Stopping the Cluster

# 1. Stop workers first (allows graceful pod eviction)
# 2. Then stop CPs (etcd will downsize gracefully)
# 3. Verify all stopped

Workers stopping first prevents etcd from losing quorum abruptly.

Section 3: Cilium LoadBalancer IP Pool Management

3.1 The LB Pool Problem

Cilium assigns LoadBalancer IPs to K8s services (Traefik, MaxScale, PostgreSQL) from a configurable IP pool. If this pool overlaps with existing infrastructure IPs (Galera VIPs, MaxScale VIPs, static services), you get IP conflicts.

3.2 Checking Current Pool

KUBECONFIG=/root/.kube/config kubectl get ciliumloadbalancerippool -o yaml

Key fields:

spec:
  blocks:
  - start: "10.0.30.200"   # Pool start
    stop: "10.0.30.250"    # Pool end

3.3 Changing the Pool (Free Up Conflicting IPs)

When K8s LB IPs collide with existing infrastructure:

# 1. Delete the old pool
kubectl delete ciliumloadbalancerippool default-pool

# 2. Apply new pool with non-overlapping range
cat <<'EOF' | kubectl apply -f -
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: default-pool
spec:
  blocks:
  - start: "10.0.30.200"
    stop: "10.0.30.250"
  disabled: false
EOF

# 3. Verify services picked up new IPs
kubectl get svc -A -o wide | grep LoadBalancer

Services automatically release old IPs and get new ones from the updated pool. This is non-disruptive — Cilium handles the transition.

3.4 Making the Change Persistent

The pool is defined in a manifest on all CP nodes: /var/lib/rancher/rke2/server/manifests/cilium-l2-lb.yaml

RKE2 auto-applies manifests from this directory on CP restart. If you only patch the live CRD, the old pool comes back on next CP reboot.

Fix the manifest on ALL 3 CP nodes (use base64 transfer — see docker-host-administration Section 8.4):

NEW_MANIFEST=$(cat <<'EOF'
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: default-pool
spec:
  blocks:
    - start: "10.0.30.200"
      stop: "10.0.30.250"
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: default-l2-policy
spec:
  interfaces:
    - ^eth[0-9]+
  externalIPs: true
  loadBalancerIPs: true
EOF
)

B64=$(echo "$NEW_MANIFEST" | base64 -w0)

# Write to all 3 CPs (discover which PVE node hosts each first!)
for combo in "NODE1 118" "NODE2 130" "NODE3 129"; do
  node=$(echo $combo | cut -d' ' -f1)
  vmid=$(echo $combo | cut -d' ' -f2)
  ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
    "ssh -o ConnectTimeout=5 root@$node \
     'qm guest exec $vmid -- sh -c \"echo $B64 | base64 -d > /var/lib/rancher/rke2/server/manifests/cilium-l2-lb.yaml\"'"
done

3.5 Fixing the IaC Source

The pool is also defined in the IaC repository (Ansible playbook that bootstraps RKE2). Update and push:

# In iac-homelab repo: epic-2-k8s/ansible/playbook.yml
# Change the LB pool start/stop values to match the new range
sed -i 's/OLD_START/NEW_START/g' epic-2-k8s/ansible/playbook.yml
sed -i 's/OLD_STOP/NEW_STOP/g' epic-2-k8s/ansible/playbook.yml
git add -A && git commit -m "fix-k8s-move-cilium-lb-pool" && git push origin main

Three places to update (all must agree):

Live CRD (kubectl patch / delete+apply)
CP manifest files (/var/lib/rancher/rke2/server/manifests/cilium-l2-lb.yaml)
IaC repo (epic-2-k8s/ansible/playbook.yml)

Section 4: ArgoCD GitOps

4.1 ArgoCD Access

UI: https://<TRAEFIK_LB_IP>/ with Host header argocd (Traefik routes by hostname)

Admin password: Retrieved from kubectl:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d

CLI: argocd binary (if installed on mgmt-runner)

4.2 App-of-Apps Pattern

The root ArgoCD Application points to a Git repo path containing child Application manifests:

# Root Application (synced from epic-5-gitops/tofu)
spec:
  source:
    repoURL: http://GITEA_URL/dominik/iac-homelab.git
    path: clusters/main/apps
    targetRevision: main
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Child apps in clusters/main/apps/:

operators.yaml (sync-wave 0): CloudNativePG, Velero
databases.yaml (sync-wave 1): PostgreSQL
backups.yaml (sync-wave 2): Velero schedules + locations

Removed apps: memory.yaml (sync-wave 3: Qdrant, Ollama, Memory API) was removed 2026-07-12 — the openclaw-memory namespace is fully deleted. See references/rke2-upgrade-and-recovery-2026-07.md Phase 3.

4.3 Adding a New Application

To deploy a new service via GitOps:

Create namespace manifest: clusters/main/<service>/namespace.yaml
Create deployment/service/ingress manifests in clusters/main/<service>/
Create ArgoCD Application manifest: clusters/main/apps/<service>.yaml
Push to Gitea — ArgoCD auto-syncs within minutes

Pitfall: K8s env var interpolation order — When using $(VAR_NAME) to reference one env var inside another (e.g. building a connection string from a secret), the referenced var MUST be listed before the referencing var. K8s processes env vars top-to-bottom; if the reference comes first, it resolves to an empty string.

# CORRECT — HINDSIGHT_DB_PASSWORD before DATABASE_URL
env:
  - name: HINDSIGHT_DB_PASSWORD
    valueFrom:
      secretKeyRef: ...
  - name: DATABASE_URL
    value: "postgresql://user:$(HINDSIGHT_DB_PASSWORD)@host/db"

# WRONG — DATABASE_URL resolves with empty password
env:
  - name: DATABASE_URL
    value: "postgresql://user:$(HINDSIGHT_DB_PASSWORD)@host/db"
  - name: HINDSIGHT_DB_PASSWORD
    valueFrom:
      secretKeyRef: ...

Pitfall: ExternalSecret API version — The cluster's ExternalSecrets Operator serves both v1 and v1beta1, but existing working ExternalSecrets all use v1. Using v1beta1 can cause "the server could not find the requested resource" errors in ArgoCD even though the CRD technically serves both versions. Always use apiVersion: external-secrets.io/v1.

Pitfall: ArgoCD caches stale manifests — When you fix a manifest (e.g. wrong API version) and push, ArgoCD may keep trying the old version. Deleting the Application and recreating it forces a clean re-read from the repo. Alternatively:

kubectl annotate application -n argocd APP_NAME \
  argocd.argoproj.io/refresh=hard --overwrite

Pitfall: StatefulSet PVC template immutability — If a StatefulSet has the wrong StorageClass in its volumeClaimTemplates, simply updating the manifest and letting ArgoCD sync won't fix it — the old PVC persists. Must delete the STS + PVC manually, then let ArgoCD recreate with the corrected manifest:

kubectl delete sts STATEFULSET_NAME -n NS --cascade=foreground
kubectl delete pvc DATA_PVC_NAME -n NS

Pitfall: kubectl rollout restart does NOT apply manifest changes — rollout restart only restarts pods with the current deployment spec. It does NOT pull new manifest changes from the repo. After pushing a manifest fix, you must kubectl apply -f <manifest> first (or wait for ArgoCD to sync), THEN rollout restart if needed. If you only rollout restart, the new pod starts with the old spec and hits the same error.

Pitfall: ArgoCD does not recreate manually deleted resources — When you kubectl delete a resource (e.g. an ExternalSecret) to force re-sync, ArgoCD may report "successfully synced (all tasks run)" but NOT recreate the deleted resource. This happens when the resource was already synced once and ArgoCD considers it "managed" but doesn't detect the drift. Fix: kubectl apply -f <manifest> manually, or delete the entire ArgoCD Application and recreate it.

Pitfall: PostgreSQL PVC lost+found conflict — When mounting a PVC directly at /var/lib/postgresql/data, PostgreSQL fails to initialize because the PVC filesystem contains a lost+found directory (created by mkfs). The PG initdb script refuses to use a non-empty data directory. Fix: Set PGDATA env var to a subdirectory:

env:
  - name: PGDATA
    value: /var/lib/postgresql/data/pgdata

The volumeMount stays at /var/lib/postgresql/data — PGDATA just tells Postgres to use a clean subdirectory inside it.

# clusters/main/apps/frigate.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: frigate
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "4"
spec:
  project: default
  source:
    repoURL: http://GITEA_URL/dominik/iac-homelab.git
    targetRevision: main
    path: clusters/main/frigate
  destination:
    server: https://kubernetes.default.svc
    namespace: frigate
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true

4.3b Migrating the Git Server Into K8s (Chicken-and-Egg)

When migrating the Git server that ArgoCD depends on (e.g. Gitea from LXC to K8s), the circular dependency requires a careful sequence:

Database choice: Galera over CNPG (user decision 2026-07-13)

When migrating Gitea to K8s, use the existing MariaDB Galera cluster (MaxScale VIP 10.0.30.70:3306) instead of deploying a new CNPG PostgreSQL cluster. Rationale:

Criterion	Galera (chosen)	CNPG (rejected)
HA	✅ 3 nodes + MaxScale	❌ 1 instance planned (no HA)
New resources	❌ None	512Mi RAM + 10Gi SSD
DB engine fit	✅ MySQL = Gitea's native DB	PG works but less common
Backup	✅ Existing Galera snapshots	Needs Barman/S3 setup
K8s-native	❌ External	✅ In-cluster

Trade-off accepted: Galera binds Gitea to existing homelab infra (not fully K8s-native), but HA + zero new resources outweigh this.

Safe sequence:

Keep the old Git server running — ArgoCD continues pulling from it
Create database on Galera (CREATE DATABASE gitea; CREATE USER gitea@'%';)
Deploy new Git server to K8s (Helm chart + ExternalSecrets, DB = Galera)
Migrate repos + metadata via Gitea Migration API (one API call per repo brings repos + issues + labels + milestones + PRs + releases + wiki — no manual SQLite→SQL conversion needed)
Verify new Git server serves identical content
Cutover DNS (e.g. git.schoen.codes → K8s LoadBalancer IP)
Update all ArgoCD Application repoURL fields: old URL → new URL
Push the URL change to BOTH Git servers (they must be in sync)
Force ArgoCD hard refresh — verify it now pulls from the new Git server
After 7 days of stable operation, shut down the old Git server

Rollback: Keep old Git server running until step 9 is confirmed. If anything breaks: revert DNS + revert ArgoCD repoURL + start old server.

Key insight: The old Git server is NEVER modified during migration (read-only mirror source). This guarantees zero data loss on rollback.

Key insight: Gitea Migration API eliminates the SQLite→SQL schema conversion entirely. Each POST /api/v1/repos/migrate call clones the repo + imports issues/labels/milestones/PRs/releases/wiki in one shot. This is vastly simpler than dumping SQLite, converting schemas, and importing into PG/MySQL manually.

Pattern reuse: This sequence applies to any self-hosted GitOps dependency migration (Gitea, GitLab, Forgejo). The Helm chart and DB details change, but the chicken-and-egg sequence is universal.

Gitea Migration API: ALLOWED_HOST_LIST (NOT ALLOWED_HOSTS)

Gitea 1.26 blocks repo migrations from private IPs by default. The setting to allowlist source hosts is under [migrations] (plural) and the key is ALLOWED_HOST_LIST — NOT ALLOWED_HOSTS.

# In Helm values (gitea.config):
migrations:
  ALLOWED_HOST_LIST: private,external,loopback,10.0.30.105
  ALLOW_LOCALNETWORKS: true

Built-in categories: loopback (localhost), private (LAN/intranet), external (public hosts), * (all hosts). Can also use CIDR (10.0.30.0/24) or wildcard hosts (*.mydomain.com).

Pitfall: [migration] (singular) is silently ignored — the section MUST be [migrations] (plural). Both ALLOWED_HOSTS and ALLOW_LOCALNETWORKS under [migration] have no effect.

Pitfall: Even ALLOWED_HOST_LIST: * fails if the section name is wrong. Verify with grep migrations /data/gitea/conf/app.ini inside the pod — the section header must be [migrations].

Requires pod restart: app.ini changes are only read at Gitea startup. After changing migration settings, delete the pod to force restart (ArgoCD selfHeal recreates it).

Gitea Migration API: PR Import Failure (Retry Without PRs)

Some repos fail during migration with:

"error while listing pull requests (page: 1, pagesize: 49). Error: not found"
"initRepository: getRepositoryByID: repository does not exist"

This is a Gitea Migration API bug affecting repos with empty or corrupted PR metadata. Fix: Delete the half-created repo, then retry with pull_requests: false:

# First: DELETE /api/v1/repos/{owner}/{name} (cleanup half-created)
# Then: POST /api/v1/repos/migrate with pull_requests=False
migrate_data = {
    "clone_addr": f"{CT108_URL}/{owner}/{name}.git",
    "repo_owner": owner, "repo_name": name,
    "mirror": False, "wiki": True, "issues": True,
    "labels": True, "milestones": True,
    "pull_requests": False,  # <-- key fix
    "releases": True, "service": "gitea",
    "auth_token": TOKEN,
}

In practice, ~17% of repos (4/23) needed this retry. All succeeded without PRs.

ArgoCD App-of-Apps: Hard Refresh for New Child Apps

When adding new Application manifests to clusters/main/apps/, the root App-of-Apps must be hard-refreshed to discover them:

# 1. Hard refresh root app to discover new child Application manifests
kubectl patch application root -n argocd --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

# 2. Wait 15-30s for root to sync (creates child Application objects)

# 3. Hard refresh the new child app to force Helm chart pull
kubectl patch application gitea -n argocd --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

Without the root refresh, new child apps don't appear at all. Without the child refresh, ArgoCD may use stale cached Helm values.

Pitfall: kubectl patch with --type merge for the annotation works reliably. kubectl annotate also works but sometimes the annotation is consumed before the next reconciliation cycle.

Reference: Full migration plan at docs/plans/2026-07-13-gitea-k8s-migration.md in the iac-homelab repo.

4.3c Gitea Helm Chart v12 Pitfalls (2026-07-14)

DB Password Injection: `additionalConfigFromEnvs` (NOT `env`)

The Gitea Helm Chart v12 generates app.ini in an init container (configure-gitea) which runs gitea migrate. If the DB password is not in app.ini, migration fails with Access denied (using password: NO).

Wrong approaches (do NOT use):

gitea.env — ignored by the chart (not a valid chart value path)
gitea.database.existingSecret — not a valid chart value
gitea.config.database.PASSWD: __placeholder__ — literal placeholder reaches the DB, not a secret reference
deployment.env — only goes into the main container, NOT the init containers that run gitea migrate

Correct approach: gitea.additionalConfigFromEnvs — this is templated into ALL init containers (init-directories, init-app-ini, configure-gitea) AND the main container:

gitea:
  config:
    database:
      DB_TYPE: mysql
      HOST: 10.0.30.70:3306
      NAME: gitea
      USER: gitea
      # No PASSWD here — injected via additionalConfigFromEnvs below
  additionalConfigFromEnvs:
    - name: GITEA__database__PASSWD
      valueFrom:
        secretKeyRef:
          name: gitea-db
          key: password

The GITEA__database__PASSWD env var overrides app.ini at runtime via Gitea's environment-based config system (GITEA__section__KEY).

Key insight: The chart template checks deployment.env (goes into all containers) and gitea.additionalConfigFromEnvs (goes into init + main containers). The gitea.env path does NOT exist in the chart template — it is silently ignored.

Valkey/Redis Auto-Deployed Despite `redis-cluster: false`

Chart v12 replaced redis-cluster with valkey-cluster. Setting redis-cluster.enabled: false alone does NOT disable the cache — you must ALSO set:

valkey-cluster:
  enabled: false

Otherwise 3 Valkey pods deploy automatically and consume resources. They also fail to form a cluster (minority partition) if DNS isn't ready, producing noisy cluster_state:fail warnings.

RWO PVC Blocks New Pod During Rollout

With persistence.accessModes: [ReadWriteOnce] and replicaCount: 1, ArgoCD's rolling update creates a new pod before the old one terminates. Both pods compete for the same RWO PVC — the new pod stays in Init:0/3 with FailedAttachVolume: Multi-Attach error.

Fix: Scale to 0 first, wait for PVC release, then scale to 1:

kubectl scale deploy -n gitea gitea --replicas=0
sleep 15
kubectl scale deploy -n gitea gitea --replicas=1

Galera Schema Creation Slowness

First-time Gitea DB migration creates ~104 tables. On Galera with synchronous 3-node replication, each CREATE TABLE + index takes longer than on a single-node DB. Total migration time: 5-8 minutes (vs <1 min on standalone MySQL). The init container shows Init:2/3 during this period — do NOT assume it's hung.

Check progress via Galera:

# On a Galera node (SSH debian@10.0.30.72, sudo mariadb):
SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='gitea';
SHOW PROCESSLIST;  -- look for "creating table" or "Committing alter table"

ArgoCD App Split: Config vs Helm

Split the Gitea deployment into two ArgoCD Applications:

gitea-config (sync-wave 1): namespace + ExternalSecrets from Git repo (clusters/main/gitea/)
gitea (sync-wave 2): Helm chart from https://dl.gitea.com/charts/ (direct Helm source, not from Git)

This ensures secrets are synced before the Helm chart tries to use them.

4.3d CoreDNS Internal DNS Overrides (2026-07-14)

When migrating services to K8s that other K8s workloads need to reach by hostname (e.g. git.schoen.codes for ArgoCD's repoURL), use a CoreDNS hosts plugin override via HelmChartConfig — NOT external DNS. This keeps resolution internal to the cluster.

RKE2 CoreDNS chart uses servers[].plugins[] structure — a flat corefile: key in valuesContent does NOT work:

# clusters/main/operators/coredns-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    servers:
      - zones:
          - zone: .
        port: 53
        plugins:
          - name: errors
          - name: health
            configBlock: |
              lameduck 10s
          - name: ready
          - name: hosts
            parameters: |
              10.0.30.203 git.schoen.codes
              fallthrough
          - name: kubernetes
            parameters: cluster.local in-addr.arpa ip6.arpa
            configBlock: |
              pods insecure
              fallthrough in-addr.arpa ip6.arpa
              ttl 30
          - name: prometheus
            parameters: 0.0.0.0:9153
          - name: forward
            parameters: . /etc/resolv.conf
          - name: cache
            parameters: 30
          - name: loop
          - name: reload
          - name: loadbalance

Pitfall: valuesContent: corefile: | does NOT work. The RKE2 CoreDNS chart template renders the Corefile from servers[].plugins[], not from a raw corefile key. A corefile: value is silently ignored.

Pitfall: HelmChartConfig applies but ConfigMap doesn't update immediately. The HelmChartConfig modifies the HelmChart spec, but the Helm controller must re-run the helm install job to render the new ConfigMap. This can take 1-2 minutes. To force immediate application, either:

Annotate the HelmChart: kubectl annotate helmchart rke2-coredns -n kube-system helm.cattle.io/retrigger="$(date +%s)" --overwrite

OR directly patch the ConfigMap (faster, but non-persistent — the HelmChartConfig ensures it survives future redeploys):

kubectl patch cm rke2-coredns-rke2-coredns -n kube-system --type merge \
  -p '{"data":{"Corefile":".:53 {\n    errors\n    health {\n        lameduck 10s\n    }\n    ready\n    hosts {\n        10.0.30.203 git.schoen.codes\n        fallthrough\n    }\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n        pods insecure\n        fallthrough in-addr.arpa ip6.arpa\n        ttl 30\n    }\n    prometheus 0.0.0.0:9153\n    forward . /etc/resolv.conf\n    cache 30\n    loop\n    reload\n    loadbalance\n}\n"}}'

Pitfall: CoreDNS reload plugin takes ~30s to pick up ConfigMap changes. After patching the ConfigMap, DNS queries for the new host may still return NXDOMAIN for 30-60s. The cache plugin may also cache the negative result. Wait 35s then retest.

Verification:

# From inside any K8s pod:
kubectl exec -n gitea deploy/gitea -- nslookup git.schoen.codes 10.43.0.10
# Expected: Name: git.schoen.codes, Address: 10.0.30.203

User preference: internal DNS, not external/public. When resolving K8s-internal service hostnames (like git.schoen.codes), use CoreDNS overrides — NOT public DNS records. This keeps traffic internal and avoids hairpin routing through the firewall.

4.3e ArgoCD Self-Management Cutover (2026-07-14)

Complete sequence to switch ArgoCD from an external Git server to a K8s-internal Gitea (chicken-and-egg resolution):

Prerequisites:

New Gitea running in K8s, repos migrated, reachable via Traefik VIP
CoreDNS override for git.schoen.codes → Traefik VIP (Section 4.3d)

Step 1: Create ArgoCD repository secret

# Generate a token on the new Gitea
kubectl exec -n gitea deploy/gitea -- gitea admin user generate-access-token \
  -u dominik -t argocd --raw --config /data/gitea/conf/app.ini

# Create K8s secret (MUST be labeled for ArgoCD to recognize it!)
kubectl create secret generic argocd-repo-k8s-gitea -n argocd \
  --from-literal=url="http://git.schoen.codes/dominik/iac-homelab.git" \
  --from-literal=username="dominik" \
  --from-literal(password)="TOKEN_VALUE" \
  --from-literal=type="git" \
  --dry-run=client -o yaml | kubectl apply -f -

# CRITICAL: Label the secret or ArgoCD ignores it
kubectl label secret argocd-repo-k8s-gitea -n argocd \
  argocd.argoproj.io/secret-type=repository --overwrite

Pitfall: Without the argocd.argoproj.io/secret-type=repository label, ArgoCD does NOT recognize the secret as a repo credential. Symptoms: ComparisonError: failed to generate manifest: authentication required: Unauthorized even though the secret exists with correct data.

Step 2: Patch all ArgoCD Application repoURLs

# Patch root + all child apps that reference the old URL
kubectl patch application root -n argocd --type json \
  -p '[{"op":"replace","path":"/spec/source/repoURL","value":"http://git.schoen.codes/dominik/iac-homelab.git"}]'

# Patch all remaining apps referencing old URL
for app in $(kubectl get applications -n argocd -o jsonpath='{range .items[?(@.spec.source.repoURL=="http://OLD_URL")]}{.metadata.name}{"\n"}{end}'); do
  kubectl patch application $app -n argocd --type json \
    -p '[{"op":"replace","path":"/spec/source/repoURL","value":"http://git.schoen.codes/dominik/iac-homelab.git"}]'
done

Step 3: Hard refresh and verify

kubectl patch application root -n argocd --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'
# Wait 20-30s
kubectl get applications -n argocd
# Root should show Synced/Healthy
kubectl get application root -n argocd -o jsonpath='{.status.sync.revision}'
# Should show a commit hash from the new Gitea

Result: ArgoCD now pulls from K8s Gitea via internal DNS → Cilium VIP → Traefik → Gitea pod. The entire GitOps loop is self-contained in K8s. Old Git server (CT108) can be kept as fallback, then decommissioned.

4.3f Traefik Migration: LXC → K8s (2026-07-14)

Migrating reverse-proxy services from a standalone Traefik LXC (CT99999) to the RKE2-bundled Traefik DaemonSet. The LXC Traefik runs on VLAN 60 (10.0.60.10), public IP NATs to it. K8s Traefik runs as DaemonSet with hostPort 80+443 on all 6 K8s nodes.

K8s Traefik architecture:

DaemonSet rke2-traefik in kube-system (not Deployment)
hostPort: 80 and hostPort: 443 on every node
IngressClass: traefik
No certResolver configured by default — needs Let's Encrypt setup

Pattern for external backends (not in K8s):

Use ExternalName Service + IngressRoute CRD:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: grafana
  namespace: proxy
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`grafana.familie-schoen.com`)
      kind: Rule
      services:
        - name: grafana-external
          port: 3000
  tls:
    certResolver: letsencrypt
---
apiVersion: v1
kind: Service
metadata:
  name: grafana-external
  namespace: proxy
spec:
  type: ExternalName
  externalName: 10.0.30.141

HA Failover VIP via Cilium L2 (implemented 2026-07-14):

Instead of pointing the firewall NAT at a single K8s node (SPOF), a LoadBalancer Service requests a VIP from the existing Cilium LB IPAM pool (10.0.30.200–250). Cilium announces the VIP via ARP on one node; auto-failover to another node on failure (~1-2s).

Firewall NAT → 10.0.30.203 (Cilium VIP)
                    ↓ L2 Announcement (ARP)
              [aktiver K8s Node — Traefik DaemonSet]
              Automatisches Failover bei Node-Ausfall (~1-2s)

Components deployed:

clusters/main/proxy/traefik-lb-service.yaml — Service type LoadBalancer with loadBalancerIP: 10.0.30.203, selects Traefik pods
clusters/main/proxy/namespace.yaml — proxy namespace for shared IngressRoutes
clusters/main/apps/proxy.yaml — ArgoCD App for the proxy namespace

Why Cilium L2 (not MetalLB/keepalived): Cilium L2 LB IPAM is already running (3 services use it: .200 postgres, .201 hindsight, .202 gitea-ssh). No additional software needed. Pool CiliumLoadBalancerIPPool = default-pool, policy CiliumL2AnnouncementPolicy = default-l2-policy.

Verification: VIP 10.0.30.203 tested — Gitea + ArgoCD reachable through it. 404 for unmatched hosts (correct — no default route).

Firewall cutover: Change NAT target 10.0.60.10 → 10.0.30.203. Rollback: revert NAT. CT99999 stays running throughout.

Remaining migration steps:

Add Let's Encrypt certResolver via HelmChartConfig for rke2-traefik
Create IngressRoute + ExternalName Service for each external backend
Handle catchall-TCP (make specific, not HostSNI(*))
Cutover firewall NAT: 10.0.60.10 → 10.0.30.203
Verify each service, then decommission CT99999 after 1 week

Pitfall: Catchall-TCP breaks SNI routing. A HostSNI(\*`)` TCP passthrough IngressRoute intercepts ALL TLS traffic on :443, breaking HTTP IngressRoutes. Either remove the catchall or make it specific (list actual SNI hosts).

Pitfall: acme.json migration. Copy the existing Let's Encrypt acme.json from CT99999 into a K8s Secret mounted into the Traefik DaemonSet to avoid re-issuing all certificates (rate limit risk).

Pitfall: Authelia on different subnet. Backends on non-10.0.30.x subnets (e.g. Authelia at 192.168.100.11) need routing verified from K8s nodes before migration.

Reference: Full migration plan at docs/plans/2026-07-14-traefik-ct99999-k8s-migration.md in iac-homelab.

Editing Traefik dynamic config on CT99999 via SSH:

When modifying Traefik's file-provider dynamic config on the LXC host (CT99999, 10.0.60.10), YAML rules contain backtick-delimited Host() expressions. Writing these via SSH heredoc breaks the backticks (shell interprets them as command substitution). Two reliable methods:

Base64 transfer (preferred — preserves all special chars):

# Encode locally, decode on remote:
cat << 'EOF' | base64 -w0
http:
  routers:
    myapp:
      rule: "Host(`myapp.example.com`)"
      ...
EOF
# Then SSH to remote and decode:
ssh root@10.0.60.10 "echo '<BASE64>' | base64 -d > /etc/traefik/conf.d/myfile.yml"

Backup before edit, then systemctl restart traefik:

cp /etc/traefik/conf.d/explicit-http.yml{,.bak-$(date +%F)}
# Apply changes
systemctl restart traefik
# Check for YAML errors:
tail -5 /var/log/traefik/traefik.log

Pitfall: Traefik's file watcher (watch: true in static config) may not immediately detect changes made via cp overwrite. If the log still shows old errors with the same timestamp, systemctl restart traefik forces a clean reload.

Pitfall: journalctl -u traefik on CT99999 may show no recent entries even after restart — Traefik logs to /var/log/traefik/ files, not journald. Check tail /var/log/traefik/traefik.log instead.

4.3g Hermes Host Git Remote Cutover to K8s Gitea (2026-07-14)

After ArgoCD is self-managing via K8s Gitea (Section 4.3e), the Hermes host's own git operations (clone, pull, push to iac-homelab) must also switch from the old Gitea (CT108) to the K8s Gitea.

Challenge: CoreDNS overrides (Section 4.3d) are K8s-internal only. The Hermes host (outside K8s) cannot resolve git.schoen.codes via CoreDNS. Public DNS must NOT be used (user preference: internal DNS only). Solution: /etc/hosts entry on the Hermes host.

Steps:

Add /etc/hosts entry on the Hermes host:
```
echo "10.0.30.203 git.schoen.codes" | sudo tee -a /etc/hosts
```
This maps git.schoen.codes to the Traefik Cilium VIP. Traefik routes by Host header, so git clone http://git.schoen.codes/... works — Traefik sees the correct Host and forwards to the Gitea pod.

Verify reachability before changing the remote:

curl -s -o /dev/null -w "%{http_code}" \
  -H "Host: git.schoen.codes" http://10.0.30.203/
# Expected: 200 (Gitea web UI)

Switch git remote:

cd ~/iac-homelab
git remote set-url origin \
  "http://USER:TOKEN@git.schoen.codes/dominik/iac-homelab.git"
git fetch origin

Pitfall: Gitea API tokens do NOT migrate between instances. The token from CT108 (10.0.30.105:3000) is invalid on K8s Gitea. Tokens are stored hashed in the DB and tied to the instance. Must generate a new token on K8s Gitea.

Generating a new token (when kubectl is available via mgmt-runner):

# SSH to mgmt-runner (VM200), then kubectl exec into Gitea pod:
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl exec -n gitea deploy/gitea -- \
   gitea admin user generate-access-token \
   -u dominik -t hermes-host --scopes all --raw \
   --config /data/gitea/conf/app.ini"

Pitfall: Hermes host has no kubectl installed. All kubectl operations go through SSH to mgmt-runner (VM200, 10.0.30.124) — see Section 1.2. The Hermes host interacts with K8s Gitea only via HTTP through the Traefik VIP.

Pitfall: Port 3000 is NOT exposed on the Traefik VIP. The K8s Gitea is reachable from outside K8s only via Traefik on ports 80/443 (with Host: git.schoen.codes), NOT via 10.0.30.203:3000. The Gitea SSH LB (10.0.30.202:2222) is a separate Cilium LB service for git over SSH.

Pitfall: JWT (internal security token) ≠ Gitea API token. The 1Password ExternalSecrets for Gitea contain internal_token, jwt_secret, secret_key, lfs_jwt_secret — these are JWTs (three base64 segments separated by dots, e.g. eyJhbGci....eyJuYmYi....Lyr6fqc...), NOT API access tokens (which are hex strings like 07efa534ea1a...). A JWT returns 401 Unauthorized when used as Authorization: token <JWT> or Authorization: Bearer <JWT>. If a user provides a dotted base64 token, it is a security token — generate a real API token via gitea admin user generate-access-token instead.

Pitfall: ArgoCD repo secret also needs the new token. After generating a new Gitea API token for the Hermes host, the ArgoCD repository secret (argocd-repo-k8s-gitea) must also be patched with the same token, otherwise ArgoCD's repo connection breaks:

kubectl patch secret argocd-repo-k8s-gitea -n argocd --type merge \
  -p '{"data":{"password":"$(echo -n TOKEN | base64)"}}'

1Password Storage of Gitea Credentials:

After generating the token and identifying the admin password, store both in 1Password vault "Hermes":

# API token (two-step — see 1password-cli skill for the credential
# field pitfall with API Credential category):
op item create --category="API Credential" \
  --title="K8s Gitea Token (hermes-gitops)" --vault="Hermes" \
  username="dominik" url="http://git.schoen.codes"
op item edit "<ITEM_ID>" --vault="Hermes" \
  credential="TOKEN_VALUE" hostname="git.schoen.codes"

# Admin login (Login category — one step works):
op item create --category="Login" \
  --title="K8s Gitea Admin (dominik)" --vault="Hermes" \
  username="dominik" password="ADMIN_PASSWORD" \
  --url="http://git.schoen.codes"

Reference: references/hermes-git-remote-cutover-2026-07.md

4.3i InfluxDB K8s Migration: Fresh Deploy + Replication (2026-07-14)

Migrating InfluxDB v2 from LXC CTs (CT109 hot + CT134 archive) to a single K8s StatefulSet. Design was developed through the Compound Engineering brainstorming process (see Workflow Correction below).

Two approaches were evaluated; user chose Option B (Fresh Deploy):

Aspect	Option A: Backup/Restore	Option B: Fresh Deploy + Replication (CHOSEN)
Data transfer	`influx backup` → `influx restore --full`	`influx replication` streaming + Flux backfill
Token handling	Preserved from backup (HA only needs URL change)	New token; HA needs URL + token change
GitOps cleanliness	Restore is a one-time imperative op	Fresh deploy via GitOps, replication is declarative
Parallel operation	No (cutover gap)	Yes (both instances live until cutover)
Complexity	Lower	Higher (replication setup + backfill)

Chosen migration phases (Option B):

Deploy empty InfluxDB on K8s (StatefulSet, DOCKER_INFLUXDB_INIT_MODE=setup, Cilium L2 LB VIP .204 from existing pool)
Configure replication: influx remote create + influx replication create on CT109 → streams every new write to K8s
Backfill existing 90-day history via Flux to() with remote host
Recreate downsampling task as LOCAL on K8s (no remote to() call)
Cutover: change HA InfluxDB integration URL + token to K8s VIP
Verify: data counts match, task runs succeed, HA writes flowing
Decommission CT109 + CT134 after 1-week stable operation

Archive structure decision: single instance with local task (chosen over two-instance approach). Both buckets (ha_hot_90d + ha_archive_5y_5m) on one K8s instance. Downsampling task runs locally (reads hot bucket, writes archive bucket — same instance). Rationale: 4 GB total data, ~5 MB/day growth — splitting into two instances adds backup complexity without meaningful risk reduction. PVC sized at 30 Gi (hot 4 GB + archive growth + headroom).

Backup strategy for K8s instance: Single daily influx backup CronJob → Norris S3 (same pattern as CNPG/Postgres). One backup, one restore — covers both buckets. Ceph RBD replica-3 + S3 offsite = double protection.

Manifest structure (GitOps):

clusters/main/
  apps/influxdb.yaml          → ArgoCD Application (App-of-Apps child)
  influxdb/
    namespace.yaml             → ExternalSecret for admin creds (1Password)
    statefulset.yaml           → InfluxDB v2.7, init mode, PVC on ceph-hdd-replica
    service.yaml               → LoadBalancer with Cilium L2 LB IPAM annotation
    init-configmap.yaml        → Post-deploy init script (create archive bucket + task)
    init-job.yaml              → ArgoCD PostSync hook Job to run init script

Pitfall: DOCKER_INFLUXDB_INIT_MODE=setup runs only on first boot. If the PVC already has data, the init mode is skipped. The PostSync init Job handles post-deploy configuration (archive bucket creation, task setup) idempotently.

Pitfall: 1Password CLI may not be authenticated on the Hermes host. OP_SERVICE_ACCOUNT_TOKEN can be set but op account list returns []. ExternalSecrets must be created via the K8s service account token (§5.3) or the 1Password Web UI in the "Kubernetes ESO" vault.

Pitfall: Downsampling Task must filter numeric fields only. Home Assistant writes string fields (icon_str, state_class_str, attribution_str) alongside numeric value fields. aggregateWindow(fn: mean) crashes on strings with unsupported input type for mean aggregate: string. Fix: add |> filter(fn: (r) => r._field == "value") before aggregation.

Pitfall: CT134 archive was effectively dead (368 KB, no new data for 12 days). The downsampling task on CT109 was failing silently — it appeared to run successfully but the 91-90 day window was empty (hot bucket only 76 days old). The task crash on string fields had been masked until the window had data.

Workflow Correction: Compound Engineering brainstorming-first (2026-07-14). User explicitly corrected the approach of jumping to execute_code and writing manifests before designing the migration concept. For infrastructure migrations, ALWAYS follow the Compound Engineering process:

Load the brainstorming skill FIRST
Explore project context + grounding scan (docs/solutions/, Hindsight)
Ask clarifying questions ONE AT A TIME (multiple choice preferred)
Propose 2-3 approaches with trade-offs
Present design incrementally, get approval
Write design doc
Transition to writing-plans skill for implementation plan Do NOT write manifests, execute code, or take implementation action until the design is approved. Even if the manifests seem straightforward, the structured exploration reveals design decisions (backup strategy, token handling, archive structure) that would otherwise be discovered too late.

Pitfall: ArgoCD ServerSideApply does NOT update ConfigMaps with embedded scripts. When a ConfigMap contains a shell script (e.g. init.sh in data), and the script is modified in Git, ArgoCD with ServerSideApply=true may show OutOfSync/Healthy but NOT apply the updated ConfigMap content. The live ConfigMap retains the old script. Workarounds:

kubectl delete cm <name> -n <ns> + hard refresh — ArgoCD recreates from Git
Execute the updated script manually via kubectl exec (faster for one-time fixes)
Consider splitting scripts into separate files mounted via a projected volume

Pitfall: influx task create --file requires option task = {} header in Flux. Without option task = { name: "task-name", every: 1h } at the top of the Flux script, influx task create fails with 400 Bad Request: invalid options: no task options defined. The --name and --every CLI flags set metadata but do NOT inject the option task block into the Flux script itself — the script must declare it explicitly.

Pitfall: InfluxDB replication CLI flags use -id suffix, not bare names. influx replication create requires --local-bucket-id (NOT --local-bucket) and --remote-bucket-id (NOT --remote-bucket). Passing the bare flag produces flag provided but not defined: -local-bucket.

Pitfall: Flux to() backfill produces surprisingly small disk footprint. A 90-day backfill of 4.25 million rows via from() |> range() |> to(host: remote) produced only 51 MB on the destination InfluxDB (vs 4.1 GB on the source). This is normal — InfluxDB v2's columnar compression is extremely effective on fresh shards without historical shard overhead. Verify success by counting rows, not checking disk size:

from(bucket: "ha_hot_90d") |> range(start: -90d) |> count() |> sum() |> group() |> sum()

Pitfall: PostSync Hook Job does NOT re-run when only the ConfigMap changes. ArgoCD's PostSync hook with BeforeHookCreation deletes the old Job and creates a new one on each sync. BUT if the Application is OutOfSync due to the ConfigMap diff but ArgoCD doesn't actually apply the ConfigMap (see ServerSideApply pitfall above), the new Job still uses the OLD ConfigMap content. The Job "succeeds" with stale data. Fix: manually delete the ConfigMap, force refresh, then manually trigger the Job or run the init script directly via kubectl exec.

Pitfall: Backup CronJob requires multi-container pattern (init + upload). The influxdb:2.7 image has NO python3, aws, rclone, or working apt-get (apt-get install hangs/crashes). The amazon/aws-cli:2 image tag does NOT exist — must use a specific version like amazon/aws-cli:2.35.22. The amazon/aws-cli image has NO tar command. Solution: use init-container (influxdb image) for influx backup + tar czf, then a separate container (aws-cli image) for aws s3 cp:

initContainers:
  - name: backup
    image: influxdb:2.7
    command: [/bin/sh, -c, |
      influx backup /shared/backup --host http://influxdb:8086 \
        --token "${DOCKER_INFLUXDB_INIT_ADMIN_TOKEN}" --org homelab
      cd /shared && tar czf backup.tar.gz backup/]
    # ...
containers:
  - name: upload
    image: amazon/aws-cli:2.35.22
    command: [/bin/sh, -c, |
      aws s3 cp /shared/backup.tar.gz s3://BUCKET/$(date +%Y-%m-%d)/backup.tar.gz \
        --endpoint-url https://rgw.nbg.nsc.noris.cloud --region nsc-nbg]
    # ...

Pitfall: AWS CLI requires UPPERCASE env vars. Using envFrom with a secret containing lowercase keys (aws_access_key_id, aws_secret_access_key) does NOT work — AWS CLI looks for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Must use explicit env with valueFrom mapping:

env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: influxdb-s3-backup
        key: aws_access_key_id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: influxdb-s3-backup
        key: aws_secret_access_key

Pitfall: S3 bucket must be created before first backup run. The CronJob fails if the bucket doesn't exist. Create it via a one-off pod. **CRITICAL: aws s3 mb with --region nsc-nbg fails with Pitfall: S3 bucket must be created before first backup run. The CronJob fails with argument of type 'NoneType' is not a container or iterable if the bucket doesn't exist. Create it via a one-off pod.

CRITICAL: aws s3 mb requires --region us-east-1, NOT nsc-nbg. Using --region nsc-nbg fails with InvalidLocationConstraint: The nsc-nbg location **Pitfall: S3 bucket must be created before first backup run.** The CronJob fails with argument of type 'NoneType' is not a container or iterable` if the bucket doesn't exist. Create it via a one-off pod:

kubectl run aws-cli-create-bucket --image=amazon/aws-cli:2.35.22 --restart=Never \
  --env=AWS_ACCESS_KEY_ID=$AKI --env=AWS_SECRET_ACCESS_KEY=$SAK \
  --command -- sh -c "aws s3 mb s3://homelab-influxdb-backup \
    --endpoint-url https://rgw.nbg.nsc.noris.cloud --region us-east-1; sleep 5"

Pitfall: aws s3 mb with --region nsc-nbg fails with InvalidLocationConstraint. Norris S3 (rgw.nbg.nsc.noris.cloud) rejects the nsc-nbg location constraint for bucket creation, even though aws s3 cp works fine with --region nsc-nbg for uploads/downloads. For s3 mb only, always use --region us-east-1 (the AWS default region suppresses the LocationConstraint header). This affects all Norris S3 bucket creations, not just InfluxDB.

Pitfall: Home Assistant OS SSH is on port 22222, not 22. HA OS (qemux86-64 image) exposes SSH on port 22222 for root. The HA configuration file is at /mnt/data/supervisor/homeassistant/configuration.yaml (NOT /config/). Restart HA via docker restart homeassistant from the SSH session. The HA Supervisor access token is in /mnt/data/supervisor/homeassistant.json (field access_token) but the Supervisor REST API returns 404 for /api/supervisor/core/restart — use docker restart instead.

Reference: references/influxdb-k8s-migration-2026-07.md

4.3j Backup CronJob for Services with PVC + External DB (2026-07-14)

When backing up a K8s service that has BOTH PVC-mounted data AND an external database (e.g. Gitea with Galera), use a multi-initContainer CronJob with dedicated RBAC for kubectl exec/kubectl cp.

Architecture (Gitea example):

initContainer 1 (gitea-files):  kubectl exec tar /data → kubectl cp → emptyDir
initContainer 2 (gitea-db):    mysqldump against Galera → emptyDir
container (upload):             aws s3 cp both files → Norris S3

RBAC required: The backup pod needs a ServiceAccount with a Role granting pods/exec and pods/get in the target namespace — without this, kubectl exec and kubectl cp fail with forbidden:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: SERVICE-backup
  namespace: NAMESPACE
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "get"]

Bind via RoleBinding to a ServiceAccount, then set serviceAccountName on the CronJob pod spec.

File backup via kubectl exec + kubectl cp:

Since the PVC is RWO and mounted by the running pod, you can't mount it directly in the backup pod. Instead:

kubectl exec into the running pod to tar czf /tmp/data.tar.gz
kubectl cp to extract the archive to the shared emptyDir
kubectl exec to clean up the temp file

initContainers:
  - name: gitea-files
    image: bitnami/kubectl:1.31
    command: [/bin/sh, -c, |
      kubectl exec -n gitea gitea-0 -- tar czf /tmp/gitea-data.tar.gz -C / data
      kubectl cp gitea/gitea-0:/tmp/gitea-data.tar.gz /shared/gitea-data.tar.gz
      kubectl exec -n gitea gitea-0 -- rm -f /tmp/gitea-data.tar.gz]
    volumeMounts:
      - name: shared
        mountPath: /shared

DB backup via mysqldump against Galera:

Use the mysql:8.0 image with DB credentials from an existing K8s secret (e.g. gitea-db):

initContainers:
  - name: gitea-db
    image: mysql:8.0
    command: [/bin/sh, -c, |
      mysqldump --host=10.0.30.70 --port=3306 --user=gitea \
        --password="${GITEA_DB_PASS}" --single-transaction \
        --routines --triggers gitea | gzip > /shared/gitea-db.sql.gz]
    env:
      - name: GITEA_DB_PASS
        valueFrom:
          secretKeyRef:
            name: gitea-db
            key: password
    volumeMounts:
      - name: shared
        mountPath: /shared

S3 credentials reuse across backup CronJobs: Multiple backup CronJobs can share the same Norris S3 account credentials (from the postgres-s3-backup 1Password item). Create a new ExternalSecret per namespace pointing to the same 1Password item keys, and a new S3 bucket per service. Bucket naming: homelab-<service>-backup (e.g. homelab-gitea-backup, homelab-influxdb-backup). Each bucket must be created via aws s3 mb before the first backup run (see §4.3i pitfall).

Schedule staggering: Offset backup schedules to avoid concurrent S3 uploads. InfluxDB at 0 3 * * * (03:00 UTC), Gitea at 30 3 * * * (03:30 UTC).

Pitfall: bitnami/kubectl:1.31 tag does NOT exist. Use full semver (1.31.4). Even then, Docker Hub rate-limiting causes 3+ min pulls. alpine/k8s:1.31.4 same issue. When kubectl images won't pull, run the backup from mgmt-runner (VM200) which has kubectl + mysqldump locally — see references/gitea-backup-cronjob-2026-07.md.

Pitfall: kubectl cp fails for large files (>~50 MB). WebSocket resets: connection reset by peer, close 1006 abnormal closure. Small files work. For large PVC tars, run from mgmt-runner directly.

Pitfall: VM200 repo remote may be stale after Gitea migration. Still points to CT108. git pull succeeds but new commits from K8s Gitea never arrive. Fix: git remote set-url origin <new-url> on VM200.

Pitfall: ArgoCD "successfully synced" may NOT create new resources. With ServerSideApply=true, new files can go undetected. Hard-refresh twice or kubectl apply -f manually. Verify with kubectl get.

Pitfall: hostPath volumes are node-local. Pod on worker-01 can't read mgmt-runner's /tmp. Use emptyDir + kubectl cp (small files) or nodeName: pinning (fragile). contain kubectl.

Pitfall: kubectl cp is unreliable for large files (>~50 MB). The websocket connection used by kubectl cp resets before large transfers complete (write: connection reset by peer, websocket: close 1006). Small files (<1 MB) work fine. For large PVC backups (e.g. 143 MB Gitea data), avoid kubectl cp — generate the tar directly in an initContainer that shares an emptyDir with the upload container, or run the backup from the mgmt-runner which has kubectl locally.

Pitfall: bitnami/kubectl:1.31 image tag does NOT exist. The bitnami/kubectl image uses full semver tags (e.g. 1.31.4), not minor-version tags like 1.31. Even with correct tags, Docker Hub rate-limiting causes extremely slow pulls (3+ min). When kubectl images won't pull, run backup from mgmt-runner (VM200) which has kubectl locally. See references/gitea-backup-cronjob-2026-07.md. Solutions:

Pre-pull the image on all worker nodes: crictl pull alpine/k8s:1.31.4
Use an image from a non-Docker-Hub registry
Run the backup from the mgmt-runner (VM200, 10.0.30.124) which already has kubectl + mysqldump installed — see references/gitea-backup-cronjob-2026-07.md for the manual fallback script. This is the fastest workaround when image pulls are blocked.

Pitfall: kubectl cp is unreliable for files >50 MB. The K8s API websocket connection resets before large file transfers complete. Small files (<1 MB) work fine. For large PVC backups (e.g. 143 MB Gitea data), avoid kubectl cp between pods — generate the data in the same pod that uploads it, or run from the mgmt-runner where kubectl cp writes to local disk (more reliable than pod-to-pod transfer).

Pitfall: hostPath volumes are node-local. A pod using hostPath: /tmp reads from the NODE it's scheduled on, not from the machine that created the files. If files are on the mgmt-runner but the pod runs on a worker, the files are invisible. Use emptyDir + kubectl cp, or run the backup from the mgmt-runner directly.

Manifest files (Gitea backup):

clusters/main/gitea/backup-rbac.yaml — SA + Role + RoleBinding
clusters/main/gitea/backup-s3-secret.yaml — ExternalSecret (Norris S3)
clusters/main/gitea/backup-cronjob.yaml — CronJob (3 containers)

Reference: references/gitea-backup-cronjob-2026-07.md

When the user asks "what's the next workload to migrate to K8s?", use this structured assessment methodology. The user migrates LXC/VM workloads to K8s incrementally — this pattern recurs for each wave.

Step 1: Inventory all running workloads across all PVE nodes

for node_ip in 10.0.20.10 10.0.20.20 10.0.20.30 10.0.20.40 10.0.20.50 10.0.20.60; do
  echo "=== $node_ip ==="
  ssh -i ~/.ssh/id_ed25519_proxmox -o ConnectTimeout=3 root@$node_ip \
    "qm list 2>/dev/null; pct list 2>/dev/null" 2>/dev/null
done

Focus on running CTs/VMs only — stopped ones are already decommissioned candidates. Note the PVE node, CT/VM ID, hostname, and resource config (cores, memory, disk).

Step 2: Check K8s resource availability

ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl top nodes 2>&1"

Workers have the most headroom (typically 80%+ free CPU/memory). CP nodes are ~50-60% utilized. Ensure the target workload fits.

Step 3: Check what's already in K8s

ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl get deploy,sts,svc -A \
   --no-headers 2>&1 | grep -v 'kube-system\|argocd\|cattle\|cilium'"

Avoid duplicating services already running in K8s.

Step 4: Evaluate each candidate against criteria

Criterion	Question
Helm chart	Does an official/maintained Helm chart exist?
Statefulness	Stateless or simple state (single PVC) = easier
Dependencies	Does K8s Traefik/other services depend on it?
Resource footprint	Fits within worker node capacity?
Risk	What breaks if migration fails? (Auth-only = low, data = high)
Value	How much operational complexity does migration eliminate?

Step 5: Rank by dependency urgency + simplicity

Prioritize workloads that unblock other migrations. Example: Authelia must migrate before Traefik can fully move to K8s (Authelia is the auth backend for every routed service). Simple, low-risk workloads that eliminate LXC sprawl are good filler between complex migrations.

Migration wave pattern:

Wave	Workload	Rationale
1	Dependency services (Auth, DNS)	Unblocks proxy layer
2	Proxy/Ingress layer	All external traffic through K8s
3	Observability (InfluxDB, Grafana)	Simple Helm charts
4	High-value complex stacks (Immich)	Multi-service but highest value
5	Low-priority containers	Cleanup

Present results as a table with: CT/VM ID, name, node, status, K8s-fit rating (⭐), and a one-line recommendation. End with a ranked recommendation and a brief rationale — the user decides what to tackle.

Pitfall: "Running" ≠ "In use". Before recommending a service for migration, verify it's actually being used. A service can be running but completely dormant:

Check reverse proxy config — grep for the service name as a middleware (e.g. forwardAuth referencing Authelia). If no router references it as a middleware, it's not protecting anything.
Check data store freshness — ls -la on the DB file (SQLite) or query row counts. A 304 KB SQLite DB unchanged since August 2025 = abandoned.
Check domain currency — if the service is configured for an old domain (e.g. familie-schoen.com when the active domain is schoen.codes), it's likely orphaned.
Check registered users — if only an admin + one user exist and neither has logged in recently, the service is decorative.

Example: Authelia (CT112) was running, had 2 users, TOTP configured, Traefik route existed — but NO router used it as a forwardAuth middleware, the SQLite DB was stale, and the domain was old. Decision: stop and decommission instead of migrating. When a service is unused, stopping the CT (not migrating) is the correct action.

4.4 Removing an Application / Operator via GitOps

When removing an operator-managed application (e.g. MariaDB Operator + its MariaDB/MaxScale CRs) from ArgoCD:

Pitfall: Simply deleting the YAML files from the repo and pushing does NOT cleanly prune operator-managed resources. ArgoCD marks the app OutOfSync but the operator's Custom Resources (CRs) hold finalizers that prevent deletion. The operator pods must be running to process the finalizer — but if you delete the operator first, the CRs are orphaned and the namespace hangs in Terminating forever.

Correct removal sequence:

Remove YAML files from repo (commit + push to Gitea)

Manually delete the CRs first (while operator is still running):

kubectl delete mariadb mariadb-galera -n mariadb --timeout=120s
kubectl delete maxscale mariadb-maxscale -n mariadb  # may 404 if already gone

Wait for CRs to be fully deleted (operator processes finalizers)

Delete the namespace:

kubectl delete namespace mariadb --timeout=30s
kubectl delete namespace mariadb-operator --timeout=30s

Delete the CRDs (ArgoCD may not prune these automatically):

kubectl delete crd backups.k8s.mariadb.com connections.k8s.mariadb.com \
  databases.k8s.mariadb.com grants.k8s.mariadb.com \
  mariadbs.k8s.mariadb.com maxscales.k8s.mariadb.com \
  restores.k8s.mariadb.com sqljobs.k8s.mariadb.com \
  users.k8s.mariadb.com --timeout=30s

Force ArgoCD refresh if apps linger:

kubectl annotate application root -n argocd \
  argocd.argoproj.io/refresh-options=hard-refresh --overwrite

Key insight: ArgoCD prune: true removes standard K8s resources (deployments, services, configmaps) automatically, but operator CRDs and their CR instances often need manual cleanup because finalizers require the operator to be alive to process them.

4.4b Removing a Non-Operator Application via GitOps

For standard apps (Deployments, Services, StatefulSets without operator CRs), removal is simpler — no finalizer issues:

Remove YAML files from repo (ArgoCD Application manifest + entire clusters/main/<service>/ directory + any GitHub workflows):

cd /root/iac-homelab
rm clusters/main/apps/<service>.yaml
rm -rf clusters/main/<service>/
rm .github/workflows/<service>-*.yml  # if applicable
git add -A && git commit -m "chore: remove <service> (unused)"
git push origin main

Delete the ArgoCD Application (cascade=foreground cleans up managed resources):
```
kubectl delete application <name> -n argocd --cascade=foreground
```
Delete the namespace (if not auto-pruned — ArgoCD's CreateNamespace=true creates it but doesn't always prune it):
```
kubectl delete ns <namespace>
# May take 10-30s if pods need to terminate first
```
Verify: kubectl get ns <namespace> → NotFound, kubectl get pods -A | grep <namespace> → empty.

Pitfall: ArgoCD Application deletion with --cascade=foreground may leave the namespace in Terminating if pods have stuck finalizers. The standard kubernetes finalizer resolves once pods are evicted. If it hangs, check for leftover ExternalSecrets or PVCs with custom finalizers.

4.5 ArgoCD Pod States During Cold Boot

After cluster cold-boot, ArgoCD pods show mixed states:

Some Running (new replicas scheduled by deployment controller)
Some Terminating (old replicas from before shutdown)
Some Unknown (node was down when status was last reported)

These resolve naturally as the deployment controller reconciles. No manual intervention needed unless pods are stuck for 10+ minutes.

4.6 Deploying Python Apps with pip in Init Containers

When deploying a Python app that isn't available as a Docker image (e.g. hindsight-api from PyPI), the common pattern is an init container that runs pip install, storing packages in an emptyDir volume shared with the main container.

Pitfall: pip install --target= does NOT install console_scripts — Using pip install --target=/opt/packages copies the Python modules but does NOT create bin/ entry points (console_scripts). The main container will fail with command not found even though the package files are present. Fix: Use python -m venv instead:

initContainers:
  - name: install-deps
    image: python:3.11-slim
    command: ["/bin/sh", "-c"]
    args:
      - |
        python -m venv /opt/venv &&
        /opt/venv/bin/pip install --no-cache-dir <package> &&
        echo "Done"
    volumeMounts:
      - name: venv
        mountPath: /opt/venv
containers:
  - name: app
    image: python:3.11-slim
    command: ["/bin/sh", "-c"]
    args: ["/opt/venv/bin/<binary> --host 0.0.0.0 --port 8888"]
    volumeMounts:
      - name: venv
        mountPath: /opt/venv
        readOnly: true
volumes:
  - name: venv
    emptyDir:
      sizeLimit: 3Gi

Pitfall: Heavy dependencies pull CUDA torch (526MB) — Packages that depend on torch (e.g. sentence-transformers, hindsight-api) will pull the full CUDA-enabled torch wheel by default, which is 526MB+ and may OOMKill the init container. Fix: Install CPU-only torch first with a dedicated index, then install the main package with --extra-index-url so pip finds CPU torch:

args:
  - |
    /opt/venv/bin/pip install --no-cache-dir \
      torch --index-url https://download.pytorch.org/whl/cpu &&
    /opt/venv/bin/pip install --no-cache-dir \
      <package-with-torch-dep> \
      --extra-index-url https://download.pytorch.org/whl/cpu &&
    echo "Done"

The --index-url for the first install ensures ONLY CPU wheels are considered. The --extra-index-url for the second install allows pip to find the already-satisfied CPU torch instead of upgrading to the default CUDA version.

Pitfall: Init container pip install is slow (~10min) — Installing 100+ packages (including torch, transformers, sentence-transformers) takes 5-10 minutes. The pod stays in Init:0/1 during this time. Set generous timeouts and don't assume CrashLoopBackOff means failure — check the init container logs first.

Section 5: External Secrets & 1Password

5.1 ClusterSecretStore

The cluster uses a single ClusterSecretStore backed by 1Password:

kubectl get clustersecretstore onepassword-store
# Should show READY=True

5.2 ExternalSecret Key Format (CRITICAL)

The remoteRef.key format is item-title/field-name — a simple slash-separated path, NOT op:// URIs and NOT a separate property field.

# CORRECT
spec:
  data:
    - secretKey: api-key
      remoteRef:
        key: noris-api-key/credential    # item-title/field-name

# WRONG — will fail with "secret reference has invalid format"
spec:
  data:
    - secretKey: api-key
      remoteRef:
        key: noris-api-key
        property: credential

Field names depend on the 1Password item category:

API Credential: credential
Password: password
Login: username, password
Database: username, password, database, host, port

5.3 1Password Vault Mismatch (CRITICAL PITFALL)

The K8s ClusterSecretStore uses a different 1Password service account than the local op CLI on the Hermes host. Items created with the local op CLI land in vault Hermes (5ythsz37hf3xminhq33drg55tm), but the K8s ESO reads from vault Kubernetes ESO (334ykdtj5kar3jlpcrztjvx2fu).

Symptom: ExternalSecret shows SecretSyncedError: no vault matched the secret reference query even though the item exists in 1Password.

Diagnosis:

# Check which vault the K8s ClusterSecretStore uses
kubectl get clustersecretstore onepassword-store -o jsonpath='{.spec.provider.onepasswordSDK.vault}'

# Check which vaults your local op CLI can see
op vault list

# Check which vaults the K8s service account can see
K8S_TOKEN=$(kubectl get secret onepassword-token -n external-secrets -o jsonpath='{.data.token}' | base64 -d)
OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op vault list

Fix: Create 1Password items in the vault that the K8s service account can see (vault 334ykdtj5kar3jlpcrztjvx2fu = "Kubernetes ESO").

The K8s service account token CAN create items (not read-only as previously assumed):

K8S_TOKEN=$(kubectl get secret onepassword-token -n external-secrets \
  -o jsonpath='{.data.token}' | base64 -d)
OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op item create \
  --category="API Credential" \
  --title="item-name" \
  --vault="334ykdtj5kar3jlpcrztjvx2fu" \
  credential=SECRET_VALUE

OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op item create \
  --category="Password" \
  --title="item-name" \
  --vault="334ykdtj5kar3jlpcrztjvx2fu" \
  password=SECRET_VALUE

Alternatively, create items via the 1Password Web UI in the correct vault, or move items from the Hermes vault to Kubernetes ESO.

5.4 Common External Secret Failures

Error	Cause	Fix
`rate limit exceeded`	1Password API throttling	Wait + retry (auto-resolves)
`no vault matched the secret reference query`	Item in wrong vault (see §5.3)	Create item in "Kubernetes ESO" vault
`secret reference has invalid format`	Wrong key format (see §5.2)	Use `item-title/field-name` format
`more than one field matched the secret reference`	Duplicate items in 1Password vault	Deduplicate items
`dial tcp: lookup my.1password.com: connect: operation not permitted`	DNS resolution from pod network	Check CoreDNS, Cilium policies

5.3 Checking External Secret Status

# All ExternalSecrets across cluster
kubectl get externalsecrets -A

# Detailed status for a failing secret
kubectl describe externalsecret -n NAMESPACE SECRET_NAME

5.5 StorageClasses Available

The cluster has two StorageClasses (both Ceph RBD via CSI):

StorageClass	Provisioner	Tier	Use Case
`ceph-flash` (default)	rbd.csi.ceph.com	SSD/NVMe	Databases, hot workloads
`ceph-hdd-replica`	rbd.csi.ceph.com	HDD	Bulk storage, archives

Pitfall: There is NO ceph-rbd StorageClass. Using it in a PVC causes the pod to stay Pending indefinitely with no PV created. Always use ceph-flash or ceph-hdd-replica.

Section 6: Nested Proxmox SSH Patterns

6.1 Discovering Which PVE Node Hosts a VM

PVE HA migrates VMs between nodes. Before operating on a VM, find which node it's on:

for node in proxmox1 proxmox2 proxmox3 proxmox4 proxmox5 proxmox6 proxmox7; do
  ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
    "ssh -o ConnectTimeout=3 root@$node 'qm list 2>/dev/null | grep VMID'" 2>/dev/null
done

6.2 Executing Commands on VMs via qm guest exec

ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "qm guest exec VMID -- sh -c 'command here'"

Output is JSON: {"exitcode": N, "out-data": "...", "err-data": "..."}

Pitfall: Complex shell commands with quotes break through nested SSH → qm guest exec. Use base64 encoding for file writes (see docker-host-administration Section 8.4).

Pitfall: qm guest exec requires the QEMU guest agent to be running. Freshly booted VMs may take 30-60s for the agent to start.

6.3 Parsing qm guest exec JSON Output

# Pipe through python3 for clean extraction
ssh ... "qm guest exec 200 -- sh -c '...'" 2>&1 | \
  python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('out-data',''))"

Section 7: IaC Repository Structure

7.1 dominik/iac-homelab (Gitea)

iac-homelab/
├── epic-1-management/         # Mgmt VM (OpenTofu + Ansible)
├── epic-2-k8s/                # 6 VMs + RKE2 bootstrap (Ansible playbook)
├── epic-3-networking-secrets/ # Cilium + External Secrets (OpenTofu)
├── epic-4-storage/            # Ceph CSI (OpenTofu)
├── epic-5-gitops/             # ArgoCD + Root App (OpenTofu)
├── epic-7-mariadb-vm/         # MariaDB Galera VMs (standalone, not K8s)
├── clusters/main/apps/        # ArgoCD App-of-Apps manifests
├── clusters/main/operators/   # Operator Helm releases
├── clusters/main/databases/   # Database cluster manifests
├── clusters/main/backups/     # Velero config
└── .github/workflows/         # CI/CD pipelines (self-hosted runner)

7.2 Gitea Token Generation

When 1Password op --reveal fails (service account limitation), generate a new Gitea API token directly on the Gitea CT:

# Gitea runs as user 'gitea', binary at /usr/local/bin/gitea
# Config at /etc/gitea/app.ini, DB at /var/lib/gitea/data/gitea.db
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "pct exec GITEA_CT -- su -s /bin/sh gitea -c \
   'GITEA_WORK_DIR=/var/lib/gitea /usr/local/bin/gitea \
    admin user generate-access-token \
    -u dominik -t TOKEN_NAME --scopes all --raw \
    --config /etc/gitea/app.ini'"

Pitfall: Gitea 1.25+ uses generate-access-token (hyphenated), not generate accesstoken. Run gitea admin user --help first to verify the subcommand name for your version.

Pitfall: Gitea refuses to run as root. Must su -s /bin/sh gitea and set GITEA_WORK_DIR environment variable.

7.3 Existing Gitea Tokens

Tokens are stored hashed in the Gitea SQLite DB — the full token value cannot be recovered. Only the last 8 characters are visible:

SELECT id, uid, name, token_last_eight FROM access_token;

If a token is lost, generate a new one (Section 7.2).

Section 8: Health Diagnostics

8.1 Full Cluster Health Check

KUBECONFIG=/root/.kube/config

# Nodes
kubectl get nodes -o wide

# Non-running pods (problems)
kubectl get pods -A | grep -v Running | grep -v Completed

# ArgoCD app sync status
kubectl get applications -n argocd

# External secrets
kubectl get externalsecrets -A

# LoadBalancer services
kubectl get svc -A -o wide | grep LoadBalancer

# Cilium LB pool
kubectl get ciliumloadbalancerippool

8.2 Common Cold-Boot Issues

Issue	Resolution
Workers NotReady for 2+ min	Check `journalctl -u rke2-agent` on worker — usually waiting for serving-kubelet.crt (503 from CP)
ceph-csi CrashLoopBackOff	Ceph cluster may not be reachable from all nodes; check Ceph mon addresses
External Secrets SecretSyncedError	1Password connectivity — check if `onepassword-store` ClusterSecretStore is Ready
ArgoCD pods Terminating	Old replicas cleaning up — wait for deployment controller to reconcile
kube-controller-manager CrashLoopBackOff	Usually transient during etcd convergence; resolves when quorum is stable

8.3 Checking RKE2 Service Logs

# On CP nodes
qm guest exec CP_VMID -- journalctl -u rke2-server --no-pager -n 20

# On Worker nodes
qm guest exec WORKER_VMID -- journalctl -u rke2-agent --no-pager -n 20

Section 9: RKE2 Version Upgrades

9.1 GitOps-First Principle

CRITICAL: RKE2 upgrades must be performed via the IaC repo (epic-2-k8s/ansible/playbook.yml) and Ansible — NEVER by manually downloading tarballs and extracting them on individual nodes. The Ansible playbook uses the lablabs.rke2 role which handles proper installation, rolling restarts, and configuration management.

The version is controlled by a single variable in the playbook:

rke2_version: v1.35.2+rke2r1  # Change this to upgrade

9.2 Prerequisites for Ansible-Based Upgrade

Ansible role installed on mgmt-runner (VM200):

cd /root/iac-homelab/epic-2-k8s/ansible
ansible-galaxy install -r requirements.yml -p /root/.ansible/roles

Inventory generated — inventory.yml is normally generated by OpenTofu (tofu apply creates it via local_file resource). If tofu isn't available, create it manually from known IPs:

all:
  vars:
    ansible_user: debian
  children:
    k8s_cluster:
      children:
        masters:
          hosts:
            rke2-cp-01:
              ansible_host: 10.0.30.51
              rke2_type: server
              rke2_server_init: true
            rke2-cp-02:
              ansible_host: 10.0.30.52
            rke2-cp-03:
              ansible_host: 10.0.30.53
        workers:
          hosts:
            rke2-worker-01:
              ansible_host: 10.0.30.61
            rke2-worker-02:
              ansible_host: 10.0.30.62
            rke2-worker-03:
              ansible_host: 10.0.30.63

RKE2 token — extract from an existing node's config:

# Find which PVE node hosts a CP VM, then:
qm guest exec CP_VMID -- sh -c "cat /etc/rancher/rke2/config.yaml"
# Token line: token: <VALUE>
# Export before running Ansible:
export RKE2_TOKEN="<value>"

SSH access from mgmt-runner to K8s VMs — Ansible connects as debian user via SSH. The mgmt-runner's SSH public key must be in /home/debian/.ssh/authorized_keys on all 6 K8s VMs. If missing, ansible will fail with Permission denied (publickey).

Generate key on VM200 (if none exists):

ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N '' -C 'vm200-ansible'"

Distribute to all 6 K8s VMs via qm guest exec (must discover which PVE node hosts each VM first — see Section 6.1):

PUBKEY=$(ssh root@10.0.30.124 "cat ~/.ssh/id_ed25519.pub")

# For each VM, on its hosting PVE node:
ssh root@PVE_NODE "qm guest exec VMID --timeout 15 -- sh -c \
  'mkdir -p /home/debian/.ssh && echo \"$PUBKEY\" >> /home/debian/.ssh/authorized_keys \
   && chown -R debian:debian /home/debian/.ssh \
   && chmod 700 /home/debian/.ssh && chmod 600 /home/debian/.ssh/authorized_keys \
   && echo OK'"

Verify connectivity before running the playbook:

ssh root@10.0.30.124 "cd /root/iac-homelab/epic-2-k8s/ansible && ansible all -m ping"
# All 6 nodes should return SUCCESS => { "ping": "pong" }

9.3 Performing the Upgrade

# On mgmt-runner (VM200):
cd /root/iac-homelab/epic-2-k8s/ansible

# 1. Edit the version in the playbook
# Change rke2_version: v1.35.2+rke2r1 → v1.35.6+rke2r1

# 2. Commit to Git (GitOps)
git add -A
git commit -m "upgrade-rke2-v1.35.6"
git push origin main

# 3. Run the playbook
export RKE2_TOKEN="<token-from-existing-node>"
ansible-playbook playbook.yml

The playbook has TWO plays:

Play 1: Installs/upgrades RKE2 on all nodes via lablabs.rke2 role
Play 2: Rolling restart — Workers first (serial: 1), then CP nodes (serial: 1). Each node waits for RKE2 service active + node Ready before proceeding to the next.

9.4 Post-Upgrade: Transient CP Restart Failures

After the playbook completes, the PLAY RECAP may show failed=1 on a CP node even though the upgrade succeeded. This happens when Ansible's systemctl restart rke2-server races with RKE2's internal reload from the binary swap — the service had already started before Ansible tried to restart it, causing a transient "already activating" error.

Diagnosis: Check the actual node state, don't trust the PLAY RECAP alone:

# On the "failed" CP node via qm guest exec:
qm guest exec CP_VMID -- sh -c "systemctl is-active rke2-server && rke2 --version"

# Cluster-wide:
kubectl get nodes -o wide
# If node is Ready + target version → failure was benign, no action needed

Key insight: Always verify with kubectl get nodes -o wide after the playbook. If all nodes are Ready and on the target version with 0 non-running pods, the upgrade succeeded regardless of Ansible's exit code on individual nodes.

9.5 Blue-Green Alternative (User Preference)

For major version upgrades or when rolling upgrades are too risky, the user prefers blue-green:

Create new VMs with new RKE2 version (via OpenTofu + Ansible)
Join them to the existing cluster
Verify new nodes are Ready
Drain + cordon old nodes (migrate workloads)
Remove old nodes from cluster
Delete old VMs

9.6 CRITICAL PITFALL: RKE2 Tarball Extraction Destroys Debian 12

NEVER extract rke2.linux-amd64.tar.gz directly to / on a Debian 12 system. The tarball contains:

bin/    → extracted to /bin/ (OVERWRITES /bin symlink!)
lib/    → extracted to /lib/ (OVERWRITES /lib symlink!)
share/  → extracted to /share/ (creates bogus directory)

Debian 12 uses usrmerge: /bin → usr/bin, /lib → usr/lib, /sbin → usr/sbin are symlinks. The tarball extraction replaces these symlinks with real directories containing only RKE2 files, making ALL system binaries (ls, sh, bash, systemctl, mount) inaccessible. The VM becomes unmanageable — no new processes can exec, qm guest exec returns exit code 29.

Correct extraction target: /usr/local/ (the tarball's bin/ contents go to /usr/local/bin/, etc.)

If already broken, see Section 10 for recovery procedure.

Section 10: Recovering a Broken VM (Symlink Destruction)

10.1 Symptoms

qm guest exec VMID -- /bin/echo hello → exit code 29
qm guest exec VMID -- sh -c "..." → exit code 29
qm guest exec VMID -- /usr/bin/ls → exit code 29
K8s node still shows Ready (kernel + RKE2 processes still running)
But no NEW processes can start (broken /bin and /lib)

10.2 Recovery: Ceph RBD Offline Mount

When the VM's disk is on Ceph RBD (vm_disks storage pool):

# 1. Stop the VM (preserve etcd quorum if CP node — need 2/3 alive)
qm stop VMID

# 2. On the PVE node hosting the VM, map the RBD image
rbd map vm_disks/vm-VMID-disk-0
# Returns /dev/rbdN (note the number!)

# 3. Wait for udev, then check partitions
sleep 2
lsblk | grep rbd
# Typically: rbdNp1 = root fs, rbdNp14 = BIOS boot, rbdNp15 = EFI

# 4. Mount the root partition
mkdir -p /mnt/vm-rescue
mount /dev/rbdNp1 /mnt/vm-rescue

# 5. Verify the damage
ls -la /mnt/vm-rescue/bin   # Should be symlink but is real dir
ls -la /mnt/vm-rescue/lib   # Should be symlink but is real dir

# 6. Fix symlinks
# Save any RKE2 binaries first:
cp /mnt/vm-rescue/bin/rke2 /mnt/vm-rescue/usr/local/bin/rke2
cp /mnt/vm-rescue/bin/rke2-*.sh /mnt/vm-rescue/usr/local/bin/
# Copy systemd units if present:
mkdir -p /mnt/vm-rescue/usr/lib/systemd/system
cp -a /mnt/vm-rescue/lib/systemd/system/rke2-*.service /mnt/vm-rescue/usr/lib/systemd/system/
cp -a /mnt/vm-rescue/lib/systemd/system/rke2-*.env /mnt/vm-rescue/usr/lib/systemd/system/

# Remove fake directories, restore symlinks:
rm -rf /mnt/vm-rescue/bin
ln -s usr/bin /mnt/vm-rescue/bin
rm -rf /mnt/vm-rescue/lib
ln -s usr/lib /mnt/vm-rescue/lib
rm -rf /mnt/vm-rescue/share  # Bogus dir from tarball

# 7. Verify fix
ls -la /mnt/vm-rescue/bin  # Should show: bin -> usr/bin
ls -la /mnt/vm-rescue/lib  # Should show: lib -> usr/lib
ls /mnt/vm-rescue/bin/bash # Should exist
ls /mnt/vm-rescue/bin/systemctl  # Should exist

# 8. Unmount, unmap, start VM
umount /mnt/vm-rescue
rbd unmap /dev/rbdN
qm start VMID

# 9. Wait 30-60s for boot, verify guest agent
qm guest exec VMID -- sh -c "echo BOOT_OK && systemctl is-active rke2-server"

10.3 RBD Mapping Pitfalls

RBD device disappears: If rbd map succeeds but /dev/rbdN vanishes before mount, re-run rbd map (previous mapping was stale). Always sleep 2 after mapping for udev to settle.
Partition devices not created: lsblk shows partitions but /dev/rbdNp1 doesn't exist as a block device. Run partprobe /dev/rbdN or just re-map (unmap + map again).
kpartx not available: PVE nodes may not have kpartx. The kernel usually creates partition devices automatically after rbd map + sleep 2. If not, partprobe /dev/rbdN helps.
Don't mount while VM is running: RBD is exclusive — stop the VM first. Mounting a live RBD image will corrupt the filesystem.

10.4 Post-Recovery: Restore Correct RKE2 Version

If the tarball extraction was for a NEW version (e.g. v1.35.6) but the cached runtime images on the node are for the OLD version (e.g. v1.35.2), RKE2 will fail to start with:

failed to pull images: image "rancher/rke2-runtime:v1.35.6-rke2r1": not found

Fix: Download the MATCHING tarball version and extract correctly:

# On the broken VM (after symlink repair + reboot):
cd /tmp
wget https://github.com/rancher/rke2/releases/download/v1.35.2%2Brke2r1/rke2.linux-amd64.tar.gz
tar xf rke2.linux-amd64.tar.gz -C /usr/local/  # CORRECT target!
systemctl restart rke2-server

Then perform the version upgrade properly via Ansible (Section 9).

Section 11: OS Updates on K8s VMs

11.1 Updating Via qm guest exec

K8s VMs don't have SSH keys from the mgmt-runner, so Ansible can't reach them directly. Use qm guest exec through Proxmox:

# Discover which PVE node hosts each VM first (Section 6.1)
# Then run apt upgrade:
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_JUMPHOST \
  "ssh root@PVE_NODE 'qm guest exec VMID --timeout 600 -- sh -c \"
    apt-get update && apt-get upgrade -y && apt-get autoremove -y
  \"'"

11.2 Interrupted dpkg Recovery

If apt upgrade is interrupted (timeout, connection drop), dpkg is left in a half-configured state. This blocks ALL subsequent apt operations on that node — including the Ansible playbook's pre_tasks (which install ceph-common and rbd-nbd). The Ansible run will fail with:

E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a'

Simple fix (if only dpkg lock is held):

qm guest exec VMID --timeout 300 -- sh -c "dpkg --configure -a && apt-get update && apt-get upgrade -y"

Full fix (if dpkg AND debconf locks are held by stale processes):

# Via qm guest exec on the PVE node hosting the VM:
qm guest exec VMID --timeout 120 -- bash -c "
  fuser -k /var/lib/dpkg/lock-frontend 2>/dev/null;
  fuser -k /var/lib/dpkg/lock 2>/dev/null;
  fuser -k /var/cache/debconf/config.dat 2>/dev/null;
  sleep 2;
  rm -f /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock;
  DEBIAN_FRONTEND=noninteractive dpkg --configure -a &&
  apt-get install -y ceph-common rbd-nbd
"

Pitfall: debconf lock — Even after clearing dpkg locks, debconf has its own lock at /var/cache/debconf/config.dat. If a stale process holds it, dpkg --configure -a fails with:

debconf: DbDriver "config": /var/cache/debconf/config.dat is locked by another process

Must fuser -k /var/cache/debconf/config.dat before retrying.

Pitfall: Interactive prompts hang under qm guest exec — Without DEBIAN_FRONTEND=noninteractive, dpkg may prompt for config file conflicts (e.g. sshd_config was locally modified) and hang indefinitely under qm guest exec (no TTY). Always set DEBIAN_FRONTEND=noninteractive to auto-keep-existing config.

Cross-reference: This must be fixed BEFORE running the Ansible upgrade playbook (Section 9.3) — the lablabs.rke2 role's pre_tasks install ceph-common and rbd-nbd via apt, which will fail if dpkg is in an interrupted state. Fix all affected nodes first, then re-run ansible-playbook playbook.yml.

11.3 Direct SSH to mgmt-runner

VM200 (mgmt-runner) at 10.0.30.124 accepts direct SSH from the Hermes host:

ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124

This bypasses the Proxmox jumphost chain and is faster for running kubectl/helm/git commands.

Section 12: Actual VM→PVE-Node Mapping (July 2026)

The tofu state records where VMs were originally placed, but PVE HA migrates VMs. The ACTUAL mapping (as of 2026-07-12) is:

VM	Name	PVE-Node	IP
118	rke2-cp-01	proxmox5	10.0.30.51
130	rke2-cp-02	proxmox3	10.0.30.52
129	rke2-cp-03	proxmox2	10.0.30.53
128	rke2-worker-01	proxmox	10.0.30.61
132	rke2-worker-02	proxmox2	10.0.30.62
131	rke2-worker-03	proxmox3	10.0.30.63
200	mgmt-runner	proxmox3	10.0.30.124

Always re-discover before operating — HA may have moved VMs since this table was written.

Section 13: Helm Chart Upgrades

13.1 Chart Inventory & Version Matrix

Before upgrading, inventory all Helm releases and compare against latest available versions:

# On VM200 (mgmt-runner):
export KUBECONFIG=/root/.kube/config

# 1. List all Helm releases across namespaces
helm list -A

# 2. Update repo cache
helm repo update

# 3. Check latest available versions
helm search repo argo/argo-cd | head -3
helm search repo external-secrets/external-secrets | head -3
# etc.

# 4. RKE2-bundled charts (already updated with RKE2 upgrade):
helm list -n kube-system | grep rke2-
# These are: cilium, traefik, coredns, metrics-server, snapshot-controller

Build a matrix: Component | Current Version | Latest Available | Update Type (Major/Minor/Patch) | Management Method (Helm-direct vs ArgoCD GitOps).

RKE2-bundled charts (Cilium, Traefik, CoreDNS, Metrics Server, Snapshot Controller) are automatically updated when RKE2 is upgraded — no separate Helm upgrade needed.

13.2 ArgoCD GitOps-Managed Charts (CNPG, Velero)

Charts deployed via ArgoCD Applications are upgraded by changing the targetRevision in the Application manifest in the IaC repo:

cd /root/iac-homelab
# Find the Application manifest
grep -r "targetRevision" clusters/main/

# Update the chart version (e.g. CNPG 0.22.1 → 0.29.0)
sed -i 's/targetRevision: 0.22.1/targetRevision: 0.29.0/' \
  clusters/main/operators/cloudnativepg.yaml
git add -A && git commit -m "feat: upgrade CNPG chart" && git push

Pitfall: ArgoCD doesn't auto-refresh after chart version bump — Even with selfHeal: true, ArgoCD may keep the old chart revision cached. Force a hard refresh:

kubectl annotate application <app-name> -n argocd \
  argocd.argoproj.io/refresh=hard --overwrite

Wait for the new revision to appear in kubectl get applications -n argocd, then monitor the rollout:

kubectl rollout status deployment/<deployment-name> -n <namespace> --timeout=300s

13.3 Helm-Direct Charts (ArgoCD, External Secrets, Ceph CSI)

Charts installed directly via helm install are upgraded with helm upgrade:

# Save current values before upgrading
helm get values <release-name> -n <namespace> > /tmp/<release>-values.yaml

# Dry-run first
helm upgrade <release-name> <repo>/<chart> \
  --version <new-version> \
  -n <namespace> \
  --reuse-values \
  --dry-run

# Actual upgrade
helm upgrade <release-name> <repo>/<chart> \
  --version <new-version> \
  -n <namespace> \
  --reuse-values \
  --timeout 300s

Pitfall: --reuse-values may not suffice for major chart versions — Major chart versions can introduce required fields that don't exist in old values. If helm upgrade fails with a template error (e.g. nil pointer evaluating interface {}), create a complete values file that includes the new required fields:

# Instead of --reuse-values, provide explicit values:
helm upgrade <release-name> <repo>/<chart> \
  --version <new-version> \
  -n <namespace> \
  -f /tmp/<release>-v3-values.yaml \
  --timeout 300s

13.4 ArgoCD v2→v3 Major Upgrade (Breaking Change)

Upgrading the ArgoCD Helm chart from v7.x (app v2.x) to v10.x (app v3.x) requires adding redis.networkPolicy.create: false to the values file. Without this, the chart fails with:

template: argo-cd/templates/redis/networkpolicy.yaml:2:22:
nil pointer evaluating interface {}.create

Safe upgrade path: First upgrade to the latest v2.x patch (chart 7.9.1 = app v2.14.11), then jump to v3.x:

# Step 1: Safe patch to latest v2.x
helm upgrade argocd argo/argo-cd --version 7.9.1 -n argocd --reuse-values

# Step 2: Create v3-compatible values file with redis.networkPolicy
cat > /tmp/argocd-v3-values.yaml << 'EOF'
# ... copy all existing values, then add:
redis:
  networkPolicy:
    create: false
EOF

# Step 3: Major upgrade to v3
helm upgrade argocd argo/argo-cd --version 10.1.3 -n argocd \
  -f /tmp/argocd-v3-values.yaml --timeout 300s

Save values files to the IaC repo (epic-2-k8s/helm/<chart>/values.yaml) for reproducible installs.

13.5 CNPG Replica Rebuild from WAL Gap

When a CNPG PostgreSQL replica falls behind and the primary has already recycled the needed WAL segments, the replica is permanently broken:

requested WAL segment 000000020000000000000008 has already been removed

Fix: Delete the broken replica's PVCs and pod — CNPG automatically rebuilds from a fresh base backup of the primary:

# 1. Identify the broken replica pod
kubectl get pods -n postgres | grep -v Running
# e.g. postgres-main-2 is in a bad state

# 2. Delete the PVCs (data + wal)
kubectl delete pvc postgres-main-2 postgres-main-2-wal -n postgres

# 3. Delete the pod so CNPG recreates it
kubectl delete pod postgres-main-2 -n postgres

# 4. CNPG creates a new replica (e.g. postgres-main-4) automatically
# Wait for it to join:
kubectl get pods -n postgres -w
# Watch for 3/3 Running

Do this BEFORE upgrading the CNPG operator — upgrading with a broken replica is risky. The operator upgrade itself may trigger a primary restart, and having only 1 healthy replica reduces safety margin.

13.6 Ceph CSI RBD Upgrade — Transient API Server Disruption

Pitfall: Upgrading the Ceph CSI RBD chart triggers a DaemonSet rolling update of ceph-csi-rbd-nodeplugin pods across ALL nodes, including control plane nodes. The nodeplugin pods mount/unmount CSI driver sockets, which can briefly disrupt the kubelet on CP nodes and cause all 3 API servers to become unreachable for 2-5 minutes.

Symptoms during the disruption:

The connection to the server 10.0.30.51:6443 was refused

Mitigation:

Schedule Ceph CSI upgrades during maintenance windows
After helm upgrade, monitor with curl -sk https://CP_IP:6443/healthz until it responds (even 401 Unauthorized means the server is up)
Once API server recovers, verify with kubectl get pods -n kube-system | grep ceph-csi-rbd
New nodeplugin pods may stay in ContainerCreating for several minutes while pulling the new CSI image — this is normal
All PVCs remain bound throughout; no data loss occurs

13.7 ExternalSecret Perpetual OutOfSync in ArgoCD

Pitfall: The External Secrets Operator adds default fields to ExternalSecret specs that aren't present in the Git YAML:

remoteRef.conversionStrategy: "Default"
remoteRef.decodingStrategy: "None"
remoteRef.metadataPolicy: "None"
target.deletionPolicy: "Retain"
target.template.engineVersion: "v2"
target.template.mergePolicy: "Replace"

ArgoCD sees these as spec drift and marks the ExternalSecret perpetually OutOfSync. The ignoreDifferences rule for /status doesn't cover /spec differences.

Fix: Either add the ESO default fields to the Git YAML (so spec matches exactly), or add ignoreDifferences for the specific JSON pointers. Adding fields to Git YAML is cleaner:

spec:
  data:
    - secretKey: access_key_id
      remoteRef:
        key: velero-s3-backup/S3 Credentials/access_key_id
        conversionStrategy: Default
        decodingStrategy: None
        metadataPolicy: None
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    template:
      engineVersion: v2
      mergePolicy: Replace

13.8 Accidental .kube/cache Commit

Pitfall: Running kubectl commands from the IaC repo root creates .kube/cache/ with discovery API responses. If git add -A is used, these cache files get committed. Always add .kube/ to .gitignore:

echo ".kube/" >> .gitignore

References

See references/ directory for session-specific details.

references/cold-boot-recovery-2026-07.md — Session detail: full cluster cold-boot, Cilium LB pool migration from .70-.89 to .200-.250, ArgoCD convergence timeline
references/rke2-cluster-topology.md — Complete topology, IP map, operator inventory, IaC repo structure
references/hindsight-k8s-deployment-2026-07.md — Deploying hindsight-api in K8s: ExternalSecret format, StorageClass, env var ordering, vault mismatch, ArgoCD cache pitfalls
references/rke2-upgrade-and-recovery-2026-07.md — Session detail: RKE2 tarball destruction, Ceph RBD offline recovery, Ansible-based upgrade prerequisites, OS update procedures, openclaw-memory removal via GitOps, Ansible SSH key prerequisite
references/helm-chart-upgrades-2026-07.md — Session detail: full Helm chart upgrade cycle (ArgoCD v2→v3, External Secrets, Ceph CSI, CNPG, Velero), CNPG replica WAL-gap rebuild, Ceph CSI transient API server disruption, ExternalSecret OutOfSync root cause
references/gitea-k8s-migration-2026-07.md — Gitea LXC→K8s migration plan: Galera over CNPG decision, Gitea Migration API for SQLite→MySQL, chicken-egg ArgoCD cutover, git.schoen.codes external URL
references/gitea-helm-chart-v12-deployment-2026-07.md — Gitea Helm Chart v12 deployment: additionalConfigFromEnvs for DB password, valkey-cluster disabling, RWO PVC blocking, Galera schema slowness, ArgoCD app split
references/traefik-lxc-to-k8s-migration-2026-07.md — Traefik CT99999 → K8s migration: service inventory, static/dynamic config, ExternalName pattern, catchall-TCP pitfall, cutover strategy
references/argocd-self-management-cutover-2026-07.md — ArgoCD chicken-and-egg cutover: CoreDNS internal DNS override, repo secret labeling, Application URL patching, self-management verification
references/hermes-git-remote-cutover-2026-07.md — Hermes host git remote switch from CT108 to K8s Gitea: /etc/hosts for external CoreDNS, token non-migration, Traefik VIP reachability, no kubectl on Hermes host
references/influxdb-k8s-migration-2026-07.md — InfluxDB CT109+CT134 → K8s migration: fresh deploy + replication strategy (Option B chosen over backup/restore), Portainer stack discovery on 10.0.30.100, Cilium VIP .204 allocation, downsampling task string-field crash fix, Compound Engineering brainstorming-first workflow correction
references/gitea-backup-cronjob-2026-07.md — Gitea backup CronJob: multi-initContainer pattern (kubectl exec tar for PVC + mysqldump for Galera DB), RBAC for pods/exec, S3 bucket per service, schedule staggering, bitnami/kubectl image pull failures, kubectl cp large file failures, ArgoCD false sync success, VM200 stale remote, mgmt-runner fallback
references/hermes-memory-system-architecture.md — Hermes memory injection pipeline: MEMORY.md/USER.md loading, frozen snapshots, system prompt tiers, config limits, alternative context injection points (SOUL.md, AGENTS.md, .cursorrules), bitnami/kubectl:1.31 tag不存在, kubectl cp large file websocket reset, VM200 repo remote staleness, S3 --region us-east-1 for s3 mb, mgmt-runner fallback approach

98 KiB Raw Permalink Blame History Unescape Escape