hermes-skills/devops/rke2-cluster-administration/SKILL.md

---
name: rke2-cluster-administration
description: "Class-level skill for RKE2 Kubernetes cluster administration on Proxmox VE. Covers cluster lifecycle (start/stop/reboot), cold-boot recovery, Cilium LB IP pool management, ArgoCD GitOps operations, node health diagnostics, and nested Proxmox SSH patterns for K8s VM management."
version: 1.0.0
tags: [rke2, kubernetes, k8s, cilium, argocd, gitops, proxmox, loadbalancer, lb-pool, cold-boot]
---

## Overview

This skill covers administration of an RKE2 Kubernetes cluster running on
Proxmox VE QEMU VMs. The cluster uses:
- **RKE2** v1.35.x (CNCF-conformant K8s)
- **Cilium** as CNI (eBPF-based, with L2 announcements + LB IP pools)
- **ArgoCD** for GitOps (App-of-Apps pattern)
- **External Secrets Operator** synced to 1Password
- **CloudNativePG** for PostgreSQL
- **Velero** for backups
- **Ceph CSI** for RBD storage
- **Traefik** as ingress controller (RKE2-bundled)

**Note**: MariaDB Galera runs as **native VMs** (VM300-302 + MaxScale
VM310), NOT in K8s. A K8s MariaDB Operator was previously deployed but
has been removed — Galera VMs are the authoritative database.

Load this skill for any RKE2 cluster lifecycle task, node diagnostics,
ArgoCD/GitOps operations, Cilium LB pool changes, or K8s service migration.

**User preference: Always check `skills_list` for relevant skills before
starting any task.** This skill was often the right one but wasn't loaded
until late in sessions, wasting time on trial-and-error. Load it early.

**CRITICAL workflow rule (user-enforced):** Before starting ANY task on this
infrastructure, run `skills_list` and search for relevant skills FIRST. This
skill alone contains the mgmt-runner IP (10.0.30.124), SSH key paths, kubectl
location, PVE node mappings, and dozens of pitfalls — all of which are lost
time if you probe blindly without loading the skill first.

---

## Section 1: Cluster Architecture & Access

### 1.1 Cluster Topology

| Role | VMID | Node | IP | Specs |
|------|------|------|----|-------|
| CP-01 | 118 | varies (HA) | 10.0.30.51 | 4c / 12GB / 40GB |
| CP-02 | 130 | varies (HA) | 10.0.30.52 | 4c / 12GB / 40GB |
| CP-03 | 129 | varies (HA) | 10.0.30.53 | 4c / 12GB / 40GB |
| Worker-01 | 128 | varies (HA) | 10.0.30.61 | 4c / 12GB / 80GB |
| Worker-02 | 132 | varies (HA) | 10.0.30.62 | 4c / 12GB / 80GB |
| Worker-03 | 131 | varies (HA) | 10.0.30.63 | 4c / 12GB / 80GB |

**RKE2 API VIP**: 10.0.30.50 (Keepalived on CP nodes)

**Note**: PVE HA migrates VMs between nodes. Always discover which node
hosts a given VM before operating on it (see Section 6.1).

### 1.2 Management VM (kubectl access)

VM 200 (`mgmt-runner-01`) has `kubectl`, `helm`, and `git` installed.
It also serves as the self-hosted GitHub Actions runner.

**Direct SSH** from Hermes host: `ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124`
— this is the fastest path for running kubectl/helm/git commands.

**Kubeconfig location**: `/root/.kube/config` on VM 200.
Must set `KUBECONFIG=/root/.kube/config` explicitly — kubectl does not
pick it up from `~/.kube/config` via `qm guest exec`.

```bash
# Run kubectl on mgmt-runner via Proxmox
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "ssh -o ConnectTimeout=5 root@NODE_HOSTING_VM200 \
   'qm guest exec 200 -- sh -c \"KUBECONFIG=/root/.kube/config kubectl get nodes\"'"
```

**Pitfall**: `kubectl` without explicit `KUBECONFIG=` returns
`localhost:8080 connection refused` — the default kubeconfig path isn't
honored through `qm guest exec`.

### 1.3 Getting Kubeconfig from a Control Plane Node

If the mgmt-runner doesn't have kubeconfig yet:

```bash
# Get kubeconfig from CP-01 (VM 118)
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "qm guest exec 118 -- cat /etc/rancher/rke2/rke2.yaml"

# Fix the server URL from 127.0.0.1 to the CP IP
# Replace 127.0.0.1 with 10.0.30.51 (CP-01 IP)
# Base64-transfer to mgmt-runner (see docker-host-administration Section 8.4)
```

---

## Section 2: Cluster Lifecycle (Start/Stop/Reboot)

### 2.1 Starting the Cluster from Cold Boot

All 6 VMs were stopped → start them in order:

```bash
# 1. Start Control Plane nodes first
# CPs: 118, 130, 129 — may be on different PVE nodes due to HA
# Start each individually to avoid HA placement races

# 2. Wait 15-30s for CPs to begin etcd convergence
sleep 15

# 3. Start Worker nodes
# Workers: 128, 131, 132

# 4. Wait for cluster to converge (2-5 minutes)
# Workers initially show NotReady — they need to retrieve
# serving-kubelet.crt from the API server (503 until CPs are ready)
```

**Pitfall**: `qm start` returns immediately ("Requesting HA start") but
HA may place the VM on a different node than expected. Always scan all
PVE nodes to find where each VM landed:

```bash
for node in proxmox1 proxmox2 proxmox3 proxmox4 proxmox5 proxmox6 proxmox7; do
  ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
    "ssh -o ConnectTimeout=3 root@$node 'qm list | grep rke2'" 2>/dev/null
done
```

### 2.2 Cold-Boot Recovery Timeline

After all VMs are started, expect this convergence sequence:

| Time | What happens |
|------|-------------|
| 0-30s | CP nodes boot, etcd begins forming quorum |
| 30-60s | Workers connect to CP via proxy (`wss://CP_IP:9345/v1-rke2/connect`) |
| 1-2min | Workers retrieve serving-kubelet.crt (was 503 during CP convergence) |
| 2-3min | Cilium agents start on all nodes, networking comes up |
| 3-5min | ArgoCD, External Secrets, operators begin reconciling |
| 5-10min | Most pods Running, some Terminating (old replicas from before shutdown) |
| 10-15min | Databases (Galera, PostgreSQL) reach quorum and become ready |

**Patience is key.** Don't panic when pods show Unknown/Terminating
early on — this is normal cold-boot behavior, not a failure.

### 2.3 Stopping the Cluster

```bash
# 1. Stop workers first (allows graceful pod eviction)
# 2. Then stop CPs (etcd will downsize gracefully)
# 3. Verify all stopped
```

Workers stopping first prevents etcd from losing quorum abruptly.

---

## Section 3: Cilium LoadBalancer IP Pool Management

### 3.1 The LB Pool Problem

Cilium assigns LoadBalancer IPs to K8s services (Traefik, MaxScale,
PostgreSQL) from a configurable IP pool. If this pool overlaps with
existing infrastructure IPs (Galera VIPs, MaxScale VIPs, static
services), you get **IP conflicts**.

### 3.2 Checking Current Pool

```bash
KUBECONFIG=/root/.kube/config kubectl get ciliumloadbalancerippool -o yaml
```

Key fields:
```yaml
spec:
  blocks:
  - start: "10.0.30.200"   # Pool start
    stop: "10.0.30.250"    # Pool end
```

### 3.3 Changing the Pool (Free Up Conflicting IPs)

When K8s LB IPs collide with existing infrastructure:

```bash
# 1. Delete the old pool
kubectl delete ciliumloadbalancerippool default-pool

# 2. Apply new pool with non-overlapping range
cat <<'EOF' | kubectl apply -f -
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: default-pool
spec:
  blocks:
  - start: "10.0.30.200"
    stop: "10.0.30.250"
  disabled: false
EOF

# 3. Verify services picked up new IPs
kubectl get svc -A -o wide | grep LoadBalancer
```

Services automatically release old IPs and get new ones from the
updated pool. This is non-disruptive — Cilium handles the transition.

### 3.4 Making the Change Persistent

The pool is defined in a manifest on all CP nodes:
`/var/lib/rancher/rke2/server/manifests/cilium-l2-lb.yaml`

RKE2 auto-applies manifests from this directory on CP restart. If you
only patch the live CRD, the old pool comes back on next CP reboot.

**Fix the manifest on ALL 3 CP nodes** (use base64 transfer — see
docker-host-administration Section 8.4):

```bash
NEW_MANIFEST=$(cat <<'EOF'
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: default-pool
spec:
  blocks:
    - start: "10.0.30.200"
      stop: "10.0.30.250"
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: default-l2-policy
spec:
  interfaces:
    - ^eth[0-9]+
  externalIPs: true
  loadBalancerIPs: true
EOF
)

B64=$(echo "$NEW_MANIFEST" | base64 -w0)

# Write to all 3 CPs (discover which PVE node hosts each first!)
for combo in "NODE1 118" "NODE2 130" "NODE3 129"; do
  node=$(echo $combo | cut -d' ' -f1)
  vmid=$(echo $combo | cut -d' ' -f2)
  ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
    "ssh -o ConnectTimeout=5 root@$node \
     'qm guest exec $vmid -- sh -c \"echo $B64 | base64 -d > /var/lib/rancher/rke2/server/manifests/cilium-l2-lb.yaml\"'"
done
```

### 3.5 Fixing the IaC Source

The pool is also defined in the IaC repository (Ansible playbook that
bootstraps RKE2). Update and push:

```bash
# In iac-homelab repo: epic-2-k8s/ansible/playbook.yml
# Change the LB pool start/stop values to match the new range
sed -i 's/OLD_START/NEW_START/g' epic-2-k8s/ansible/playbook.yml
sed -i 's/OLD_STOP/NEW_STOP/g' epic-2-k8s/ansible/playbook.yml
git add -A && git commit -m "fix-k8s-move-cilium-lb-pool" && git push origin main
```

**Three places to update** (all must agree):
1. Live CRD (`kubectl patch` / delete+apply)
2. CP manifest files (`/var/lib/rancher/rke2/server/manifests/cilium-l2-lb.yaml`)
3. IaC repo (`epic-2-k8s/ansible/playbook.yml`)

---

## Section 4: ArgoCD GitOps

### 4.1 ArgoCD Access

- **UI**: `https://<TRAEFIK_LB_IP>/` with Host header `argocd`
  (Traefik routes by hostname)
- **Admin password**: Retrieved from kubectl:
  ```bash
  kubectl -n argocd get secret argocd-initial-admin-secret \
    -o jsonpath='{.data.password}' | base64 -d
  ```
- **CLI**: `argocd` binary (if installed on mgmt-runner)

### 4.2 App-of-Apps Pattern

The root ArgoCD Application points to a Git repo path containing
child Application manifests:

```yaml
# Root Application (synced from epic-5-gitops/tofu)
spec:
  source:
    repoURL: http://GITEA_URL/dominik/iac-homelab.git
    path: clusters/main/apps
    targetRevision: main
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
```

Child apps in `clusters/main/apps/`:
- `operators.yaml` (sync-wave 0): CloudNativePG, Velero
- `databases.yaml` (sync-wave 1): PostgreSQL
- `backups.yaml` (sync-wave 2): Velero schedules + locations

**Removed apps**: `memory.yaml` (sync-wave 3: Qdrant, Ollama, Memory API)
was removed 2026-07-12 — the `openclaw-memory` namespace is fully deleted.
See `references/rke2-upgrade-and-recovery-2026-07.md` Phase 3.

### 4.3 Adding a New Application

To deploy a new service via GitOps:

1. Create namespace manifest: `clusters/main/<service>/namespace.yaml`
2. Create deployment/service/ingress manifests in `clusters/main/<service>/`
3. Create ArgoCD Application manifest: `clusters/main/apps/<service>.yaml`
4. Push to Gitea — ArgoCD auto-syncs within minutes

**Pitfall: K8s env var interpolation order** — When using `$(VAR_NAME)`
to reference one env var inside another (e.g. building a connection
string from a secret), the referenced var MUST be listed **before** the
referencing var. K8s processes env vars top-to-bottom; if the reference
comes first, it resolves to an empty string.

```yaml
# CORRECT — HINDSIGHT_DB_PASSWORD before DATABASE_URL
env:
  - name: HINDSIGHT_DB_PASSWORD
    valueFrom:
      secretKeyRef: ...
  - name: DATABASE_URL
    value: "postgresql://user:$(HINDSIGHT_DB_PASSWORD)@host/db"

# WRONG — DATABASE_URL resolves with empty password
env:
  - name: DATABASE_URL
    value: "postgresql://user:$(HINDSIGHT_DB_PASSWORD)@host/db"
  - name: HINDSIGHT_DB_PASSWORD
    valueFrom:
      secretKeyRef: ...
```

**Pitfall: ExternalSecret API version** — The cluster's
ExternalSecrets Operator serves both `v1` and `v1beta1`, but existing
working ExternalSecrets all use `v1`. Using `v1beta1` can cause
`"the server could not find the requested resource"` errors in ArgoCD
even though the CRD technically serves both versions. Always use
`apiVersion: external-secrets.io/v1`.

**Pitfall: ArgoCD caches stale manifests** — When you fix a manifest
(e.g. wrong API version) and push, ArgoCD may keep trying the old
version. Deleting the Application and recreating it forces a clean
re-read from the repo. Alternatively:
```bash
kubectl annotate application -n argocd APP_NAME \
  argocd.argoproj.io/refresh=hard --overwrite
```

**Pitfall: StatefulSet PVC template immutability** — If a StatefulSet
has the wrong StorageClass in its `volumeClaimTemplates`, simply
updating the manifest and letting ArgoCD sync won't fix it — the old
PVC persists. Must delete the STS + PVC manually, then let ArgoCD
recreate with the corrected manifest:
```bash
kubectl delete sts STATEFULSET_NAME -n NS --cascade=foreground
kubectl delete pvc DATA_PVC_NAME -n NS
```

**Pitfall: `kubectl rollout restart` does NOT apply manifest changes**
— `rollout restart` only restarts pods with the *current* deployment
spec. It does NOT pull new manifest changes from the repo. After
pushing a manifest fix, you must `kubectl apply -f <manifest>` first
(or wait for ArgoCD to sync), THEN `rollout restart` if needed. If you
only `rollout restart`, the new pod starts with the old spec and hits
the same error.

**Pitfall: ArgoCD does not recreate manually deleted resources** —
When you `kubectl delete` a resource (e.g. an ExternalSecret) to force
re-sync, ArgoCD may report "successfully synced (all tasks run)" but
NOT recreate the deleted resource. This happens when the resource was
already synced once and ArgoCD considers it "managed" but doesn't
detect the drift. Fix: `kubectl apply -f <manifest>` manually, or
delete the entire ArgoCD Application and recreate it.

**Pitfall: PostgreSQL PVC `lost+found` conflict** — When mounting a PVC
directly at `/var/lib/postgresql/data`, PostgreSQL fails to initialize
because the PVC filesystem contains a `lost+found` directory (created by
mkfs). The PG initdb script refuses to use a non-empty data directory.
**Fix**: Set `PGDATA` env var to a subdirectory:
```yaml
env:
  - name: PGDATA
    value: /var/lib/postgresql/data/pgdata
```
The volumeMount stays at `/var/lib/postgresql/data` — PGDATA just tells
Postgres to use a clean subdirectory inside it.

```yaml
# clusters/main/apps/frigate.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: frigate
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "4"
spec:
  project: default
  source:
    repoURL: http://GITEA_URL/dominik/iac-homelab.git
    targetRevision: main
    path: clusters/main/frigate
  destination:
    server: https://kubernetes.default.svc
    namespace: frigate
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
```

### 4.3b Migrating the Git Server Into K8s (Chicken-and-Egg)

When migrating the Git server that ArgoCD depends on (e.g. Gitea from
LXC to K8s), the circular dependency requires a careful sequence:

**Database choice: Galera over CNPG (user decision 2026-07-13)**

When migrating Gitea to K8s, use the existing **MariaDB Galera cluster**
(MaxScale VIP `10.0.30.70:3306`) instead of deploying a new CNPG
PostgreSQL cluster. Rationale:

| Criterion | Galera (chosen) | CNPG (rejected) |
|-----------|-----------------|-----------------|
| HA | ✅ 3 nodes + MaxScale | ❌ 1 instance planned (no HA) |
| New resources | ❌ None | 512Mi RAM + 10Gi SSD |
| DB engine fit | ✅ MySQL = Gitea's native DB | PG works but less common |
| Backup | ✅ Existing Galera snapshots | Needs Barman/S3 setup |
| K8s-native | ❌ External | ✅ In-cluster |

Trade-off accepted: Galera binds Gitea to existing homelab infra (not
fully K8s-native), but HA + zero new resources outweigh this.

**Safe sequence:**
1. Keep the old Git server running — ArgoCD continues pulling from it
2. Create database on Galera (`CREATE DATABASE gitea; CREATE USER gitea@'%';`)
3. Deploy new Git server to K8s (Helm chart + ExternalSecrets, DB = Galera)
4. Migrate repos + metadata via Gitea Migration API (one API call per repo
   brings repos + issues + labels + milestones + PRs + releases + wiki —
   no manual SQLite→SQL conversion needed)
5. Verify new Git server serves identical content
6. Cutover DNS (e.g. `git.schoen.codes` → K8s LoadBalancer IP)
7. Update all ArgoCD Application `repoURL` fields: old URL → new URL
8. Push the URL change to BOTH Git servers (they must be in sync)
9. Force ArgoCD hard refresh — verify it now pulls from the new Git server
10. After 7 days of stable operation, shut down the old Git server

**Rollback:** Keep old Git server running until step 9 is confirmed. If
anything breaks: revert DNS + revert ArgoCD `repoURL` + start old server.

**Key insight:** The old Git server is NEVER modified during migration
(read-only mirror source). This guarantees zero data loss on rollback.

**Key insight:** Gitea Migration API eliminates the SQLite→SQL schema
conversion entirely. Each `POST /api/v1/repos/migrate` call clones the
repo + imports issues/labels/milestones/PRs/releases/wiki in one shot.
This is vastly simpler than dumping SQLite, converting schemas, and
importing into PG/MySQL manually.

**Pattern reuse:** This sequence applies to any self-hosted GitOps
dependency migration (Gitea, GitLab, Forgejo). The Helm chart and DB
details change, but the chicken-and-egg sequence is universal.

#### Gitea Migration API: ALLOWED_HOST_LIST (NOT ALLOWED_HOSTS)

Gitea 1.26 blocks repo migrations from private IPs by default. The
setting to allowlist source hosts is under `[migrations]` (plural)
and the key is `ALLOWED_HOST_LIST` — NOT `ALLOWED_HOSTS`.

```yaml
# In Helm values (gitea.config):
migrations:
  ALLOWED_HOST_LIST: private,external,loopback,10.0.30.105
  ALLOW_LOCALNETWORKS: true
```

Built-in categories: `loopback` (localhost), `private` (LAN/intranet),
`external` (public hosts), `*` (all hosts). Can also use CIDR
(`10.0.30.0/24`) or wildcard hosts (`*.mydomain.com`).

**Pitfall:** `[migration]` (singular) is silently ignored — the
section MUST be `[migrations]` (plural). Both `ALLOWED_HOSTS` and
`ALLOW_LOCALNETWORKS` under `[migration]` have no effect.

**Pitfall:** Even `ALLOWED_HOST_LIST: *` fails if the section name is
wrong. Verify with `grep migrations /data/gitea/conf/app.ini` inside
the pod — the section header must be `[migrations]`.

**Requires pod restart:** app.ini changes are only read at Gitea
startup. After changing migration settings, delete the pod to force
restart (ArgoCD selfHeal recreates it).

#### Gitea Migration API: PR Import Failure (Retry Without PRs)

Some repos fail during migration with:
- `"error while listing pull requests (page: 1, pagesize: 49). Error: not found"`
- `"initRepository: getRepositoryByID: repository does not exist"`

This is a Gitea Migration API bug affecting repos with empty or
corrupted PR metadata. **Fix:** Delete the half-created repo, then
retry with `pull_requests: false`:

```python
# First: DELETE /api/v1/repos/{owner}/{name} (cleanup half-created)
# Then: POST /api/v1/repos/migrate with pull_requests=False
migrate_data = {
    "clone_addr": f"{CT108_URL}/{owner}/{name}.git",
    "repo_owner": owner, "repo_name": name,
    "mirror": False, "wiki": True, "issues": True,
    "labels": True, "milestones": True,
    "pull_requests": False,  # <-- key fix
    "releases": True, "service": "gitea",
    "auth_token": TOKEN,
}
```

In practice, ~17% of repos (4/23) needed this retry. All succeeded
without PRs.

#### ArgoCD App-of-Apps: Hard Refresh for New Child Apps

When adding new Application manifests to `clusters/main/apps/`, the
root App-of-Apps must be hard-refreshed to discover them:

```bash
# 1. Hard refresh root app to discover new child Application manifests
kubectl patch application root -n argocd --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

# 2. Wait 15-30s for root to sync (creates child Application objects)

# 3. Hard refresh the new child app to force Helm chart pull
kubectl patch application gitea -n argocd --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'
```

Without the root refresh, new child apps don't appear at all. Without
the child refresh, ArgoCD may use stale cached Helm values.

**Pitfall:** `kubectl patch` with `--type merge` for the annotation
works reliably. `kubectl annotate` also works but sometimes the
annotation is consumed before the next reconciliation cycle.

**Reference:** Full migration plan at
`docs/plans/2026-07-13-gitea-k8s-migration.md` in the iac-homelab repo.

### 4.3c Gitea Helm Chart v12 Pitfalls (2026-07-14)

#### DB Password Injection: `additionalConfigFromEnvs` (NOT `env`)

The Gitea Helm Chart v12 generates `app.ini` in an init container
(`configure-gitea`) which runs `gitea migrate`. If the DB password is
not in `app.ini`, migration fails with `Access denied (using password: NO)`.

**Wrong approaches (do NOT use):**
- `gitea.env` — ignored by the chart (not a valid chart value path)
- `gitea.database.existingSecret` — not a valid chart value
- `gitea.config.database.PASSWD: __placeholder__` — literal placeholder
  reaches the DB, not a secret reference
- `deployment.env` — only goes into the main container, NOT the init
  containers that run `gitea migrate`

**Correct approach:** `gitea.additionalConfigFromEnvs` — this is
templated into ALL init containers (init-directories, init-app-ini,
configure-gitea) AND the main container:

```yaml
gitea:
  config:
    database:
      DB_TYPE: mysql
      HOST: 10.0.30.70:3306
      NAME: gitea
      USER: gitea
      # No PASSWD here — injected via additionalConfigFromEnvs below
  additionalConfigFromEnvs:
    - name: GITEA__database__PASSWD
      valueFrom:
        secretKeyRef:
          name: gitea-db
          key: password
```

The `GITEA__database__PASSWD` env var overrides `app.ini` at runtime via
Gitea's environment-based config system (`GITEA__section__KEY`).

**Key insight:** The chart template checks `deployment.env` (goes into
all containers) and `gitea.additionalConfigFromEnvs` (goes into init +
main containers). The `gitea.env` path does NOT exist in the chart
template — it is silently ignored.

#### Valkey/Redis Auto-Deployed Despite `redis-cluster: false`

Chart v12 replaced `redis-cluster` with `valkey-cluster`. Setting
`redis-cluster.enabled: false` alone does NOT disable the cache — you
must ALSO set:

```yaml
valkey-cluster:
  enabled: false
```

Otherwise 3 Valkey pods deploy automatically and consume resources.
They also fail to form a cluster (minority partition) if DNS isn't
ready, producing noisy `cluster_state:fail` warnings.

#### RWO PVC Blocks New Pod During Rollout

With `persistence.accessModes: [ReadWriteOnce]` and `replicaCount: 1`,
ArgoCD's rolling update creates a new pod before the old one terminates.
Both pods compete for the same RWO PVC — the new pod stays in
`Init:0/3` with `FailedAttachVolume: Multi-Attach error`.

**Fix:** Scale to 0 first, wait for PVC release, then scale to 1:
```bash
kubectl scale deploy -n gitea gitea --replicas=0
sleep 15
kubectl scale deploy -n gitea gitea --replicas=1
```

#### Galera Schema Creation Slowness

First-time Gitea DB migration creates ~104 tables. On Galera with
synchronous 3-node replication, each `CREATE TABLE` + index takes
longer than on a single-node DB. Total migration time: 5-8 minutes
(vs <1 min on standalone MySQL). The init container shows
`Init:2/3` during this period — do NOT assume it's hung.

Check progress via Galera:
```bash
# On a Galera node (SSH debian@10.0.30.72, sudo mariadb):
SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='gitea';
SHOW PROCESSLIST;  -- look for "creating table" or "Committing alter table"
```

#### ArgoCD App Split: Config vs Helm

Split the Gitea deployment into two ArgoCD Applications:
- `gitea-config` (sync-wave 1): namespace + ExternalSecrets from Git
  repo (`clusters/main/gitea/`)
- `gitea` (sync-wave 2): Helm chart from `https://dl.gitea.com/charts/`
  (direct Helm source, not from Git)

This ensures secrets are synced before the Helm chart tries to use them.

### 4.3d CoreDNS Internal DNS Overrides (2026-07-14)

When migrating services to K8s that other K8s workloads need to reach
by hostname (e.g. `git.schoen.codes` for ArgoCD's repoURL), use a
CoreDNS `hosts` plugin override via `HelmChartConfig` — NOT external
DNS. This keeps resolution internal to the cluster.

**RKE2 CoreDNS chart uses `servers[].plugins[]` structure** — a flat
`corefile:` key in `valuesContent` does NOT work:

```yaml
# clusters/main/operators/coredns-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    servers:
      - zones:
          - zone: .
        port: 53
        plugins:
          - name: errors
          - name: health
            configBlock: |
              lameduck 10s
          - name: ready
          - name: hosts
            parameters: |
              10.0.30.203 git.schoen.codes
              fallthrough
          - name: kubernetes
            parameters: cluster.local in-addr.arpa ip6.arpa
            configBlock: |
              pods insecure
              fallthrough in-addr.arpa ip6.arpa
              ttl 30
          - name: prometheus
            parameters: 0.0.0.0:9153
          - name: forward
            parameters: . /etc/resolv.conf
          - name: cache
            parameters: 30
          - name: loop
          - name: reload
          - name: loadbalance
```

**Pitfall: `valuesContent: corefile: |` does NOT work.** The RKE2
CoreDNS chart template renders the Corefile from `servers[].plugins[]`,
not from a raw `corefile` key. A `corefile:` value is silently ignored.

**Pitfall: HelmChartConfig applies but ConfigMap doesn't update
immediately.** The HelmChartConfig modifies the HelmChart spec, but
the Helm controller must re-run the helm install job to render the new
ConfigMap. This can take 1-2 minutes. To force immediate application,
either:
- Annotate the HelmChart: `kubectl annotate helmchart rke2-coredns -n
  kube-system helm.cattle.io/retrigger="$(date +%s)" --overwrite`
- OR directly patch the ConfigMap (faster, but non-persistent — the
  HelmChartConfig ensures it survives future redeploys):
  ```bash
  kubectl patch cm rke2-coredns-rke2-coredns -n kube-system --type merge \
    -p '{"data":{"Corefile":".:53 {\n    errors\n    health {\n        lameduck 10s\n    }\n    ready\n    hosts {\n        10.0.30.203 git.schoen.codes\n        fallthrough\n    }\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n        pods insecure\n        fallthrough in-addr.arpa ip6.arpa\n        ttl 30\n    }\n    prometheus 0.0.0.0:9153\n    forward . /etc/resolv.conf\n    cache 30\n    loop\n    reload\n    loadbalance\n}\n"}}'
  ```

**Pitfall: CoreDNS `reload` plugin takes ~30s to pick up ConfigMap
changes.** After patching the ConfigMap, DNS queries for the new host
may still return NXDOMAIN for 30-60s. The `cache` plugin may also
cache the negative result. Wait 35s then retest.

**Verification:**
```bash
# From inside any K8s pod:
kubectl exec -n gitea deploy/gitea -- nslookup git.schoen.codes 10.43.0.10
# Expected: Name: git.schoen.codes, Address: 10.0.30.203
```

**User preference: internal DNS, not external/public.** When resolving
K8s-internal service hostnames (like `git.schoen.codes`), use CoreDNS
overrides — NOT public DNS records. This keeps traffic internal and
avoids hairpin routing through the firewall.

### 4.3e ArgoCD Self-Management Cutover (2026-07-14)

Complete sequence to switch ArgoCD from an external Git server to a
K8s-internal Gitea (chicken-and-egg resolution):

**Prerequisites:**
- New Gitea running in K8s, repos migrated, reachable via Traefik VIP
- CoreDNS override for `git.schoen.codes` → Traefik VIP (Section 4.3d)

**Step 1: Create ArgoCD repository secret**
```bash
# Generate a token on the new Gitea
kubectl exec -n gitea deploy/gitea -- gitea admin user generate-access-token \
  -u dominik -t argocd --raw --config /data/gitea/conf/app.ini

# Create K8s secret (MUST be labeled for ArgoCD to recognize it!)
kubectl create secret generic argocd-repo-k8s-gitea -n argocd \
  --from-literal=url="http://git.schoen.codes/dominik/iac-homelab.git" \
  --from-literal=username="dominik" \
  --from-literal(password)="TOKEN_VALUE" \
  --from-literal=type="git" \
  --dry-run=client -o yaml | kubectl apply -f -

# CRITICAL: Label the secret or ArgoCD ignores it
kubectl label secret argocd-repo-k8s-gitea -n argocd \
  argocd.argoproj.io/secret-type=repository --overwrite
```

**Pitfall: Without the `argocd.argoproj.io/secret-type=repository`
label, ArgoCD does NOT recognize the secret as a repo credential.**
Symptoms: `ComparisonError: failed to generate manifest: authentication
required: Unauthorized` even though the secret exists with correct data.

**Step 2: Patch all ArgoCD Application repoURLs**
```bash
# Patch root + all child apps that reference the old URL
kubectl patch application root -n argocd --type json \
  -p '[{"op":"replace","path":"/spec/source/repoURL","value":"http://git.schoen.codes/dominik/iac-homelab.git"}]'

# Patch all remaining apps referencing old URL
for app in $(kubectl get applications -n argocd -o jsonpath='{range .items[?(@.spec.source.repoURL=="http://OLD_URL")]}{.metadata.name}{"\n"}{end}'); do
  kubectl patch application $app -n argocd --type json \
    -p '[{"op":"replace","path":"/spec/source/repoURL","value":"http://git.schoen.codes/dominik/iac-homelab.git"}]'
done
```

**Step 3: Hard refresh and verify**
```bash
kubectl patch application root -n argocd --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'
# Wait 20-30s
kubectl get applications -n argocd
# Root should show Synced/Healthy
kubectl get application root -n argocd -o jsonpath='{.status.sync.revision}'
# Should show a commit hash from the new Gitea
```

**Result:** ArgoCD now pulls from K8s Gitea via internal DNS → Cilium
VIP → Traefik → Gitea pod. The entire GitOps loop is self-contained in
K8s. Old Git server (CT108) can be kept as fallback, then decommissioned.

### 4.3f Traefik Migration: LXC → K8s (2026-07-14)

Migrating reverse-proxy services from a standalone Traefik LXC (CT99999)
to the RKE2-bundled Traefik DaemonSet. The LXC Traefik runs on VLAN 60
(10.0.60.10), public IP NATs to it. K8s Traefik runs as DaemonSet with
hostPort 80+443 on all 6 K8s nodes.

**K8s Traefik architecture:**
- DaemonSet `rke2-traefik` in `kube-system` (not Deployment)
- `hostPort: 80` and `hostPort: 443` on every node
- IngressClass: `traefik`
- No certResolver configured by default — needs Let's Encrypt setup

**Pattern for external backends (not in K8s):**

Use `ExternalName` Service + `IngressRoute` CRD:

```yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: grafana
  namespace: proxy
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`grafana.familie-schoen.com`)
      kind: Rule
      services:
        - name: grafana-external
          port: 3000
  tls:
    certResolver: letsencrypt
---
apiVersion: v1
kind: Service
metadata:
  name: grafana-external
  namespace: proxy
spec:
  type: ExternalName
  externalName: 10.0.30.141
```

**HA Failover VIP via Cilium L2 (implemented 2026-07-14):**

Instead of pointing the firewall NAT at a single K8s node (SPOF), a
`LoadBalancer` Service requests a VIP from the existing Cilium LB IPAM
pool (`10.0.30.200–250`). Cilium announces the VIP via ARP on one node;
auto-failover to another node on failure (~1-2s).

```
Firewall NAT → 10.0.30.203 (Cilium VIP)
                    ↓ L2 Announcement (ARP)
              [aktiver K8s Node — Traefik DaemonSet]
              Automatisches Failover bei Node-Ausfall (~1-2s)
```

Components deployed:
- `clusters/main/proxy/traefik-lb-service.yaml` — `Service` type
  `LoadBalancer` with `loadBalancerIP: 10.0.30.203`, selects Traefik pods
- `clusters/main/proxy/namespace.yaml` — `proxy` namespace for shared
  IngressRoutes
- `clusters/main/apps/proxy.yaml` — ArgoCD App for the proxy namespace

**Why Cilium L2 (not MetalLB/keepalived):** Cilium L2 LB IPAM is already
running (3 services use it: .200 postgres, .201 hindsight, .202 gitea-ssh).
No additional software needed. Pool `CiliumLoadBalancerIPPool` =
`default-pool`, policy `CiliumL2AnnouncementPolicy` = `default-l2-policy`.

**Verification:** VIP `10.0.30.203` tested — Gitea + ArgoCD reachable
through it. `404` for unmatched hosts (correct — no default route).

**Firewall cutover:** Change NAT target `10.0.60.10` → `10.0.30.203`.
Rollback: revert NAT. CT99999 stays running throughout.

**Remaining migration steps:**
1. Add Let's Encrypt certResolver via HelmChartConfig for `rke2-traefik`
2. Create IngressRoute + ExternalName Service for each external backend
3. Handle catchall-TCP (make specific, not HostSNI(`*`))
4. Cutover firewall NAT: `10.0.60.10` → `10.0.30.203`
5. Verify each service, then decommission CT99999 after 1 week

**Pitfall: Catchall-TCP breaks SNI routing.** A `HostSNI(\`*\`)` TCP
passthrough IngressRoute intercepts ALL TLS traffic on :443, breaking
HTTP IngressRoutes. Either remove the catchall or make it specific
(list actual SNI hosts).

**Pitfall: acme.json migration.** Copy the existing Let's Encrypt
`acme.json` from CT99999 into a K8s Secret mounted into the Traefik
DaemonSet to avoid re-issuing all certificates (rate limit risk).

**Pitfall: Authelia on different subnet.** Backends on non-10.0.30.x
subnets (e.g. Authelia at 192.168.100.11) need routing verified from
K8s nodes before migration.

**Reference:** Full migration plan at
`docs/plans/2026-07-14-traefik-ct99999-k8s-migration.md` in iac-homelab.

**Editing Traefik dynamic config on CT99999 via SSH:**

When modifying Traefik's file-provider dynamic config on the LXC host
(CT99999, `10.0.60.10`), YAML rules contain backtick-delimited Host()
expressions. Writing these via SSH heredoc breaks the backticks (shell
interprets them as command substitution). Two reliable methods:

1. **Base64 transfer** (preferred — preserves all special chars):
```bash
# Encode locally, decode on remote:
cat << 'EOF' | base64 -w0
http:
  routers:
    myapp:
      rule: "Host(`myapp.example.com`)"
      ...
EOF
# Then SSH to remote and decode:
ssh root@10.0.60.10 "echo '<BASE64>' | base64 -d > /etc/traefik/conf.d/myfile.yml"
```

2. **Backup before edit, then `systemctl restart traefik`**:
```bash
cp /etc/traefik/conf.d/explicit-http.yml{,.bak-$(date +%F)}
# Apply changes
systemctl restart traefik
# Check for YAML errors:
tail -5 /var/log/traefik/traefik.log
```

**Pitfall:** Traefik's file watcher (`watch: true` in static config) may
not immediately detect changes made via `cp` overwrite. If the log still
shows old errors with the same timestamp, `systemctl restart traefik`
forces a clean reload.

**Pitfall:** `journalctl -u traefik` on CT99999 may show no recent
entries even after restart — Traefik logs to `/var/log/traefik/`
files, not journald. Check `tail /var/log/traefik/traefik.log` instead.

### 4.3g Hermes Host Git Remote Cutover to K8s Gitea (2026-07-14)

After ArgoCD is self-managing via K8s Gitea (Section 4.3e), the Hermes
host's own git operations (clone, pull, push to `iac-homelab`) must also
switch from the old Gitea (CT108) to the K8s Gitea.

**Challenge:** CoreDNS overrides (Section 4.3d) are K8s-internal only.
The Hermes host (outside K8s) cannot resolve `git.schoen.codes` via
CoreDNS. Public DNS must NOT be used (user preference: internal DNS
only). Solution: `/etc/hosts` entry on the Hermes host.

**Steps:**

1. **Add `/etc/hosts` entry** on the Hermes host:
   ```bash
   echo "10.0.30.203 git.schoen.codes" | sudo tee -a /etc/hosts
   ```
   This maps `git.schoen.codes` to the Traefik Cilium VIP. Traefik
   routes by Host header, so `git clone http://git.schoen.codes/...`
   works — Traefik sees the correct Host and forwards to the Gitea pod.

2. **Verify reachability** before changing the remote:
   ```bash
   curl -s -o /dev/null -w "%{http_code}" \
     -H "Host: git.schoen.codes" http://10.0.30.203/
   # Expected: 200 (Gitea web UI)
   ```

3. **Switch git remote:**
   ```bash
   cd ~/iac-homelab
   git remote set-url origin \
     "http://USER:TOKEN@git.schoen.codes/dominik/iac-homelab.git"
   git fetch origin
   ```

**Pitfall: Gitea API tokens do NOT migrate between instances.** The
token from CT108 (`10.0.30.105:3000`) is invalid on K8s Gitea. Tokens
are stored hashed in the DB and tied to the instance. Must generate a
new token on K8s Gitea.

**Generating a new token (when kubectl is available via mgmt-runner):**
```bash
# SSH to mgmt-runner (VM200), then kubectl exec into Gitea pod:
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl exec -n gitea deploy/gitea -- \
   gitea admin user generate-access-token \
   -u dominik -t hermes-host --scopes all --raw \
   --config /data/gitea/conf/app.ini"
```

**Pitfall: Hermes host has no `kubectl` installed.** All kubectl
operations go through SSH to mgmt-runner (VM200, `10.0.30.124`) — see
Section 1.2. The Hermes host interacts with K8s Gitea only via HTTP
through the Traefik VIP.

**Pitfall: Port 3000 is NOT exposed on the Traefik VIP.** The K8s
Gitea is reachable from outside K8s only via Traefik on ports 80/443
(with `Host: git.schoen.codes`), NOT via `10.0.30.203:3000`. The Gitea
SSH LB (`10.0.30.202:2222`) is a separate Cilium LB service for git
over SSH.

**Pitfall: JWT (internal security token) ≠ Gitea API token.** The
1Password ExternalSecrets for Gitea contain `internal_token`,
`jwt_secret`, `secret_key`, `lfs_jwt_secret` — these are JWTs (three
base64 segments separated by dots, e.g.
`eyJhbGci....eyJuYmYi....Lyr6fqc...`), NOT API access tokens (which
are hex strings like `07efa534ea1a...`). A JWT returns `401
Unauthorized` when used as `Authorization: token <JWT>` or
`Authorization: Bearer <JWT>`. If a user provides a dotted base64
token, it is a security token — generate a real API token via
`gitea admin user generate-access-token` instead.

**Pitfall: ArgoCD repo secret also needs the new token.** After
generating a new Gitea API token for the Hermes host, the ArgoCD
repository secret (`argocd-repo-k8s-gitea`) must also be patched with
the same token, otherwise ArgoCD's repo connection breaks:
```bash
kubectl patch secret argocd-repo-k8s-gitea -n argocd --type merge \
  -p '{"data":{"password":"$(echo -n TOKEN | base64)"}}'
```

**1Password Storage of Gitea Credentials:**

After generating the token and identifying the admin password, store
both in 1Password vault "Hermes":

```bash
# API token (two-step — see 1password-cli skill for the credential
# field pitfall with API Credential category):
op item create --category="API Credential" \
  --title="K8s Gitea Token (hermes-gitops)" --vault="Hermes" \
  username="dominik" url="http://git.schoen.codes"
op item edit "<ITEM_ID>" --vault="Hermes" \
  credential="TOKEN_VALUE" hostname="git.schoen.codes"

# Admin login (Login category — one step works):
op item create --category="Login" \
  --title="K8s Gitea Admin (dominik)" --vault="Hermes" \
  username="dominik" password="ADMIN_PASSWORD" \
  --url="http://git.schoen.codes"
```

**Reference:** `references/hermes-git-remote-cutover-2026-07.md`

### 4.3i InfluxDB K8s Migration: Fresh Deploy + Replication (2026-07-14)

Migrating InfluxDB v2 from LXC CTs (CT109 hot + CT134 archive) to a single
K8s StatefulSet. Design was developed through the Compound Engineering
brainstorming process (see Workflow Correction below).

**Two approaches were evaluated; user chose Option B (Fresh Deploy):**

| Aspect | Option A: Backup/Restore | Option B: Fresh Deploy + Replication (CHOSEN) |
|--------|--------------------------|-----------------------------------------------|
| Data transfer | `influx backup` → `influx restore --full` | `influx replication` streaming + Flux backfill |
| Token handling | Preserved from backup (HA only needs URL change) | New token; HA needs URL + token change |
| GitOps cleanliness | Restore is a one-time imperative op | Fresh deploy via GitOps, replication is declarative |
| Parallel operation | No (cutover gap) | Yes (both instances live until cutover) |
| Complexity | Lower | Higher (replication setup + backfill) |

**Chosen migration phases (Option B):**

1. Deploy empty InfluxDB on K8s (StatefulSet, `DOCKER_INFLUXDB_INIT_MODE=setup`,
   Cilium L2 LB VIP `.204` from existing pool)
2. Configure replication: `influx remote create` + `influx replication create`
   on CT109 → streams every new write to K8s
3. Backfill existing 90-day history via Flux `to()` with remote host
4. Recreate downsampling task as LOCAL on K8s (no remote `to()` call)
5. Cutover: change HA InfluxDB integration URL + token to K8s VIP
6. Verify: data counts match, task runs succeed, HA writes flowing
7. Decommission CT109 + CT134 after 1-week stable operation

**Archive structure decision: single instance with local task (chosen over
two-instance approach).** Both buckets (`ha_hot_90d` + `ha_archive_5y_5m`)
on one K8s instance. Downsampling task runs locally (reads hot bucket, writes
archive bucket — same instance). Rationale: 4 GB total data, ~5 MB/day growth
— splitting into two instances adds backup complexity without meaningful
risk reduction. PVC sized at 30 Gi (hot 4 GB + archive growth + headroom).

**Backup strategy for K8s instance:** Single daily `influx backup` CronJob
→ Norris S3 (same pattern as CNPG/Postgres). One backup, one restore —
covers both buckets. Ceph RBD replica-3 + S3 offsite = double protection.

**Manifest structure (GitOps):**
```
clusters/main/
  apps/influxdb.yaml          → ArgoCD Application (App-of-Apps child)
  influxdb/
    namespace.yaml             → ExternalSecret for admin creds (1Password)
    statefulset.yaml           → InfluxDB v2.7, init mode, PVC on ceph-hdd-replica
    service.yaml               → LoadBalancer with Cilium L2 LB IPAM annotation
    init-configmap.yaml        → Post-deploy init script (create archive bucket + task)
    init-job.yaml              → ArgoCD PostSync hook Job to run init script
```

**Pitfall: `DOCKER_INFLUXDB_INIT_MODE=setup` runs only on first boot.** If the
PVC already has data, the init mode is skipped. The PostSync init Job handles
post-deploy configuration (archive bucket creation, task setup) idempotently.

**Pitfall: 1Password CLI may not be authenticated on the Hermes host.**
`OP_SERVICE_ACCOUNT_TOKEN` can be set but `op account list` returns `[]`.
ExternalSecrets must be created via the K8s service account token (§5.3) or
the 1Password Web UI in the "Kubernetes ESO" vault.

**Pitfall: Downsampling Task must filter numeric fields only.** Home Assistant
writes string fields (`icon_str`, `state_class_str`, `attribution_str`) alongside
numeric `value` fields. `aggregateWindow(fn: mean)` crashes on strings with
`unsupported input type for mean aggregate: string`. Fix: add
`|> filter(fn: (r) => r._field == "value")` before aggregation.

**Pitfall: CT134 archive was effectively dead (368 KB, no new data for 12 days).**
The downsampling task on CT109 was failing silently — it appeared to run
successfully but the 91-90 day window was empty (hot bucket only 76 days old).
The task crash on string fields had been masked until the window had data.

**Workflow Correction: Compound Engineering brainstorming-first (2026-07-14).**
User explicitly corrected the approach of jumping to `execute_code` and writing
manifests before designing the migration concept. For infrastructure migrations,
ALWAYS follow the Compound Engineering process:
1. Load the `brainstorming` skill FIRST
2. Explore project context + grounding scan (docs/solutions/, Hindsight)
3. Ask clarifying questions ONE AT A TIME (multiple choice preferred)
4. Propose 2-3 approaches with trade-offs
5. Present design incrementally, get approval
6. Write design doc
7. Transition to `writing-plans` skill for implementation plan
Do NOT write manifests, execute code, or take implementation action until
the design is approved. Even if the manifests seem straightforward, the
structured exploration reveals design decisions (backup strategy, token
handling, archive structure) that would otherwise be discovered too late.

**Pitfall: ArgoCD ServerSideApply does NOT update ConfigMaps with embedded scripts.**
When a ConfigMap contains a shell script (e.g. `init.sh` in `data`), and the script is
modified in Git, ArgoCD with `ServerSideApply=true` may show `OutOfSync/Healthy` but
NOT apply the updated ConfigMap content. The live ConfigMap retains the old script.
Workarounds:
1. `kubectl delete cm <name> -n <ns>` + hard refresh — ArgoCD recreates from Git
2. Execute the updated script manually via `kubectl exec` (faster for one-time fixes)
3. Consider splitting scripts into separate files mounted via a projected volume

**Pitfall: `influx task create --file` requires `option task = {}` header in Flux.**
Without `option task = { name: "task-name", every: 1h }` at the top of the Flux script,
`influx task create` fails with `400 Bad Request: invalid options: no task options defined`.
The `--name` and `--every` CLI flags set metadata but do NOT inject the `option task`
block into the Flux script itself — the script must declare it explicitly.

**Pitfall: InfluxDB replication CLI flags use `-id` suffix, not bare names.**
`influx replication create` requires `--local-bucket-id` (NOT `--local-bucket`) and
`--remote-bucket-id` (NOT `--remote-bucket`). Passing the bare flag produces
`flag provided but not defined: -local-bucket`.

**Pitfall: Flux `to()` backfill produces surprisingly small disk footprint.**
A 90-day backfill of 4.25 million rows via `from() |> range() |> to(host: remote)`
produced only 51 MB on the destination InfluxDB (vs 4.1 GB on the source). This is
normal — InfluxDB v2's columnar compression is extremely effective on fresh shards
without historical shard overhead. Verify success by counting rows, not checking disk size:
```flux
from(bucket: "ha_hot_90d") |> range(start: -90d) |> count() |> sum() |> group() |> sum()
```

**Pitfall: PostSync Hook Job does NOT re-run when only the ConfigMap changes.**
ArgoCD's `PostSync` hook with `BeforeHookCreation` deletes the old Job and creates a
new one on each sync. BUT if the Application is `OutOfSync` due to the ConfigMap diff
but ArgoCD doesn't actually apply the ConfigMap (see ServerSideApply pitfall above),
the new Job still uses the OLD ConfigMap content. The Job "succeeds" with stale data.
Fix: manually delete the ConfigMap, force refresh, then manually trigger the Job or
run the init script directly via `kubectl exec`.

**Pitfall: Backup CronJob requires multi-container pattern (init + upload).**
The `influxdb:2.7` image has NO `python3`, `aws`, `rclone`, or working `apt-get`
(apt-get install hangs/crashes). The `amazon/aws-cli:2` image tag does NOT exist —
must use a specific version like `amazon/aws-cli:2.35.22`. The `amazon/aws-cli` image
has NO `tar` command. Solution: use init-container (influxdb image) for
`influx backup` + `tar czf`, then a separate container (aws-cli image) for
`aws s3 cp`:

```yaml
initContainers:
  - name: backup
    image: influxdb:2.7
    command: [/bin/sh, -c, |
      influx backup /shared/backup --host http://influxdb:8086 \
        --token "${DOCKER_INFLUXDB_INIT_ADMIN_TOKEN}" --org homelab
      cd /shared && tar czf backup.tar.gz backup/]
    # ...
containers:
  - name: upload
    image: amazon/aws-cli:2.35.22
    command: [/bin/sh, -c, |
      aws s3 cp /shared/backup.tar.gz s3://BUCKET/$(date +%Y-%m-%d)/backup.tar.gz \
        --endpoint-url https://rgw.nbg.nsc.noris.cloud --region nsc-nbg]
    # ...
```

**Pitfall: AWS CLI requires UPPERCASE env vars.** Using `envFrom` with a secret
containing lowercase keys (`aws_access_key_id`, `aws_secret_access_key`) does NOT
work — AWS CLI looks for `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
Must use explicit `env` with `valueFrom` mapping:

```yaml
env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: influxdb-s3-backup
        key: aws_access_key_id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: influxdb-s3-backup
        key: aws_secret_access_key
```

**Pitfall: S3 bucket must be created before first backup run.** The CronJob
fails if the bucket doesn't exist. Create it via a one-off pod.
**CRITICAL: `aws s3 mb` with `--region nsc-nbg` fails with
**Pitfall: S3 bucket must be created before first backup run.** The CronJob
fails with `argument of type 'NoneType' is not a container or iterable` if the
bucket doesn't exist. Create it via a one-off pod.

**CRITICAL: `aws s3 mb` requires `--region us-east-1`, NOT `nsc-nbg`.** Using
`--region nsc-nbg` fails with `InvalidLocationConstraint: The nsc-nbg location
**Pitfall: S3 bucket must be created before first backup run.** The CronJob
fails with `argument of type 'NoneType' is not a container or iterable` if the
bucket doesn't exist. Create it via a one-off pod:
```bash
kubectl run aws-cli-create-bucket --image=amazon/aws-cli:2.35.22 --restart=Never \
  --env=AWS_ACCESS_KEY_ID=$AKI --env=AWS_SECRET_ACCESS_KEY=$SAK \
  --command -- sh -c "aws s3 mb s3://homelab-influxdb-backup \
    --endpoint-url https://rgw.nbg.nsc.noris.cloud --region us-east-1; sleep 5"
```

**Pitfall: `aws s3 mb` with `--region nsc-nbg` fails with `InvalidLocationConstraint`.**
Norris S3 (rgw.nbg.nsc.noris.cloud) rejects the `nsc-nbg` location constraint for
bucket creation, even though `aws s3 cp` works fine with `--region nsc-nbg` for
uploads/downloads. For `s3 mb` only, always use `--region us-east-1` (the AWS
default region suppresses the LocationConstraint header). This affects all
Norris S3 bucket creations, not just InfluxDB.

**Pitfall: Home Assistant OS SSH is on port 22222, not 22.** HA OS (qemux86-64
image) exposes SSH on port 22222 for root. The HA configuration file is at
`/mnt/data/supervisor/homeassistant/configuration.yaml` (NOT `/config/`).
Restart HA via `docker restart homeassistant` from the SSH session. The HA
Supervisor access token is in `/mnt/data/supervisor/homeassistant.json` (field
`access_token`) but the Supervisor REST API returns 404 for
`/api/supervisor/core/restart` — use `docker restart` instead.

**Reference:** `references/influxdb-k8s-migration-2026-07.md`

### 4.3j Backup CronJob for Services with PVC + External DB (2026-07-14)

When backing up a K8s service that has BOTH PVC-mounted data AND an
external database (e.g. Gitea with Galera), use a multi-initContainer
CronJob with dedicated RBAC for `kubectl exec`/`kubectl cp`.

**Architecture (Gitea example):**

```
initContainer 1 (gitea-files):  kubectl exec tar /data → kubectl cp → emptyDir
initContainer 2 (gitea-db):    mysqldump against Galera → emptyDir
container (upload):             aws s3 cp both files → Norris S3
```

**RBAC required:** The backup pod needs a ServiceAccount with a Role
granting `pods/exec` and `pods/get` in the target namespace — without
this, `kubectl exec` and `kubectl cp` fail with forbidden:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: SERVICE-backup
  namespace: NAMESPACE
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "get"]
```

Bind via RoleBinding to a ServiceAccount, then set
`serviceAccountName` on the CronJob pod spec.

**File backup via kubectl exec + kubectl cp:**

Since the PVC is RWO and mounted by the running pod, you can't mount
it directly in the backup pod. Instead:
1. `kubectl exec` into the running pod to `tar czf /tmp/data.tar.gz`
2. `kubectl cp` to extract the archive to the shared emptyDir
3. `kubectl exec` to clean up the temp file

```yaml
initContainers:
  - name: gitea-files
    image: bitnami/kubectl:1.31
    command: [/bin/sh, -c, |
      kubectl exec -n gitea gitea-0 -- tar czf /tmp/gitea-data.tar.gz -C / data
      kubectl cp gitea/gitea-0:/tmp/gitea-data.tar.gz /shared/gitea-data.tar.gz
      kubectl exec -n gitea gitea-0 -- rm -f /tmp/gitea-data.tar.gz]
    volumeMounts:
      - name: shared
        mountPath: /shared
```

**DB backup via mysqldump against Galera:**

Use the `mysql:8.0` image with DB credentials from an existing K8s
secret (e.g. `gitea-db`):

```yaml
initContainers:
  - name: gitea-db
    image: mysql:8.0
    command: [/bin/sh, -c, |
      mysqldump --host=10.0.30.70 --port=3306 --user=gitea \
        --password="${GITEA_DB_PASS}" --single-transaction \
        --routines --triggers gitea | gzip > /shared/gitea-db.sql.gz]
    env:
      - name: GITEA_DB_PASS
        valueFrom:
          secretKeyRef:
            name: gitea-db
            key: password
    volumeMounts:
      - name: shared
        mountPath: /shared
```

**S3 credentials reuse across backup CronJobs:** Multiple backup
CronJobs can share the same Norris S3 account credentials (from the
`postgres-s3-backup` 1Password item). Create a new ExternalSecret per
namespace pointing to the same 1Password item keys, and a new S3
bucket per service. Bucket naming: `homelab-<service>-backup` (e.g.
`homelab-gitea-backup`, `homelab-influxdb-backup`). Each bucket must
be created via `aws s3 mb` before the first backup run (see §4.3i
pitfall).

**Schedule staggering:** Offset backup schedules to avoid concurrent
S3 uploads. InfluxDB at `0 3 * * *` (03:00 UTC), Gitea at
`30 3 * * *` (03:30 UTC).

**Pitfall: `bitnami/kubectl:1.31` tag does NOT exist.** Use full semver
(`1.31.4`). Even then, Docker Hub rate-limiting causes 3+ min pulls.
`alpine/k8s:1.31.4` same issue. When kubectl images won't pull, run the
backup from mgmt-runner (VM200) which has kubectl + mysqldump locally —
see `references/gitea-backup-cronjob-2026-07.md`.

**Pitfall: `kubectl cp` fails for large files (>~50 MB).** WebSocket
resets: `connection reset by peer`, `close 1006 abnormal closure`. Small
files work. For large PVC tars, run from mgmt-runner directly.

**Pitfall: VM200 repo remote may be stale after Gitea migration.** Still
points to CT108. `git pull` succeeds but new commits from K8s Gitea never
arrive. Fix: `git remote set-url origin <new-url>` on VM200.

**Pitfall: ArgoCD "successfully synced" may NOT create new resources.**
With `ServerSideApply=true`, new files can go undetected. Hard-refresh
twice or `kubectl apply -f` manually. Verify with `kubectl get`.

**Pitfall: hostPath volumes are node-local.** Pod on worker-01 can't read
mgmt-runner's `/tmp`. Use `emptyDir` + `kubectl cp` (small files) or
`nodeName:` pinning (fragile).
contain kubectl.

**Pitfall: `kubectl cp` is unreliable for large files (>~50 MB).** The
websocket connection used by `kubectl cp` resets before large transfers
complete (`write: connection reset by peer`, `websocket: close 1006`).
Small files (<1 MB) work fine. For large PVC backups (e.g. 143 MB Gitea
data), avoid `kubectl cp` — generate the tar directly in an initContainer
that shares an emptyDir with the upload container, or run the backup from
the mgmt-runner which has kubectl locally.

**Pitfall: `bitnami/kubectl:1.31` image tag does NOT exist.** The bitnami/kubectl
image uses full semver tags (e.g. `1.31.4`), not minor-version tags like `1.31`.
Even with correct tags, Docker Hub rate-limiting causes extremely slow pulls
(3+ min). When kubectl images won't pull, run backup from mgmt-runner (VM200)
which has kubectl locally. See `references/gitea-backup-cronjob-2026-07.md`. Solutions:
1. Pre-pull the image on all worker nodes: `crictl pull alpine/k8s:1.31.4`
2. Use an image from a non-Docker-Hub registry
3. **Run the backup from the mgmt-runner** (VM200, 10.0.30.124) which already
   has kubectl + mysqldump installed — see
   `references/gitea-backup-cronjob-2026-07.md` for the manual fallback script.
   This is the fastest workaround when image pulls are blocked.

**Pitfall: `kubectl cp` is unreliable for files >50 MB.** The K8s API websocket
connection resets before large file transfers complete. Small files (<1 MB)
work fine. For large PVC backups (e.g. 143 MB Gitea data), avoid `kubectl cp`
between pods — generate the data in the same pod that uploads it, or run from
the mgmt-runner where `kubectl cp` writes to local disk (more reliable than
pod-to-pod transfer).

**Pitfall: hostPath volumes are node-local.** A pod using `hostPath: /tmp`
reads from the NODE it's scheduled on, not from the machine that created the
files. If files are on the mgmt-runner but the pod runs on a worker, the files
are invisible. Use `emptyDir` + `kubectl cp`, or run the backup from the
mgmt-runner directly.

**Manifest files (Gitea backup):**
- `clusters/main/gitea/backup-rbac.yaml` — SA + Role + RoleBinding
- `clusters/main/gitea/backup-s3-secret.yaml` — ExternalSecret (Norris S3)
- `clusters/main/gitea/backup-cronjob.yaml` — CronJob (3 containers)

**Reference:** `references/gitea-backup-cronjob-2026-07.md`

When the user asks "what's the next workload to migrate to K8s?", use
this structured assessment methodology. The user migrates LXC/VM
workloads to K8s incrementally — this pattern recurs for each wave.

**Step 1: Inventory all running workloads across all PVE nodes**

```bash
for node_ip in 10.0.20.10 10.0.20.20 10.0.20.30 10.0.20.40 10.0.20.50 10.0.20.60; do
  echo "=== $node_ip ==="
  ssh -i ~/.ssh/id_ed25519_proxmox -o ConnectTimeout=3 root@$node_ip \
    "qm list 2>/dev/null; pct list 2>/dev/null" 2>/dev/null
done
```

Focus on **running** CTs/VMs only — stopped ones are already decommissioned
candidates. Note the PVE node, CT/VM ID, hostname, and resource config
(cores, memory, disk).

**Step 2: Check K8s resource availability**

```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl top nodes 2>&1"
```

Workers have the most headroom (typically 80%+ free CPU/memory).
CP nodes are ~50-60% utilized. Ensure the target workload fits.

**Step 3: Check what's already in K8s**

```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl get deploy,sts,svc -A \
   --no-headers 2>&1 | grep -v 'kube-system\|argocd\|cattle\|cilium'"
```

Avoid duplicating services already running in K8s.

**Step 4: Evaluate each candidate against criteria**

| Criterion | Question |
|-----------|----------|
| **Helm chart** | Does an official/maintained Helm chart exist? |
| **Statefulness** | Stateless or simple state (single PVC) = easier |
| **Dependencies** | Does K8s Traefik/other services depend on it? |
| **Resource footprint** | Fits within worker node capacity? |
| **Risk** | What breaks if migration fails? (Auth-only = low, data = high) |
| **Value** | How much operational complexity does migration eliminate? |

**Step 5: Rank by dependency urgency + simplicity**

Prioritize workloads that **unblock other migrations**. Example: Authelia
must migrate before Traefik can fully move to K8s (Authelia is the auth
backend for every routed service). Simple, low-risk workloads that
eliminate LXC sprawl are good filler between complex migrations.

**Migration wave pattern:**

| Wave | Workload | Rationale |
|------|----------|-----------|
| 1 | Dependency services (Auth, DNS) | Unblocks proxy layer |
| 2 | Proxy/Ingress layer | All external traffic through K8s |
| 3 | Observability (InfluxDB, Grafana) | Simple Helm charts |
| 4 | High-value complex stacks (Immich) | Multi-service but highest value |
| 5 | Low-priority containers | Cleanup |

**Present results as a table** with: CT/VM ID, name, node, status,
K8s-fit rating (⭐), and a one-line recommendation. End with a ranked
recommendation and a brief rationale — the user decides what to tackle.

**Pitfall: "Running" ≠ "In use".** Before recommending a service for
migration, verify it's actually being used. A service can be running
but completely dormant:

1. **Check reverse proxy config** — grep for the service name as a
   middleware (e.g. `forwardAuth` referencing Authelia). If no router
   references it as a middleware, it's not protecting anything.
2. **Check data store freshness** — `ls -la` on the DB file (SQLite) or
   query row counts. A 304 KB SQLite DB unchanged since August 2025 =
   abandoned.
3. **Check domain currency** — if the service is configured for an old
   domain (e.g. `familie-schoen.com` when the active domain is
   `schoen.codes`), it's likely orphaned.
4. **Check registered users** — if only an admin + one user exist and
   neither has logged in recently, the service is decorative.

Example: Authelia (CT112) was running, had 2 users, TOTP configured,
Traefik route existed — but NO router used it as a `forwardAuth`
middleware, the SQLite DB was stale, and the domain was old. Decision:
stop and decommission instead of migrating. When a service is unused,
stopping the CT (not migrating) is the correct action.

### 4.4 Removing an Application / Operator via GitOps

When removing an operator-managed application (e.g. MariaDB Operator +
its MariaDB/MaxScale CRs) from ArgoCD:

**Pitfall**: Simply deleting the YAML files from the repo and pushing
does **NOT** cleanly prune operator-managed resources. ArgoCD marks the
app `OutOfSync` but the operator's Custom Resources (CRs) hold
finalizers that prevent deletion. The operator pods must be running to
process the finalizer — but if you delete the operator first, the CRs
are orphaned and the namespace hangs in `Terminating` forever.

**Correct removal sequence**:

1. **Remove YAML files from repo** (commit + push to Gitea)
2. **Manually delete the CRs first** (while operator is still running):
   ```bash
   kubectl delete mariadb mariadb-galera -n mariadb --timeout=120s
   kubectl delete maxscale mariadb-maxscale -n mariadb  # may 404 if already gone
   ```
3. **Wait for CRs to be fully deleted** (operator processes finalizers)
4. **Delete the namespace**:
   ```bash
   kubectl delete namespace mariadb --timeout=30s
   kubectl delete namespace mariadb-operator --timeout=30s
   ```
5. **Delete the CRDs** (ArgoCD may not prune these automatically):
   ```bash
   kubectl delete crd backups.k8s.mariadb.com connections.k8s.mariadb.com \
     databases.k8s.mariadb.com grants.k8s.mariadb.com \
     mariadbs.k8s.mariadb.com maxscales.k8s.mariadb.com \
     restores.k8s.mariadb.com sqljobs.k8s.mariadb.com \
     users.k8s.mariadb.com --timeout=30s
   ```
6. **Force ArgoCD refresh** if apps linger:
   ```bash
   kubectl annotate application root -n argocd \
     argocd.argoproj.io/refresh-options=hard-refresh --overwrite
   ```

**Key insight**: ArgoCD `prune: true` removes standard K8s resources
(deployments, services, configmaps) automatically, but **operator CRDs
and their CR instances often need manual cleanup** because finalizers
require the operator to be alive to process them.

### 4.4b Removing a Non-Operator Application via GitOps

For standard apps (Deployments, Services, StatefulSets without operator
CRs), removal is simpler — no finalizer issues:

1. **Remove YAML files from repo** (ArgoCD Application manifest +
   entire `clusters/main/<service>/` directory + any GitHub workflows):
   ```bash
   cd /root/iac-homelab
   rm clusters/main/apps/<service>.yaml
   rm -rf clusters/main/<service>/
   rm .github/workflows/<service>-*.yml  # if applicable
   git add -A && git commit -m "chore: remove <service> (unused)"
   git push origin main
   ```

2. **Delete the ArgoCD Application** (cascade=foreground cleans up
   managed resources):
   ```bash
   kubectl delete application <name> -n argocd --cascade=foreground
   ```

3. **Delete the namespace** (if not auto-pruned — ArgoCD's
   `CreateNamespace=true` creates it but doesn't always prune it):
   ```bash
   kubectl delete ns <namespace>
   # May take 10-30s if pods need to terminate first
   ```

4. **Verify**: `kubectl get ns <namespace>` → NotFound, `kubectl get pods -A
   | grep <namespace>` → empty.

**Pitfall**: ArgoCD Application deletion with `--cascade=foreground` may
leave the namespace in `Terminating` if pods have stuck finalizers. The
standard `kubernetes` finalizer resolves once pods are evicted. If it
hangs, check for leftover ExternalSecrets or PVCs with custom
finalizers.

### 4.5 ArgoCD Pod States During Cold Boot

After cluster cold-boot, ArgoCD pods show mixed states:
- Some `Running` (new replicas scheduled by deployment controller)
- Some `Terminating` (old replicas from before shutdown)
- Some `Unknown` (node was down when status was last reported)

These resolve naturally as the deployment controller reconciles. No
manual intervention needed unless pods are stuck for 10+ minutes.

### 4.6 Deploying Python Apps with pip in Init Containers

When deploying a Python app that isn't available as a Docker image
(e.g. `hindsight-api` from PyPI), the common pattern is an init
container that runs `pip install`, storing packages in an `emptyDir`
volume shared with the main container.

**Pitfall: `pip install --target=` does NOT install console_scripts**
— Using `pip install --target=/opt/packages` copies the Python modules
but does NOT create bin/ entry points (console_scripts). The main
container will fail with `command not found` even though the package
files are present. **Fix**: Use `python -m venv` instead:

```yaml
initContainers:
  - name: install-deps
    image: python:3.11-slim
    command: ["/bin/sh", "-c"]
    args:
      - |
        python -m venv /opt/venv &&
        /opt/venv/bin/pip install --no-cache-dir <package> &&
        echo "Done"
    volumeMounts:
      - name: venv
        mountPath: /opt/venv
containers:
  - name: app
    image: python:3.11-slim
    command: ["/bin/sh", "-c"]
    args: ["/opt/venv/bin/<binary> --host 0.0.0.0 --port 8888"]
    volumeMounts:
      - name: venv
        mountPath: /opt/venv
        readOnly: true
volumes:
  - name: venv
    emptyDir:
      sizeLimit: 3Gi
```

**Pitfall: Heavy dependencies pull CUDA torch (526MB)** — Packages
that depend on `torch` (e.g. `sentence-transformers`,
`hindsight-api`) will pull the full CUDA-enabled torch wheel by
default, which is 526MB+ and may OOMKill the init container. **Fix**:
Install CPU-only torch first with a dedicated index, then install the
main package with `--extra-index-url` so pip finds CPU torch:

```yaml
args:
  - |
    /opt/venv/bin/pip install --no-cache-dir \
      torch --index-url https://download.pytorch.org/whl/cpu &&
    /opt/venv/bin/pip install --no-cache-dir \
      <package-with-torch-dep> \
      --extra-index-url https://download.pytorch.org/whl/cpu &&
    echo "Done"
```

The `--index-url` for the first install ensures ONLY CPU wheels are
considered. The `--extra-index-url` for the second install allows pip
to find the already-satisfied CPU torch instead of upgrading to the
default CUDA version.

**Pitfall: Init container pip install is slow (~10min)** — Installing
100+ packages (including torch, transformers, sentence-transformers)
takes 5-10 minutes. The pod stays in `Init:0/1` during this time. Set
generous timeouts and don't assume CrashLoopBackOff means failure —
check the init container logs first.

---

## Section 5: External Secrets & 1Password

### 5.1 ClusterSecretStore

The cluster uses a single `ClusterSecretStore` backed by 1Password:

```bash
kubectl get clustersecretstore onepassword-store
# Should show READY=True
```

### 5.2 ExternalSecret Key Format (CRITICAL)

The `remoteRef.key` format is **`item-title/field-name`** — a simple
slash-separated path, NOT `op://` URIs and NOT a separate `property`
field.

```yaml
# CORRECT
spec:
  data:
    - secretKey: api-key
      remoteRef:
        key: noris-api-key/credential    # item-title/field-name

# WRONG — will fail with "secret reference has invalid format"
spec:
  data:
    - secretKey: api-key
      remoteRef:
        key: noris-api-key
        property: credential
```

Field names depend on the 1Password item category:
- **API Credential**: `credential`
- **Password**: `password`
- **Login**: `username`, `password`
- **Database**: `username`, `password`, `database`, `host`, `port`

### 5.3 1Password Vault Mismatch (CRITICAL PITFALL)

The K8s `ClusterSecretStore` uses a **different 1Password service
account** than the local `op` CLI on the Hermes host. Items created
with the local `op` CLI land in vault **Hermes**
(`5ythsz37hf3xminhq33drg55tm`), but the K8s ESO reads from vault
**Kubernetes ESO** (`334ykdtj5kar3jlpcrztjvx2fu`).

**Symptom**: ExternalSecret shows `SecretSyncedError: no vault matched
the secret reference query` even though the item exists in 1Password.

**Diagnosis**:
```bash
# Check which vault the K8s ClusterSecretStore uses
kubectl get clustersecretstore onepassword-store -o jsonpath='{.spec.provider.onepasswordSDK.vault}'

# Check which vaults your local op CLI can see
op vault list

# Check which vaults the K8s service account can see
K8S_TOKEN=$(kubectl get secret onepassword-token -n external-secrets -o jsonpath='{.data.token}' | base64 -d)
OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op vault list
```

**Fix**: Create 1Password items in the vault that the K8s service
account can see (vault `334ykdtj5kar3jlpcrztjvx2fu` = "Kubernetes ESO").

The K8s service account token CAN create items (not read-only as
previously assumed):
```bash
K8S_TOKEN=$(kubectl get secret onepassword-token -n external-secrets \
  -o jsonpath='{.data.token}' | base64 -d)
OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op item create \
  --category="API Credential" \
  --title="item-name" \
  --vault="334ykdtj5kar3jlpcrztjvx2fu" \
  credential=SECRET_VALUE

OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op item create \
  --category="Password" \
  --title="item-name" \
  --vault="334ykdtj5kar3jlpcrztjvx2fu" \
  password=SECRET_VALUE
```

Alternatively, create items via the 1Password Web UI in the correct
vault, or move items from the Hermes vault to Kubernetes ESO.

### 5.4 Common External Secret Failures

| Error | Cause | Fix |
|-------|-------|-----|
| `rate limit exceeded` | 1Password API throttling | Wait + retry (auto-resolves) |
| `no vault matched the secret reference query` | Item in wrong vault (see §5.3) | Create item in "Kubernetes ESO" vault |
| `secret reference has invalid format` | Wrong key format (see §5.2) | Use `item-title/field-name` format |
| `more than one field matched the secret reference` | Duplicate items in 1Password vault | Deduplicate items |
| `dial tcp: lookup my.1password.com: connect: operation not permitted` | DNS resolution from pod network | Check CoreDNS, Cilium policies |

### 5.3 Checking External Secret Status

```bash
# All ExternalSecrets across cluster
kubectl get externalsecrets -A

# Detailed status for a failing secret
kubectl describe externalsecret -n NAMESPACE SECRET_NAME
```

### 5.5 StorageClasses Available

The cluster has two StorageClasses (both Ceph RBD via CSI):

| StorageClass | Provisioner | Tier | Use Case |
|---|---|---|---|
| `ceph-flash` (default) | rbd.csi.ceph.com | SSD/NVMe | Databases, hot workloads |
| `ceph-hdd-replica` | rbd.csi.ceph.com | HDD | Bulk storage, archives |

**Pitfall**: There is NO `ceph-rbd` StorageClass. Using it in a PVC
causes the pod to stay `Pending` indefinitely with no PV created.
Always use `ceph-flash` or `ceph-hdd-replica`.

---

## Section 6: Nested Proxmox SSH Patterns

### 6.1 Discovering Which PVE Node Hosts a VM

PVE HA migrates VMs between nodes. Before operating on a VM, find
which node it's on:

```bash
for node in proxmox1 proxmox2 proxmox3 proxmox4 proxmox5 proxmox6 proxmox7; do
  ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
    "ssh -o ConnectTimeout=3 root@$node 'qm list 2>/dev/null | grep VMID'" 2>/dev/null
done
```

### 6.2 Executing Commands on VMs via qm guest exec

```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "qm guest exec VMID -- sh -c 'command here'"
```

Output is JSON: `{"exitcode": N, "out-data": "...", "err-data": "..."}`

**Pitfall**: Complex shell commands with quotes break through nested
SSH → qm guest exec. Use base64 encoding for file writes (see
docker-host-administration Section 8.4).

**Pitfall**: `qm guest exec` requires the QEMU guest agent to be
running. Freshly booted VMs may take 30-60s for the agent to start.

### 6.3 Parsing qm guest exec JSON Output

```bash
# Pipe through python3 for clean extraction
ssh ... "qm guest exec 200 -- sh -c '...'" 2>&1 | \
  python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('out-data',''))"
```

---

## Section 7: IaC Repository Structure

### 7.1 dominik/iac-homelab (Gitea)

```
iac-homelab/
├── epic-1-management/         # Mgmt VM (OpenTofu + Ansible)
├── epic-2-k8s/                # 6 VMs + RKE2 bootstrap (Ansible playbook)
├── epic-3-networking-secrets/ # Cilium + External Secrets (OpenTofu)
├── epic-4-storage/            # Ceph CSI (OpenTofu)
├── epic-5-gitops/             # ArgoCD + Root App (OpenTofu)
├── epic-7-mariadb-vm/         # MariaDB Galera VMs (standalone, not K8s)
├── clusters/main/apps/        # ArgoCD App-of-Apps manifests
├── clusters/main/operators/   # Operator Helm releases
├── clusters/main/databases/   # Database cluster manifests
├── clusters/main/backups/     # Velero config
└── .github/workflows/         # CI/CD pipelines (self-hosted runner)
```

### 7.2 Gitea Token Generation

When 1Password `op --reveal` fails (service account limitation),
generate a new Gitea API token directly on the Gitea CT:

```bash
# Gitea runs as user 'gitea', binary at /usr/local/bin/gitea
# Config at /etc/gitea/app.ini, DB at /var/lib/gitea/data/gitea.db
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "pct exec GITEA_CT -- su -s /bin/sh gitea -c \
   'GITEA_WORK_DIR=/var/lib/gitea /usr/local/bin/gitea \
    admin user generate-access-token \
    -u dominik -t TOKEN_NAME --scopes all --raw \
    --config /etc/gitea/app.ini'"
```

**Pitfall**: Gitea 1.25+ uses `generate-access-token` (hyphenated),
not `generate accesstoken`. Run `gitea admin user --help` first to
verify the subcommand name for your version.

**Pitfall**: Gitea refuses to run as root. Must `su -s /bin/sh gitea`
and set `GITEA_WORK_DIR` environment variable.

### 7.3 Existing Gitea Tokens

Tokens are stored hashed in the Gitea SQLite DB — the full token value
cannot be recovered. Only the last 8 characters are visible:

```sql
SELECT id, uid, name, token_last_eight FROM access_token;
```

If a token is lost, generate a new one (Section 7.2).

---

## Section 8: Health Diagnostics

### 8.1 Full Cluster Health Check

```bash
KUBECONFIG=/root/.kube/config

# Nodes
kubectl get nodes -o wide

# Non-running pods (problems)
kubectl get pods -A | grep -v Running | grep -v Completed

# ArgoCD app sync status
kubectl get applications -n argocd

# External secrets
kubectl get externalsecrets -A

# LoadBalancer services
kubectl get svc -A -o wide | grep LoadBalancer

# Cilium LB pool
kubectl get ciliumloadbalancerippool
```

### 8.2 Common Cold-Boot Issues

| Issue | Resolution |
|-------|-----------|
| Workers NotReady for 2+ min | Check `journalctl -u rke2-agent` on worker — usually waiting for serving-kubelet.crt (503 from CP) |
| ceph-csi CrashLoopBackOff | Ceph cluster may not be reachable from all nodes; check Ceph mon addresses |
| External Secrets SecretSyncedError | 1Password connectivity — check if `onepassword-store` ClusterSecretStore is Ready |
| ArgoCD pods Terminating | Old replicas cleaning up — wait for deployment controller to reconcile |
| kube-controller-manager CrashLoopBackOff | Usually transient during etcd convergence; resolves when quorum is stable |

### 8.3 Checking RKE2 Service Logs

```bash
# On CP nodes
qm guest exec CP_VMID -- journalctl -u rke2-server --no-pager -n 20

# On Worker nodes
qm guest exec WORKER_VMID -- journalctl -u rke2-agent --no-pager -n 20
```

---

## Section 9: RKE2 Version Upgrades

### 9.1 GitOps-First Principle

**CRITICAL**: RKE2 upgrades must be performed via the IaC repo
(`epic-2-k8s/ansible/playbook.yml`) and Ansible — NEVER by manually
downloading tarballs and extracting them on individual nodes. The
Ansible playbook uses the `lablabs.rke2` role which handles proper
installation, rolling restarts, and configuration management.

The version is controlled by a single variable in the playbook:
```yaml
rke2_version: v1.35.2+rke2r1  # Change this to upgrade
```

### 9.2 Prerequisites for Ansible-Based Upgrade

1. **Ansible role installed** on mgmt-runner (VM200):
   ```bash
   cd /root/iac-homelab/epic-2-k8s/ansible
   ansible-galaxy install -r requirements.yml -p /root/.ansible/roles
   ```

2. **Inventory generated** — `inventory.yml` is normally generated by
   OpenTofu (`tofu apply` creates it via `local_file` resource). If
   tofu isn't available, create it manually from known IPs:
   ```yaml
   all:
     vars:
       ansible_user: debian
     children:
       k8s_cluster:
         children:
           masters:
             hosts:
               rke2-cp-01:
                 ansible_host: 10.0.30.51
                 rke2_type: server
                 rke2_server_init: true
               rke2-cp-02:
                 ansible_host: 10.0.30.52
               rke2-cp-03:
                 ansible_host: 10.0.30.53
           workers:
             hosts:
               rke2-worker-01:
                 ansible_host: 10.0.30.61
               rke2-worker-02:
                 ansible_host: 10.0.30.62
               rke2-worker-03:
                 ansible_host: 10.0.30.63
   ```

3. **RKE2 token** — extract from an existing node's config:
   ```bash
   # Find which PVE node hosts a CP VM, then:
   qm guest exec CP_VMID -- sh -c "cat /etc/rancher/rke2/config.yaml"
   # Token line: token: <VALUE>
   # Export before running Ansible:
   export RKE2_TOKEN="<value>"
   ```

4. **SSH access from mgmt-runner to K8s VMs** — Ansible connects as
   `debian` user via SSH. The mgmt-runner's SSH public key must be in
   `/home/debian/.ssh/authorized_keys` on all 6 K8s VMs. If missing,
   `ansible` will fail with `Permission denied (publickey)`.

   **Generate key on VM200** (if none exists):
   ```bash
   ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
     "ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N '' -C 'vm200-ansible'"
   ```

   **Distribute to all 6 K8s VMs** via `qm guest exec` (must discover
   which PVE node hosts each VM first — see Section 6.1):
   ```bash
   PUBKEY=$(ssh root@10.0.30.124 "cat ~/.ssh/id_ed25519.pub")

   # For each VM, on its hosting PVE node:
   ssh root@PVE_NODE "qm guest exec VMID --timeout 15 -- sh -c \
     'mkdir -p /home/debian/.ssh && echo \"$PUBKEY\" >> /home/debian/.ssh/authorized_keys \
      && chown -R debian:debian /home/debian/.ssh \
      && chmod 700 /home/debian/.ssh && chmod 600 /home/debian/.ssh/authorized_keys \
      && echo OK'"
   ```

   **Verify connectivity** before running the playbook:
   ```bash
   ssh root@10.0.30.124 "cd /root/iac-homelab/epic-2-k8s/ansible && ansible all -m ping"
   # All 6 nodes should return SUCCESS => { "ping": "pong" }
   ```

### 9.3 Performing the Upgrade

```bash
# On mgmt-runner (VM200):
cd /root/iac-homelab/epic-2-k8s/ansible

# 1. Edit the version in the playbook
# Change rke2_version: v1.35.2+rke2r1 → v1.35.6+rke2r1

# 2. Commit to Git (GitOps)
git add -A
git commit -m "upgrade-rke2-v1.35.6"
git push origin main

# 3. Run the playbook
export RKE2_TOKEN="<token-from-existing-node>"
ansible-playbook playbook.yml
```

The playbook has TWO plays:
- Play 1: Installs/upgrades RKE2 on all nodes via `lablabs.rke2` role
- Play 2: Rolling restart — Workers first (`serial: 1`), then CP nodes
  (`serial: 1`). Each node waits for RKE2 service active + node Ready
  before proceeding to the next.

### 9.4 Post-Upgrade: Transient CP Restart Failures

After the playbook completes, the PLAY RECAP may show `failed=1` on a CP
node even though the upgrade succeeded. This happens when Ansible's
`systemctl restart rke2-server` races with RKE2's internal reload from
the binary swap — the service had already started before Ansible tried
to restart it, causing a transient "already activating" error.

**Diagnosis**: Check the actual node state, don't trust the PLAY RECAP alone:
```bash
# On the "failed" CP node via qm guest exec:
qm guest exec CP_VMID -- sh -c "systemctl is-active rke2-server && rke2 --version"

# Cluster-wide:
kubectl get nodes -o wide
# If node is Ready + target version → failure was benign, no action needed
```

**Key insight**: Always verify with `kubectl get nodes -o wide` after the
playbook. If all nodes are Ready and on the target version with 0
non-running pods, the upgrade succeeded regardless of Ansible's exit
code on individual nodes.

### 9.5 Blue-Green Alternative (User Preference)

For major version upgrades or when rolling upgrades are too risky,
the user prefers blue-green:
1. Create new VMs with new RKE2 version (via OpenTofu + Ansible)
2. Join them to the existing cluster
3. Verify new nodes are Ready
4. Drain + cordon old nodes (migrate workloads)
5. Remove old nodes from cluster
6. Delete old VMs

### 9.6 CRITICAL PITFALL: RKE2 Tarball Extraction Destroys Debian 12

**NEVER** extract `rke2.linux-amd64.tar.gz` directly to `/` on a
Debian 12 system. The tarball contains:
```
bin/    → extracted to /bin/ (OVERWRITES /bin symlink!)
lib/    → extracted to /lib/ (OVERWRITES /lib symlink!)
share/  → extracted to /share/ (creates bogus directory)
```

Debian 12 uses **usrmerge**: `/bin → usr/bin`, `/lib → usr/lib`,
`/sbin → usr/sbin` are symlinks. The tarball extraction replaces
these symlinks with real directories containing only RKE2 files,
making ALL system binaries (`ls`, `sh`, `bash`, `systemctl`, `mount`)
inaccessible. The VM becomes unmanageable — no new processes can
exec, `qm guest exec` returns exit code 29.

**Correct extraction target**: `/usr/local/` (the tarball's `bin/`
contents go to `/usr/local/bin/`, etc.)

**If already broken**, see Section 10 for recovery procedure.

---

## Section 10: Recovering a Broken VM (Symlink Destruction)

### 10.1 Symptoms

- `qm guest exec VMID -- /bin/echo hello` → exit code 29
- `qm guest exec VMID -- sh -c "..."` → exit code 29
- `qm guest exec VMID -- /usr/bin/ls` → exit code 29
- K8s node still shows `Ready` (kernel + RKE2 processes still running)
- But no NEW processes can start (broken `/bin` and `/lib`)

### 10.2 Recovery: Ceph RBD Offline Mount

When the VM's disk is on Ceph RBD (`vm_disks` storage pool):

```bash
# 1. Stop the VM (preserve etcd quorum if CP node — need 2/3 alive)
qm stop VMID

# 2. On the PVE node hosting the VM, map the RBD image
rbd map vm_disks/vm-VMID-disk-0
# Returns /dev/rbdN (note the number!)

# 3. Wait for udev, then check partitions
sleep 2
lsblk | grep rbd
# Typically: rbdNp1 = root fs, rbdNp14 = BIOS boot, rbdNp15 = EFI

# 4. Mount the root partition
mkdir -p /mnt/vm-rescue
mount /dev/rbdNp1 /mnt/vm-rescue

# 5. Verify the damage
ls -la /mnt/vm-rescue/bin   # Should be symlink but is real dir
ls -la /mnt/vm-rescue/lib   # Should be symlink but is real dir

# 6. Fix symlinks
# Save any RKE2 binaries first:
cp /mnt/vm-rescue/bin/rke2 /mnt/vm-rescue/usr/local/bin/rke2
cp /mnt/vm-rescue/bin/rke2-*.sh /mnt/vm-rescue/usr/local/bin/
# Copy systemd units if present:
mkdir -p /mnt/vm-rescue/usr/lib/systemd/system
cp -a /mnt/vm-rescue/lib/systemd/system/rke2-*.service /mnt/vm-rescue/usr/lib/systemd/system/
cp -a /mnt/vm-rescue/lib/systemd/system/rke2-*.env /mnt/vm-rescue/usr/lib/systemd/system/

# Remove fake directories, restore symlinks:
rm -rf /mnt/vm-rescue/bin
ln -s usr/bin /mnt/vm-rescue/bin
rm -rf /mnt/vm-rescue/lib
ln -s usr/lib /mnt/vm-rescue/lib
rm -rf /mnt/vm-rescue/share  # Bogus dir from tarball

# 7. Verify fix
ls -la /mnt/vm-rescue/bin  # Should show: bin -> usr/bin
ls -la /mnt/vm-rescue/lib  # Should show: lib -> usr/lib
ls /mnt/vm-rescue/bin/bash # Should exist
ls /mnt/vm-rescue/bin/systemctl  # Should exist

# 8. Unmount, unmap, start VM
umount /mnt/vm-rescue
rbd unmap /dev/rbdN
qm start VMID

# 9. Wait 30-60s for boot, verify guest agent
qm guest exec VMID -- sh -c "echo BOOT_OK && systemctl is-active rke2-server"
```

### 10.3 RBD Mapping Pitfalls

- **RBD device disappears**: If `rbd map` succeeds but `/dev/rbdN`
  vanishes before mount, re-run `rbd map` (previous mapping was stale).
  Always `sleep 2` after mapping for udev to settle.
- **Partition devices not created**: `lsblk` shows partitions but
  `/dev/rbdNp1` doesn't exist as a block device. Run `partprobe
  /dev/rbdN` or just re-map (unmap + map again).
- **kpartx not available**: PVE nodes may not have `kpartx`. The
  kernel usually creates partition devices automatically after
  `rbd map` + `sleep 2`. If not, `partprobe /dev/rbdN` helps.
- **Don't mount while VM is running**: RBD is exclusive — stop the VM
  first. Mounting a live RBD image will corrupt the filesystem.

### 10.4 Post-Recovery: Restore Correct RKE2 Version

If the tarball extraction was for a NEW version (e.g. v1.35.6) but
the cached runtime images on the node are for the OLD version (e.g.
v1.35.2), RKE2 will fail to start with:
```
failed to pull images: image "rancher/rke2-runtime:v1.35.6-rke2r1": not found
```

**Fix**: Download the MATCHING tarball version and extract correctly:
```bash
# On the broken VM (after symlink repair + reboot):
cd /tmp
wget https://github.com/rancher/rke2/releases/download/v1.35.2%2Brke2r1/rke2.linux-amd64.tar.gz
tar xf rke2.linux-amd64.tar.gz -C /usr/local/  # CORRECT target!
systemctl restart rke2-server
```

Then perform the version upgrade properly via Ansible (Section 9).

---

## Section 11: OS Updates on K8s VMs

### 11.1 Updating Via qm guest exec

K8s VMs don't have SSH keys from the mgmt-runner, so Ansible can't
reach them directly. Use `qm guest exec` through Proxmox:

```bash
# Discover which PVE node hosts each VM first (Section 6.1)
# Then run apt upgrade:
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_JUMPHOST \
  "ssh root@PVE_NODE 'qm guest exec VMID --timeout 600 -- sh -c \"
    apt-get update && apt-get upgrade -y && apt-get autoremove -y
  \"'"
```

### 11.2 Interrupted dpkg Recovery

If apt upgrade is interrupted (timeout, connection drop), dpkg is left
in a half-configured state. This blocks ALL subsequent apt operations
on that node — including the Ansible playbook's pre_tasks (which install
`ceph-common` and `rbd-nbd`). The Ansible run will fail with:
```
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a'
```

**Simple fix** (if only dpkg lock is held):
```bash
qm guest exec VMID --timeout 300 -- sh -c "dpkg --configure -a && apt-get update && apt-get upgrade -y"
```

**Full fix** (if dpkg AND debconf locks are held by stale processes):
```bash
# Via qm guest exec on the PVE node hosting the VM:
qm guest exec VMID --timeout 120 -- bash -c "
  fuser -k /var/lib/dpkg/lock-frontend 2>/dev/null;
  fuser -k /var/lib/dpkg/lock 2>/dev/null;
  fuser -k /var/cache/debconf/config.dat 2>/dev/null;
  sleep 2;
  rm -f /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock;
  DEBIAN_FRONTEND=noninteractive dpkg --configure -a &&
  apt-get install -y ceph-common rbd-nbd
"
```

**Pitfall: debconf lock** — Even after clearing dpkg locks, `debconf`
has its own lock at `/var/cache/debconf/config.dat`. If a stale process
holds it, `dpkg --configure -a` fails with:
```
debconf: DbDriver "config": /var/cache/debconf/config.dat is locked by another process
```
Must `fuser -k /var/cache/debconf/config.dat` before retrying.

**Pitfall: Interactive prompts hang under qm guest exec** — Without
`DEBIAN_FRONTEND=noninteractive`, dpkg may prompt for config file
conflicts (e.g. `sshd_config` was locally modified) and hang indefinitely
under `qm guest exec` (no TTY). Always set
`DEBIAN_FRONTEND=noninteractive` to auto-keep-existing config.

**Cross-reference**: This must be fixed BEFORE running the Ansible
upgrade playbook (Section 9.3) — the `lablabs.rke2` role's pre_tasks
install `ceph-common` and `rbd-nbd` via apt, which will fail if dpkg
is in an interrupted state. Fix all affected nodes first, then re-run
`ansible-playbook playbook.yml`.

### 11.3 Direct SSH to mgmt-runner

VM200 (mgmt-runner) at 10.0.30.124 accepts direct SSH from the Hermes
host:
```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124
```
This bypasses the Proxmox jumphost chain and is faster for running
kubectl/helm/git commands.

---

## Section 12: Actual VM→PVE-Node Mapping (July 2026)

The tofu state records where VMs were originally placed, but PVE HA
migrates VMs. The ACTUAL mapping (as of 2026-07-12) is:

| VM | Name | PVE-Node | IP |
|----|------|----------|-----|
| 118 | rke2-cp-01 | proxmox5 | 10.0.30.51 |
| 130 | rke2-cp-02 | proxmox3 | 10.0.30.52 |
| 129 | rke2-cp-03 | proxmox2 | 10.0.30.53 |
| 128 | rke2-worker-01 | proxmox | 10.0.30.61 |
| 132 | rke2-worker-02 | proxmox2 | 10.0.30.62 |
| 131 | rke2-worker-03 | proxmox3 | 10.0.30.63 |
| 200 | mgmt-runner | proxmox3 | 10.0.30.124 |

**Always re-discover** before operating — HA may have moved VMs since
this table was written.

---

## Section 13: Helm Chart Upgrades

### 13.1 Chart Inventory & Version Matrix

Before upgrading, inventory all Helm releases and compare against
latest available versions:

```bash
# On VM200 (mgmt-runner):
export KUBECONFIG=/root/.kube/config

# 1. List all Helm releases across namespaces
helm list -A

# 2. Update repo cache
helm repo update

# 3. Check latest available versions
helm search repo argo/argo-cd | head -3
helm search repo external-secrets/external-secrets | head -3
# etc.

# 4. RKE2-bundled charts (already updated with RKE2 upgrade):
helm list -n kube-system | grep rke2-
# These are: cilium, traefik, coredns, metrics-server, snapshot-controller
```

Build a matrix: Component | Current Version | Latest Available |
Update Type (Major/Minor/Patch) | Management Method (Helm-direct vs ArgoCD GitOps).

**RKE2-bundled charts** (Cilium, Traefik, CoreDNS, Metrics Server,
Snapshot Controller) are automatically updated when RKE2 is upgraded —
no separate Helm upgrade needed.

### 13.2 ArgoCD GitOps-Managed Charts (CNPG, Velero)

Charts deployed via ArgoCD Applications are upgraded by changing the
`targetRevision` in the Application manifest in the IaC repo:

```bash
cd /root/iac-homelab
# Find the Application manifest
grep -r "targetRevision" clusters/main/

# Update the chart version (e.g. CNPG 0.22.1 → 0.29.0)
sed -i 's/targetRevision: 0.22.1/targetRevision: 0.29.0/' \
  clusters/main/operators/cloudnativepg.yaml
git add -A && git commit -m "feat: upgrade CNPG chart" && git push
```

**Pitfall: ArgoCD doesn't auto-refresh after chart version bump** —
Even with `selfHeal: true`, ArgoCD may keep the old chart revision
cached. Force a hard refresh:
```bash
kubectl annotate application <app-name> -n argocd \
  argocd.argoproj.io/refresh=hard --overwrite
```

Wait for the new revision to appear in `kubectl get applications -n argocd`,
then monitor the rollout:
```bash
kubectl rollout status deployment/<deployment-name> -n <namespace> --timeout=300s
```

### 13.3 Helm-Direct Charts (ArgoCD, External Secrets, Ceph CSI)

Charts installed directly via `helm install` are upgraded with
`helm upgrade`:

```bash
# Save current values before upgrading
helm get values <release-name> -n <namespace> > /tmp/<release>-values.yaml

# Dry-run first
helm upgrade <release-name> <repo>/<chart> \
  --version <new-version> \
  -n <namespace> \
  --reuse-values \
  --dry-run

# Actual upgrade
helm upgrade <release-name> <repo>/<chart> \
  --version <new-version> \
  -n <namespace> \
  --reuse-values \
  --timeout 300s
```

**Pitfall: `--reuse-values` may not suffice for major chart versions** —
Major chart versions can introduce required fields that don't exist in
old values. If `helm upgrade` fails with a template error (e.g. `nil
pointer evaluating interface {}`), create a complete values file that
includes the new required fields:
```bash
# Instead of --reuse-values, provide explicit values:
helm upgrade <release-name> <repo>/<chart> \
  --version <new-version> \
  -n <namespace> \
  -f /tmp/<release>-v3-values.yaml \
  --timeout 300s
```

### 13.4 ArgoCD v2→v3 Major Upgrade (Breaking Change)

Upgrading the ArgoCD Helm chart from v7.x (app v2.x) to v10.x (app
v3.x) requires adding `redis.networkPolicy.create: false` to the
values file. Without this, the chart fails with:
```
template: argo-cd/templates/redis/networkpolicy.yaml:2:22:
nil pointer evaluating interface {}.create
```

**Safe upgrade path**: First upgrade to the latest v2.x patch
(chart 7.9.1 = app v2.14.11), then jump to v3.x:
```bash
# Step 1: Safe patch to latest v2.x
helm upgrade argocd argo/argo-cd --version 7.9.1 -n argocd --reuse-values

# Step 2: Create v3-compatible values file with redis.networkPolicy
cat > /tmp/argocd-v3-values.yaml << 'EOF'
# ... copy all existing values, then add:
redis:
  networkPolicy:
    create: false
EOF

# Step 3: Major upgrade to v3
helm upgrade argocd argo/argo-cd --version 10.1.3 -n argocd \
  -f /tmp/argocd-v3-values.yaml --timeout 300s
```

**Save values files to the IaC repo** (`epic-2-k8s/helm/<chart>/values.yaml`)
for reproducible installs.

### 13.5 CNPG Replica Rebuild from WAL Gap

When a CNPG PostgreSQL replica falls behind and the primary has
already recycled the needed WAL segments, the replica is permanently
broken:
```
requested WAL segment 000000020000000000000008 has already been removed
```

**Fix**: Delete the broken replica's PVCs and pod — CNPG automatically
rebuilds from a fresh base backup of the primary:
```bash
# 1. Identify the broken replica pod
kubectl get pods -n postgres | grep -v Running
# e.g. postgres-main-2 is in a bad state

# 2. Delete the PVCs (data + wal)
kubectl delete pvc postgres-main-2 postgres-main-2-wal -n postgres

# 3. Delete the pod so CNPG recreates it
kubectl delete pod postgres-main-2 -n postgres

# 4. CNPG creates a new replica (e.g. postgres-main-4) automatically
# Wait for it to join:
kubectl get pods -n postgres -w
# Watch for 3/3 Running
```

**Do this BEFORE upgrading the CNPG operator** — upgrading with a
broken replica is risky. The operator upgrade itself may trigger a
primary restart, and having only 1 healthy replica reduces safety margin.

### 13.6 Ceph CSI RBD Upgrade — Transient API Server Disruption

**Pitfall**: Upgrading the Ceph CSI RBD chart triggers a DaemonSet
rolling update of `ceph-csi-rbd-nodeplugin` pods across ALL nodes,
including control plane nodes. The nodeplugin pods mount/unmount CSI
driver sockets, which can briefly disrupt the kubelet on CP nodes and
cause all 3 API servers to become unreachable for 2-5 minutes.

**Symptoms during the disruption**:
```
The connection to the server 10.0.30.51:6443 was refused
```

**Mitigation**:
- Schedule Ceph CSI upgrades during maintenance windows
- After `helm upgrade`, monitor with `curl -sk https://CP_IP:6443/healthz`
  until it responds (even 401 Unauthorized means the server is up)
- Once API server recovers, verify with `kubectl get pods -n kube-system | grep ceph-csi-rbd`
- New nodeplugin pods may stay in `ContainerCreating` for several
  minutes while pulling the new CSI image — this is normal
- All PVCs remain bound throughout; no data loss occurs

### 13.7 ExternalSecret Perpetual OutOfSync in ArgoCD

**Pitfall**: The External Secrets Operator adds default fields to
ExternalSecret specs that aren't present in the Git YAML:
- `remoteRef.conversionStrategy: "Default"`
- `remoteRef.decodingStrategy: "None"`
- `remoteRef.metadataPolicy: "None"`
- `target.deletionPolicy: "Retain"`
- `target.template.engineVersion: "v2"`
- `target.template.mergePolicy: "Replace"`

ArgoCD sees these as spec drift and marks the ExternalSecret
perpetually `OutOfSync`. The `ignoreDifferences` rule for
`/status` doesn't cover `/spec` differences.

**Fix**: Either add the ESO default fields to the Git YAML (so spec
matches exactly), or add `ignoreDifferences` for the specific JSON
pointers. Adding fields to Git YAML is cleaner:
```yaml
spec:
  data:
    - secretKey: access_key_id
      remoteRef:
        key: velero-s3-backup/S3 Credentials/access_key_id
        conversionStrategy: Default
        decodingStrategy: None
        metadataPolicy: None
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    template:
      engineVersion: v2
      mergePolicy: Replace
```

### 13.8 Accidental .kube/cache Commit

**Pitfall**: Running `kubectl` commands from the IaC repo root creates
`.kube/cache/` with discovery API responses. If `git add -A` is used,
these cache files get committed. Always add `.kube/` to `.gitignore`:
```bash
echo ".kube/" >> .gitignore
```

---

## References

See `references/` directory for session-specific details.

- `references/cold-boot-recovery-2026-07.md` — Session detail: full cluster cold-boot, Cilium LB pool migration from .70-.89 to .200-.250, ArgoCD convergence timeline
- `references/rke2-cluster-topology.md` — Complete topology, IP map, operator inventory, IaC repo structure
- `references/hindsight-k8s-deployment-2026-07.md` — Deploying hindsight-api in K8s: ExternalSecret format, StorageClass, env var ordering, vault mismatch, ArgoCD cache pitfalls
- `references/rke2-upgrade-and-recovery-2026-07.md` — Session detail: RKE2 tarball destruction, Ceph RBD offline recovery, Ansible-based upgrade prerequisites, OS update procedures, openclaw-memory removal via GitOps, Ansible SSH key prerequisite
- `references/helm-chart-upgrades-2026-07.md` — Session detail: full Helm chart upgrade cycle (ArgoCD v2→v3, External Secrets, Ceph CSI, CNPG, Velero), CNPG replica WAL-gap rebuild, Ceph CSI transient API server disruption, ExternalSecret OutOfSync root cause
- `references/gitea-k8s-migration-2026-07.md` — Gitea LXC→K8s migration plan: Galera over CNPG decision, Gitea Migration API for SQLite→MySQL, chicken-egg ArgoCD cutover, git.schoen.codes external URL
- `references/gitea-helm-chart-v12-deployment-2026-07.md` — Gitea Helm Chart v12 deployment: additionalConfigFromEnvs for DB password, valkey-cluster disabling, RWO PVC blocking, Galera schema slowness, ArgoCD app split
- `references/traefik-lxc-to-k8s-migration-2026-07.md` — Traefik CT99999 → K8s migration: service inventory, static/dynamic config, ExternalName pattern, catchall-TCP pitfall, cutover strategy
- `references/argocd-self-management-cutover-2026-07.md` — ArgoCD chicken-and-egg cutover: CoreDNS internal DNS override, repo secret labeling, Application URL patching, self-management verification
- `references/hermes-git-remote-cutover-2026-07.md` — Hermes host git remote switch from CT108 to K8s Gitea: /etc/hosts for external CoreDNS, token non-migration, Traefik VIP reachability, no kubectl on Hermes host
- `references/influxdb-k8s-migration-2026-07.md` — InfluxDB CT109+CT134 → K8s migration: fresh deploy + replication strategy (Option B chosen over backup/restore), Portainer stack discovery on 10.0.30.100, Cilium VIP .204 allocation, downsampling task string-field crash fix, Compound Engineering brainstorming-first workflow correction
- `references/gitea-backup-cronjob-2026-07.md` — Gitea backup CronJob: multi-initContainer pattern (kubectl exec tar for PVC + mysqldump for Galera DB), RBAC for pods/exec, S3 bucket per service, schedule staggering, bitnami/kubectl image pull failures, kubectl cp large file failures, ArgoCD false sync success, VM200 stale remote, mgmt-runner fallback
- `references/hermes-memory-system-architecture.md` — Hermes memory injection pipeline: MEMORY.md/USER.md loading, frozen snapshots, system prompt tiers, config limits, alternative context injection points (SOUL.md, AGENTS.md, .cursorrules), `bitnami/kubectl:1.31` tag不存在, `kubectl cp` large file websocket reset, VM200 repo remote staleness, S3 `--region us-east-1` for `s3 mb`, mgmt-runner fallback approach