hermes-skills/devops/rke2-cluster-administration/SKILL.md

---
name: rke2-cluster-administration
description: "Class-level skill for RKE2 Kubernetes cluster administration on Proxmox VE. Covers cluster lifecycle (start/stop/reboot), cold-boot recovery, Cilium LB IP pool management, ArgoCD GitOps operations, node health diagnostics, and nested Proxmox SSH patterns for K8s VM management."
version: 1.0.0
tags: [rke2, kubernetes, k8s, cilium, argocd, gitops, proxmox, loadbalancer, lb-pool, cold-boot]
---

## Overview

This skill covers administration of an RKE2 Kubernetes cluster running on
Proxmox VE QEMU VMs. The cluster uses:
- **RKE2** v1.35.x (CNCF-conformant K8s)
- **Cilium** as CNI (eBPF-based, with L2 announcements + LB IP pools)
- **ArgoCD** for GitOps (App-of-Apps pattern)
- **External Secrets Operator** synced to 1Password
- **CloudNativePG** for PostgreSQL
- **Velero** for backups
- **Ceph CSI** for RBD storage
- **Traefik** as ingress controller (RKE2-bundled)

**Note**: MariaDB Galera runs as **native VMs** (VM300-302 + MaxScale
VM310), NOT in K8s. A K8s MariaDB Operator was previously deployed but
has been removed — Galera VMs are the authoritative database.

Load this skill for any RKE2 cluster lifecycle task, node diagnostics,
ArgoCD/GitOps operations, Cilium LB pool changes, or K8s service migration.

---

## Section 1: Cluster Architecture & Access

### 1.1 Cluster Topology

| Role | VMID | Node | IP | Specs |
|------|------|------|----|-------|
| CP-01 | 118 | varies (HA) | 10.0.30.51 | 4c / 12GB / 40GB |
| CP-02 | 130 | varies (HA) | 10.0.30.52 | 4c / 12GB / 40GB |
| CP-03 | 129 | varies (HA) | 10.0.30.53 | 4c / 12GB / 40GB |
| Worker-01 | 128 | varies (HA) | 10.0.30.61 | 4c / 12GB / 80GB |
| Worker-02 | 132 | varies (HA) | 10.0.30.62 | 4c / 12GB / 80GB |
| Worker-03 | 131 | varies (HA) | 10.0.30.63 | 4c / 12GB / 80GB |

**RKE2 API VIP**: 10.0.30.50 (Keepalived on CP nodes)

**Note**: PVE HA migrates VMs between nodes. Always discover which node
hosts a given VM before operating on it (see Section 6.1).

### 1.2 Management VM (kubectl access)

VM 200 (`mgmt-runner-01`) has `kubectl`, `helm`, and `git` installed.
It also serves as the self-hosted GitHub Actions runner.

**Kubeconfig location**: `/root/.kube/config` on VM 200.
Must set `KUBECONFIG=/root/.kube/config` explicitly — kubectl does not
pick it up from `~/.kube/config` via `qm guest exec`.

```bash
# Run kubectl on mgmt-runner via Proxmox
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "ssh -o ConnectTimeout=5 root@NODE_HOSTING_VM200 \
   'qm guest exec 200 -- sh -c \"KUBECONFIG=/root/.kube/config kubectl get nodes\"'"
```

**Pitfall**: `kubectl` without explicit `KUBECONFIG=` returns
`localhost:8080 connection refused` — the default kubeconfig path isn't
honored through `qm guest exec`.

### 1.3 Getting Kubeconfig from a Control Plane Node

If the mgmt-runner doesn't have kubeconfig yet:

```bash
# Get kubeconfig from CP-01 (VM 118)
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "qm guest exec 118 -- cat /etc/rancher/rke2/rke2.yaml"

# Fix the server URL from 127.0.0.1 to the CP IP
# Replace 127.0.0.1 with 10.0.30.51 (CP-01 IP)
# Base64-transfer to mgmt-runner (see docker-host-administration Section 8.4)
```

---

## Section 2: Cluster Lifecycle (Start/Stop/Reboot)

### 2.1 Starting the Cluster from Cold Boot

All 6 VMs were stopped → start them in order:

```bash
# 1. Start Control Plane nodes first
# CPs: 118, 130, 129 — may be on different PVE nodes due to HA
# Start each individually to avoid HA placement races

# 2. Wait 15-30s for CPs to begin etcd convergence
sleep 15

# 3. Start Worker nodes
# Workers: 128, 131, 132

# 4. Wait for cluster to converge (2-5 minutes)
# Workers initially show NotReady — they need to retrieve
# serving-kubelet.crt from the API server (503 until CPs are ready)
```

**Pitfall**: `qm start` returns immediately ("Requesting HA start") but
HA may place the VM on a different node than expected. Always scan all
PVE nodes to find where each VM landed:

```bash
for node in proxmox1 proxmox2 proxmox3 proxmox4 proxmox5 proxmox6 proxmox7; do
  ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
    "ssh -o ConnectTimeout=3 root@$node 'qm list | grep rke2'" 2>/dev/null
done
```

### 2.2 Cold-Boot Recovery Timeline

After all VMs are started, expect this convergence sequence:

| Time | What happens |
|------|-------------|
| 0-30s | CP nodes boot, etcd begins forming quorum |
| 30-60s | Workers connect to CP via proxy (`wss://CP_IP:9345/v1-rke2/connect`) |
| 1-2min | Workers retrieve serving-kubelet.crt (was 503 during CP convergence) |
| 2-3min | Cilium agents start on all nodes, networking comes up |
| 3-5min | ArgoCD, External Secrets, operators begin reconciling |
| 5-10min | Most pods Running, some Terminating (old replicas from before shutdown) |
| 10-15min | Databases (Galera, PostgreSQL) reach quorum and become ready |

**Patience is key.** Don't panic when pods show Unknown/Terminating
early on — this is normal cold-boot behavior, not a failure.

### 2.3 Stopping the Cluster

```bash
# 1. Stop workers first (allows graceful pod eviction)
# 2. Then stop CPs (etcd will downsize gracefully)
# 3. Verify all stopped
```

Workers stopping first prevents etcd from losing quorum abruptly.

---

## Section 3: Cilium LoadBalancer IP Pool Management

### 3.1 The LB Pool Problem

Cilium assigns LoadBalancer IPs to K8s services (Traefik, MaxScale,
PostgreSQL) from a configurable IP pool. If this pool overlaps with
existing infrastructure IPs (Galera VIPs, MaxScale VIPs, static
services), you get **IP conflicts**.

### 3.2 Checking Current Pool

```bash
KUBECONFIG=/root/.kube/config kubectl get ciliumloadbalancerippool -o yaml
```

Key fields:
```yaml
spec:
  blocks:
  - start: "10.0.30.200"   # Pool start
    stop: "10.0.30.250"    # Pool end
```

### 3.3 Changing the Pool (Free Up Conflicting IPs)

When K8s LB IPs collide with existing infrastructure:

```bash
# 1. Delete the old pool
kubectl delete ciliumloadbalancerippool default-pool

# 2. Apply new pool with non-overlapping range
cat <<'EOF' | kubectl apply -f -
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: default-pool
spec:
  blocks:
  - start: "10.0.30.200"
    stop: "10.0.30.250"
  disabled: false
EOF

# 3. Verify services picked up new IPs
kubectl get svc -A -o wide | grep LoadBalancer
```

Services automatically release old IPs and get new ones from the
updated pool. This is non-disruptive — Cilium handles the transition.

### 3.4 Making the Change Persistent

The pool is defined in a manifest on all CP nodes:
`/var/lib/rancher/rke2/server/manifests/cilium-l2-lb.yaml`

RKE2 auto-applies manifests from this directory on CP restart. If you
only patch the live CRD, the old pool comes back on next CP reboot.

**Fix the manifest on ALL 3 CP nodes** (use base64 transfer — see
docker-host-administration Section 8.4):

```bash
NEW_MANIFEST=$(cat <<'EOF'
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: default-pool
spec:
  blocks:
    - start: "10.0.30.200"
      stop: "10.0.30.250"
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: default-l2-policy
spec:
  interfaces:
    - ^eth[0-9]+
  externalIPs: true
  loadBalancerIPs: true
EOF
)

B64=$(echo "$NEW_MANIFEST" | base64 -w0)

# Write to all 3 CPs (discover which PVE node hosts each first!)
for combo in "NODE1 118" "NODE2 130" "NODE3 129"; do
  node=$(echo $combo | cut -d' ' -f1)
  vmid=$(echo $combo | cut -d' ' -f2)
  ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
    "ssh -o ConnectTimeout=5 root@$node \
     'qm guest exec $vmid -- sh -c \"echo $B64 | base64 -d > /var/lib/rancher/rke2/server/manifests/cilium-l2-lb.yaml\"'"
done
```

### 3.5 Fixing the IaC Source

The pool is also defined in the IaC repository (Ansible playbook that
bootstraps RKE2). Update and push:

```bash
# In iac-homelab repo: epic-2-k8s/ansible/playbook.yml
# Change the LB pool start/stop values to match the new range
sed -i 's/OLD_START/NEW_START/g' epic-2-k8s/ansible/playbook.yml
sed -i 's/OLD_STOP/NEW_STOP/g' epic-2-k8s/ansible/playbook.yml
git add -A && git commit -m "fix-k8s-move-cilium-lb-pool" && git push origin main
```

**Three places to update** (all must agree):
1. Live CRD (`kubectl patch` / delete+apply)
2. CP manifest files (`/var/lib/rancher/rke2/server/manifests/cilium-l2-lb.yaml`)
3. IaC repo (`epic-2-k8s/ansible/playbook.yml`)

---

## Section 4: ArgoCD GitOps

### 4.1 ArgoCD Access

- **UI**: `https://<TRAEFIK_LB_IP>/` with Host header `argocd`
  (Traefik routes by hostname)
- **Admin password**: Retrieved from kubectl:
  ```bash
  kubectl -n argocd get secret argocd-initial-admin-secret \
    -o jsonpath='{.data.password}' | base64 -d
  ```
- **CLI**: `argocd` binary (if installed on mgmt-runner)

### 4.2 App-of-Apps Pattern

The root ArgoCD Application points to a Git repo path containing
child Application manifests:

```yaml
# Root Application (synced from epic-5-gitops/tofu)
spec:
  source:
    repoURL: http://GITEA_URL/dominik/iac-homelab.git
    path: clusters/main/apps
    targetRevision: main
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
```

Child apps in `clusters/main/apps/`:
- `operators.yaml` (sync-wave 0): CloudNativePG, Velero
- `databases.yaml` (sync-wave 1): PostgreSQL
- `backups.yaml` (sync-wave 2): Velero schedules + locations

**Removed apps**: `memory.yaml` (sync-wave 3: Qdrant, Ollama, Memory API)
was removed 2026-07-12 — the `openclaw-memory` namespace is fully deleted.
See `references/rke2-upgrade-and-recovery-2026-07.md` Phase 3.

### 4.3 Adding a New Application

To deploy a new service via GitOps:

1. Create namespace manifest: `clusters/main/<service>/namespace.yaml`
2. Create deployment/service/ingress manifests in `clusters/main/<service>/`
3. Create ArgoCD Application manifest: `clusters/main/apps/<service>.yaml`
4. Push to Gitea — ArgoCD auto-syncs within minutes

**Pitfall: K8s env var interpolation order** — When using `$(VAR_NAME)`
to reference one env var inside another (e.g. building a connection
string from a secret), the referenced var MUST be listed **before** the
referencing var. K8s processes env vars top-to-bottom; if the reference
comes first, it resolves to an empty string.

```yaml
# CORRECT — HINDSIGHT_DB_PASSWORD before DATABASE_URL
env:
  - name: HINDSIGHT_DB_PASSWORD
    valueFrom:
      secretKeyRef: ...
  - name: DATABASE_URL
    value: "postgresql://user:$(HINDSIGHT_DB_PASSWORD)@host/db"

# WRONG — DATABASE_URL resolves with empty password
env:
  - name: DATABASE_URL
    value: "postgresql://user:$(HINDSIGHT_DB_PASSWORD)@host/db"
  - name: HINDSIGHT_DB_PASSWORD
    valueFrom:
      secretKeyRef: ...
```

**Pitfall: ExternalSecret API version** — The cluster's
ExternalSecrets Operator serves both `v1` and `v1beta1`, but existing
working ExternalSecrets all use `v1`. Using `v1beta1` can cause
`"the server could not find the requested resource"` errors in ArgoCD
even though the CRD technically serves both versions. Always use
`apiVersion: external-secrets.io/v1`.

**Pitfall: ArgoCD caches stale manifests** — When you fix a manifest
(e.g. wrong API version) and push, ArgoCD may keep trying the old
version. Deleting the Application and recreating it forces a clean
re-read from the repo. Alternatively:
```bash
kubectl annotate application -n argocd APP_NAME \
  argocd.argoproj.io/refresh=hard --overwrite
```

**Pitfall: StatefulSet PVC template immutability** — If a StatefulSet
has the wrong StorageClass in its `volumeClaimTemplates`, simply
updating the manifest and letting ArgoCD sync won't fix it — the old
PVC persists. Must delete the STS + PVC manually, then let ArgoCD
recreate with the corrected manifest:
```bash
kubectl delete sts STATEFULSET_NAME -n NS --cascade=foreground
kubectl delete pvc DATA_PVC_NAME -n NS
```

**Pitfall: `kubectl rollout restart` does NOT apply manifest changes**
— `rollout restart` only restarts pods with the *current* deployment
spec. It does NOT pull new manifest changes from the repo. After
pushing a manifest fix, you must `kubectl apply -f <manifest>` first
(or wait for ArgoCD to sync), THEN `rollout restart` if needed. If you
only `rollout restart`, the new pod starts with the old spec and hits
the same error.

**Pitfall: ArgoCD does not recreate manually deleted resources** —
When you `kubectl delete` a resource (e.g. an ExternalSecret) to force
re-sync, ArgoCD may report "successfully synced (all tasks run)" but
NOT recreate the deleted resource. This happens when the resource was
already synced once and ArgoCD considers it "managed" but doesn't
detect the drift. Fix: `kubectl apply -f <manifest>` manually, or
delete the entire ArgoCD Application and recreate it.

**Pitfall: PostgreSQL PVC `lost+found` conflict** — When mounting a PVC
directly at `/var/lib/postgresql/data`, PostgreSQL fails to initialize
because the PVC filesystem contains a `lost+found` directory (created by
mkfs). The PG initdb script refuses to use a non-empty data directory.
**Fix**: Set `PGDATA` env var to a subdirectory:
```yaml
env:
  - name: PGDATA
    value: /var/lib/postgresql/data/pgdata
```
The volumeMount stays at `/var/lib/postgresql/data` — PGDATA just tells
Postgres to use a clean subdirectory inside it.

```yaml
# clusters/main/apps/frigate.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: frigate
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "4"
spec:
  project: default
  source:
    repoURL: http://GITEA_URL/dominik/iac-homelab.git
    targetRevision: main
    path: clusters/main/frigate
  destination:
    server: https://kubernetes.default.svc
    namespace: frigate
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
```

### 4.4 Removing an Application / Operator via GitOps

When removing an operator-managed application (e.g. MariaDB Operator +
its MariaDB/MaxScale CRs) from ArgoCD:

**Pitfall**: Simply deleting the YAML files from the repo and pushing
does **NOT** cleanly prune operator-managed resources. ArgoCD marks the
app `OutOfSync` but the operator's Custom Resources (CRs) hold
finalizers that prevent deletion. The operator pods must be running to
process the finalizer — but if you delete the operator first, the CRs
are orphaned and the namespace hangs in `Terminating` forever.

**Correct removal sequence**:

1. **Remove YAML files from repo** (commit + push to Gitea)
2. **Manually delete the CRs first** (while operator is still running):
   ```bash
   kubectl delete mariadb mariadb-galera -n mariadb --timeout=120s
   kubectl delete maxscale mariadb-maxscale -n mariadb  # may 404 if already gone
   ```
3. **Wait for CRs to be fully deleted** (operator processes finalizers)
4. **Delete the namespace**:
   ```bash
   kubectl delete namespace mariadb --timeout=30s
   kubectl delete namespace mariadb-operator --timeout=30s
   ```
5. **Delete the CRDs** (ArgoCD may not prune these automatically):
   ```bash
   kubectl delete crd backups.k8s.mariadb.com connections.k8s.mariadb.com \
     databases.k8s.mariadb.com grants.k8s.mariadb.com \
     mariadbs.k8s.mariadb.com maxscales.k8s.mariadb.com \
     restores.k8s.mariadb.com sqljobs.k8s.mariadb.com \
     users.k8s.mariadb.com --timeout=30s
   ```
6. **Force ArgoCD refresh** if apps linger:
   ```bash
   kubectl annotate application root -n argocd \
     argocd.argoproj.io/refresh-options=hard-refresh --overwrite
   ```

**Key insight**: ArgoCD `prune: true` removes standard K8s resources
(deployments, services, configmaps) automatically, but **operator CRDs
and their CR instances often need manual cleanup** because finalizers
require the operator to be alive to process them.

### 4.4b Removing a Non-Operator Application via GitOps

For standard apps (Deployments, Services, StatefulSets without operator
CRs), removal is simpler — no finalizer issues:

1. **Remove YAML files from repo** (ArgoCD Application manifest +
   entire `clusters/main/<service>/` directory + any GitHub workflows):
   ```bash
   cd /root/iac-homelab
   rm clusters/main/apps/<service>.yaml
   rm -rf clusters/main/<service>/
   rm .github/workflows/<service>-*.yml  # if applicable
   git add -A && git commit -m "chore: remove <service> (unused)"
   git push origin main
   ```

2. **Delete the ArgoCD Application** (cascade=foreground cleans up
   managed resources):
   ```bash
   kubectl delete application <name> -n argocd --cascade=foreground
   ```

3. **Delete the namespace** (if not auto-pruned — ArgoCD's
   `CreateNamespace=true` creates it but doesn't always prune it):
   ```bash
   kubectl delete ns <namespace>
   # May take 10-30s if pods need to terminate first
   ```

4. **Verify**: `kubectl get ns <namespace>` → NotFound, `kubectl get pods -A
   | grep <namespace>` → empty.

**Pitfall**: ArgoCD Application deletion with `--cascade=foreground` may
leave the namespace in `Terminating` if pods have stuck finalizers. The
standard `kubernetes` finalizer resolves once pods are evicted. If it
hangs, check for leftover ExternalSecrets or PVCs with custom
finalizers.

### 4.5 ArgoCD Pod States During Cold Boot

After cluster cold-boot, ArgoCD pods show mixed states:
- Some `Running` (new replicas scheduled by deployment controller)
- Some `Terminating` (old replicas from before shutdown)
- Some `Unknown` (node was down when status was last reported)

These resolve naturally as the deployment controller reconciles. No
manual intervention needed unless pods are stuck for 10+ minutes.

### 4.6 Deploying Python Apps with pip in Init Containers

When deploying a Python app that isn't available as a Docker image
(e.g. `hindsight-api` from PyPI), the common pattern is an init
container that runs `pip install`, storing packages in an `emptyDir`
volume shared with the main container.

**Pitfall: `pip install --target=` does NOT install console_scripts**
— Using `pip install --target=/opt/packages` copies the Python modules
but does NOT create bin/ entry points (console_scripts). The main
container will fail with `command not found` even though the package
files are present. **Fix**: Use `python -m venv` instead:

```yaml
initContainers:
  - name: install-deps
    image: python:3.11-slim
    command: ["/bin/sh", "-c"]
    args:
      - |
        python -m venv /opt/venv &&
        /opt/venv/bin/pip install --no-cache-dir <package> &&
        echo "Done"
    volumeMounts:
      - name: venv
        mountPath: /opt/venv
containers:
  - name: app
    image: python:3.11-slim
    command: ["/bin/sh", "-c"]
    args: ["/opt/venv/bin/<binary> --host 0.0.0.0 --port 8888"]
    volumeMounts:
      - name: venv
        mountPath: /opt/venv
        readOnly: true
volumes:
  - name: venv
    emptyDir:
      sizeLimit: 3Gi
```

**Pitfall: Heavy dependencies pull CUDA torch (526MB)** — Packages
that depend on `torch` (e.g. `sentence-transformers`,
`hindsight-api`) will pull the full CUDA-enabled torch wheel by
default, which is 526MB+ and may OOMKill the init container. **Fix**:
Install CPU-only torch first with a dedicated index, then install the
main package with `--extra-index-url` so pip finds CPU torch:

```yaml
args:
  - |
    /opt/venv/bin/pip install --no-cache-dir \
      torch --index-url https://download.pytorch.org/whl/cpu &&
    /opt/venv/bin/pip install --no-cache-dir \
      <package-with-torch-dep> \
      --extra-index-url https://download.pytorch.org/whl/cpu &&
    echo "Done"
```

The `--index-url` for the first install ensures ONLY CPU wheels are
considered. The `--extra-index-url` for the second install allows pip
to find the already-satisfied CPU torch instead of upgrading to the
default CUDA version.

**Pitfall: Init container pip install is slow (~10min)** — Installing
100+ packages (including torch, transformers, sentence-transformers)
takes 5-10 minutes. The pod stays in `Init:0/1` during this time. Set
generous timeouts and don't assume CrashLoopBackOff means failure —
check the init container logs first.

---

## Section 5: External Secrets & 1Password

### 5.1 ClusterSecretStore

The cluster uses a single `ClusterSecretStore` backed by 1Password:

```bash
kubectl get clustersecretstore onepassword-store
# Should show READY=True
```

### 5.2 ExternalSecret Key Format (CRITICAL)

The `remoteRef.key` format is **`item-title/field-name`** — a simple
slash-separated path, NOT `op://` URIs and NOT a separate `property`
field.

```yaml
# CORRECT
spec:
  data:
    - secretKey: api-key
      remoteRef:
        key: noris-api-key/credential    # item-title/field-name

# WRONG — will fail with "secret reference has invalid format"
spec:
  data:
    - secretKey: api-key
      remoteRef:
        key: noris-api-key
        property: credential
```

Field names depend on the 1Password item category:
- **API Credential**: `credential`
- **Password**: `password`
- **Login**: `username`, `password`
- **Database**: `username`, `password`, `database`, `host`, `port`

### 5.3 1Password Vault Mismatch (CRITICAL PITFALL)

The K8s `ClusterSecretStore` uses a **different 1Password service
account** than the local `op` CLI on the Hermes host. Items created
with the local `op` CLI land in vault **Hermes**
(`5ythsz37hf3xminhq33drg55tm`), but the K8s ESO reads from vault
**Kubernetes ESO** (`334ykdtj5kar3jlpcrztjvx2fu`).

**Symptom**: ExternalSecret shows `SecretSyncedError: no vault matched
the secret reference query` even though the item exists in 1Password.

**Diagnosis**:
```bash
# Check which vault the K8s ClusterSecretStore uses
kubectl get clustersecretstore onepassword-store -o jsonpath='{.spec.provider.onepasswordSDK.vault}'

# Check which vaults your local op CLI can see
op vault list

# Check which vaults the K8s service account can see
K8S_TOKEN=$(kubectl get secret onepassword-token -n external-secrets -o jsonpath='{.data.token}' | base64 -d)
OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op vault list
```

**Fix**: Create 1Password items in the vault that the K8s service
account can see (vault `334ykdtj5kar3jlpcrztjvx2fu` = "Kubernetes ESO").

The K8s service account token CAN create items (not read-only as
previously assumed):
```bash
K8S_TOKEN=$(kubectl get secret onepassword-token -n external-secrets \
  -o jsonpath='{.data.token}' | base64 -d)
OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op item create \
  --category="API Credential" \
  --title="item-name" \
  --vault="334ykdtj5kar3jlpcrztjvx2fu" \
  credential=SECRET_VALUE

OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op item create \
  --category="Password" \
  --title="item-name" \
  --vault="334ykdtj5kar3jlpcrztjvx2fu" \
  password=SECRET_VALUE
```

Alternatively, create items via the 1Password Web UI in the correct
vault, or move items from the Hermes vault to Kubernetes ESO.

### 5.4 Common External Secret Failures

| Error | Cause | Fix |
|-------|-------|-----|
| `rate limit exceeded` | 1Password API throttling | Wait + retry (auto-resolves) |
| `no vault matched the secret reference query` | Item in wrong vault (see §5.3) | Create item in "Kubernetes ESO" vault |
| `secret reference has invalid format` | Wrong key format (see §5.2) | Use `item-title/field-name` format |
| `more than one field matched the secret reference` | Duplicate items in 1Password vault | Deduplicate items |
| `dial tcp: lookup my.1password.com: connect: operation not permitted` | DNS resolution from pod network | Check CoreDNS, Cilium policies |

### 5.3 Checking External Secret Status

```bash
# All ExternalSecrets across cluster
kubectl get externalsecrets -A

# Detailed status for a failing secret
kubectl describe externalsecret -n NAMESPACE SECRET_NAME
```

### 5.5 StorageClasses Available

The cluster has two StorageClasses (both Ceph RBD via CSI):

| StorageClass | Provisioner | Tier | Use Case |
|---|---|---|---|
| `ceph-flash` (default) | rbd.csi.ceph.com | SSD/NVMe | Databases, hot workloads |
| `ceph-hdd-replica` | rbd.csi.ceph.com | HDD | Bulk storage, archives |

**Pitfall**: There is NO `ceph-rbd` StorageClass. Using it in a PVC
causes the pod to stay `Pending` indefinitely with no PV created.
Always use `ceph-flash` or `ceph-hdd-replica`.

---

## Section 6: Nested Proxmox SSH Patterns

### 6.1 Discovering Which PVE Node Hosts a VM

PVE HA migrates VMs between nodes. Before operating on a VM, find
which node it's on:

```bash
for node in proxmox1 proxmox2 proxmox3 proxmox4 proxmox5 proxmox6 proxmox7; do
  ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
    "ssh -o ConnectTimeout=3 root@$node 'qm list 2>/dev/null | grep VMID'" 2>/dev/null
done
```

### 6.2 Executing Commands on VMs via qm guest exec

```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "qm guest exec VMID -- sh -c 'command here'"
```

Output is JSON: `{"exitcode": N, "out-data": "...", "err-data": "..."}`

**Pitfall**: Complex shell commands with quotes break through nested
SSH → qm guest exec. Use base64 encoding for file writes (see
docker-host-administration Section 8.4).

**Pitfall**: `qm guest exec` requires the QEMU guest agent to be
running. Freshly booted VMs may take 30-60s for the agent to start.

### 6.3 Parsing qm guest exec JSON Output

```bash
# Pipe through python3 for clean extraction
ssh ... "qm guest exec 200 -- sh -c '...'" 2>&1 | \
  python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('out-data',''))"
```

---

## Section 7: IaC Repository Structure

### 7.1 dominik/iac-homelab (Gitea)

```
iac-homelab/
├── epic-1-management/         # Mgmt VM (OpenTofu + Ansible)
├── epic-2-k8s/                # 6 VMs + RKE2 bootstrap (Ansible playbook)
├── epic-3-networking-secrets/ # Cilium + External Secrets (OpenTofu)
├── epic-4-storage/            # Ceph CSI (OpenTofu)
├── epic-5-gitops/             # ArgoCD + Root App (OpenTofu)
├── epic-7-mariadb-vm/         # MariaDB Galera VMs (standalone, not K8s)
├── clusters/main/apps/        # ArgoCD App-of-Apps manifests
├── clusters/main/operators/   # Operator Helm releases
├── clusters/main/databases/   # Database cluster manifests
├── clusters/main/backups/     # Velero config
└── .github/workflows/         # CI/CD pipelines (self-hosted runner)
```

### 7.2 Gitea Token Generation

When 1Password `op --reveal` fails (service account limitation),
generate a new Gitea API token directly on the Gitea CT:

```bash
# Gitea runs as user 'gitea', binary at /usr/local/bin/gitea
# Config at /etc/gitea/app.ini, DB at /var/lib/gitea/data/gitea.db
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_NODE \
  "pct exec GITEA_CT -- su -s /bin/sh gitea -c \
   'GITEA_WORK_DIR=/var/lib/gitea /usr/local/bin/gitea \
    admin user generate-access-token \
    -u dominik -t TOKEN_NAME --scopes all --raw \
    --config /etc/gitea/app.ini'"
```

**Pitfall**: Gitea 1.25+ uses `generate-access-token` (hyphenated),
not `generate accesstoken`. Run `gitea admin user --help` first to
verify the subcommand name for your version.

**Pitfall**: Gitea refuses to run as root. Must `su -s /bin/sh gitea`
and set `GITEA_WORK_DIR` environment variable.

### 7.3 Existing Gitea Tokens

Tokens are stored hashed in the Gitea SQLite DB — the full token value
cannot be recovered. Only the last 8 characters are visible:

```sql
SELECT id, uid, name, token_last_eight FROM access_token;
```

If a token is lost, generate a new one (Section 7.2).

---

## Section 8: Health Diagnostics

### 8.1 Full Cluster Health Check

```bash
KUBECONFIG=/root/.kube/config

# Nodes
kubectl get nodes -o wide

# Non-running pods (problems)
kubectl get pods -A | grep -v Running | grep -v Completed

# ArgoCD app sync status
kubectl get applications -n argocd

# External secrets
kubectl get externalsecrets -A

# LoadBalancer services
kubectl get svc -A -o wide | grep LoadBalancer

# Cilium LB pool
kubectl get ciliumloadbalancerippool
```

### 8.2 Common Cold-Boot Issues

| Issue | Resolution |
|-------|-----------|
| Workers NotReady for 2+ min | Check `journalctl -u rke2-agent` on worker — usually waiting for serving-kubelet.crt (503 from CP) |
| ceph-csi CrashLoopBackOff | Ceph cluster may not be reachable from all nodes; check Ceph mon addresses |
| External Secrets SecretSyncedError | 1Password connectivity — check if `onepassword-store` ClusterSecretStore is Ready |
| ArgoCD pods Terminating | Old replicas cleaning up — wait for deployment controller to reconcile |
| kube-controller-manager CrashLoopBackOff | Usually transient during etcd convergence; resolves when quorum is stable |

### 8.3 Checking RKE2 Service Logs

```bash
# On CP nodes
qm guest exec CP_VMID -- journalctl -u rke2-server --no-pager -n 20

# On Worker nodes
qm guest exec WORKER_VMID -- journalctl -u rke2-agent --no-pager -n 20
```

---

## Section 9: RKE2 Version Upgrades

### 9.1 GitOps-First Principle

**CRITICAL**: RKE2 upgrades must be performed via the IaC repo
(`epic-2-k8s/ansible/playbook.yml`) and Ansible — NEVER by manually
downloading tarballs and extracting them on individual nodes. The
Ansible playbook uses the `lablabs.rke2` role which handles proper
installation, rolling restarts, and configuration management.

The version is controlled by a single variable in the playbook:
```yaml
rke2_version: v1.35.2+rke2r1  # Change this to upgrade
```

### 9.2 Prerequisites for Ansible-Based Upgrade

1. **Ansible role installed** on mgmt-runner (VM200):
   ```bash
   cd /root/iac-homelab/epic-2-k8s/ansible
   ansible-galaxy install -r requirements.yml -p /root/.ansible/roles
   ```

2. **Inventory generated** — `inventory.yml` is normally generated by
   OpenTofu (`tofu apply` creates it via `local_file` resource). If
   tofu isn't available, create it manually from known IPs:
   ```yaml
   all:
     vars:
       ansible_user: debian
     children:
       k8s_cluster:
         children:
           masters:
             hosts:
               rke2-cp-01:
                 ansible_host: 10.0.30.51
                 rke2_type: server
                 rke2_server_init: true
               rke2-cp-02:
                 ansible_host: 10.0.30.52
               rke2-cp-03:
                 ansible_host: 10.0.30.53
           workers:
             hosts:
               rke2-worker-01:
                 ansible_host: 10.0.30.61
               rke2-worker-02:
                 ansible_host: 10.0.30.62
               rke2-worker-03:
                 ansible_host: 10.0.30.63
   ```

3. **RKE2 token** — extract from an existing node's config:
   ```bash
   # Find which PVE node hosts a CP VM, then:
   qm guest exec CP_VMID -- sh -c "cat /etc/rancher/rke2/config.yaml"
   # Token line: token: <VALUE>
   # Export before running Ansible:
   export RKE2_TOKEN="<value>"
   ```

4. **SSH access from mgmt-runner to K8s VMs** — Ansible connects as
   `debian` user via SSH. The mgmt-runner's SSH public key must be in
   `/home/debian/.ssh/authorized_keys` on all 6 K8s VMs. If missing,
   `ansible` will fail with `Permission denied (publickey)`.

   **Generate key on VM200** (if none exists):
   ```bash
   ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
     "ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N '' -C 'vm200-ansible'"
   ```

   **Distribute to all 6 K8s VMs** via `qm guest exec` (must discover
   which PVE node hosts each VM first — see Section 6.1):
   ```bash
   PUBKEY=$(ssh root@10.0.30.124 "cat ~/.ssh/id_ed25519.pub")

   # For each VM, on its hosting PVE node:
   ssh root@PVE_NODE "qm guest exec VMID --timeout 15 -- sh -c \
     'mkdir -p /home/debian/.ssh && echo \"$PUBKEY\" >> /home/debian/.ssh/authorized_keys \
      && chown -R debian:debian /home/debian/.ssh \
      && chmod 700 /home/debian/.ssh && chmod 600 /home/debian/.ssh/authorized_keys \
      && echo OK'"
   ```

   **Verify connectivity** before running the playbook:
   ```bash
   ssh root@10.0.30.124 "cd /root/iac-homelab/epic-2-k8s/ansible && ansible all -m ping"
   # All 6 nodes should return SUCCESS => { "ping": "pong" }
   ```

### 9.3 Performing the Upgrade

```bash
# On mgmt-runner (VM200):
cd /root/iac-homelab/epic-2-k8s/ansible

# 1. Edit the version in the playbook
# Change rke2_version: v1.35.2+rke2r1 → v1.35.6+rke2r1

# 2. Commit to Git (GitOps)
git add -A
git commit -m "upgrade-rke2-v1.35.6"
git push origin main

# 3. Run the playbook
export RKE2_TOKEN="<token-from-existing-node>"
ansible-playbook playbook.yml
```

The playbook has TWO plays:
- Play 1: Installs/upgrades RKE2 on all nodes via `lablabs.rke2` role
- Play 2: Rolling restart — Workers first (`serial: 1`), then CP nodes
  (`serial: 1`). Each node waits for RKE2 service active + node Ready
  before proceeding to the next.

### 9.4 Post-Upgrade: Transient CP Restart Failures

After the playbook completes, the PLAY RECAP may show `failed=1` on a CP
node even though the upgrade succeeded. This happens when Ansible's
`systemctl restart rke2-server` races with RKE2's internal reload from
the binary swap — the service had already started before Ansible tried
to restart it, causing a transient "already activating" error.

**Diagnosis**: Check the actual node state, don't trust the PLAY RECAP alone:
```bash
# On the "failed" CP node via qm guest exec:
qm guest exec CP_VMID -- sh -c "systemctl is-active rke2-server && rke2 --version"

# Cluster-wide:
kubectl get nodes -o wide
# If node is Ready + target version → failure was benign, no action needed
```

**Key insight**: Always verify with `kubectl get nodes -o wide` after the
playbook. If all nodes are Ready and on the target version with 0
non-running pods, the upgrade succeeded regardless of Ansible's exit
code on individual nodes.

### 9.5 Blue-Green Alternative (User Preference)

For major version upgrades or when rolling upgrades are too risky,
the user prefers blue-green:
1. Create new VMs with new RKE2 version (via OpenTofu + Ansible)
2. Join them to the existing cluster
3. Verify new nodes are Ready
4. Drain + cordon old nodes (migrate workloads)
5. Remove old nodes from cluster
6. Delete old VMs

### 9.6 CRITICAL PITFALL: RKE2 Tarball Extraction Destroys Debian 12

**NEVER** extract `rke2.linux-amd64.tar.gz` directly to `/` on a
Debian 12 system. The tarball contains:
```
bin/    → extracted to /bin/ (OVERWRITES /bin symlink!)
lib/    → extracted to /lib/ (OVERWRITES /lib symlink!)
share/  → extracted to /share/ (creates bogus directory)
```

Debian 12 uses **usrmerge**: `/bin → usr/bin`, `/lib → usr/lib`,
`/sbin → usr/sbin` are symlinks. The tarball extraction replaces
these symlinks with real directories containing only RKE2 files,
making ALL system binaries (`ls`, `sh`, `bash`, `systemctl`, `mount`)
inaccessible. The VM becomes unmanageable — no new processes can
exec, `qm guest exec` returns exit code 29.

**Correct extraction target**: `/usr/local/` (the tarball's `bin/`
contents go to `/usr/local/bin/`, etc.)

**If already broken**, see Section 10 for recovery procedure.

---

## Section 10: Recovering a Broken VM (Symlink Destruction)

### 10.1 Symptoms

- `qm guest exec VMID -- /bin/echo hello` → exit code 29
- `qm guest exec VMID -- sh -c "..."` → exit code 29
- `qm guest exec VMID -- /usr/bin/ls` → exit code 29
- K8s node still shows `Ready` (kernel + RKE2 processes still running)
- But no NEW processes can start (broken `/bin` and `/lib`)

### 10.2 Recovery: Ceph RBD Offline Mount

When the VM's disk is on Ceph RBD (`vm_disks` storage pool):

```bash
# 1. Stop the VM (preserve etcd quorum if CP node — need 2/3 alive)
qm stop VMID

# 2. On the PVE node hosting the VM, map the RBD image
rbd map vm_disks/vm-VMID-disk-0
# Returns /dev/rbdN (note the number!)

# 3. Wait for udev, then check partitions
sleep 2
lsblk | grep rbd
# Typically: rbdNp1 = root fs, rbdNp14 = BIOS boot, rbdNp15 = EFI

# 4. Mount the root partition
mkdir -p /mnt/vm-rescue
mount /dev/rbdNp1 /mnt/vm-rescue

# 5. Verify the damage
ls -la /mnt/vm-rescue/bin   # Should be symlink but is real dir
ls -la /mnt/vm-rescue/lib   # Should be symlink but is real dir

# 6. Fix symlinks
# Save any RKE2 binaries first:
cp /mnt/vm-rescue/bin/rke2 /mnt/vm-rescue/usr/local/bin/rke2
cp /mnt/vm-rescue/bin/rke2-*.sh /mnt/vm-rescue/usr/local/bin/
# Copy systemd units if present:
mkdir -p /mnt/vm-rescue/usr/lib/systemd/system
cp -a /mnt/vm-rescue/lib/systemd/system/rke2-*.service /mnt/vm-rescue/usr/lib/systemd/system/
cp -a /mnt/vm-rescue/lib/systemd/system/rke2-*.env /mnt/vm-rescue/usr/lib/systemd/system/

# Remove fake directories, restore symlinks:
rm -rf /mnt/vm-rescue/bin
ln -s usr/bin /mnt/vm-rescue/bin
rm -rf /mnt/vm-rescue/lib
ln -s usr/lib /mnt/vm-rescue/lib
rm -rf /mnt/vm-rescue/share  # Bogus dir from tarball

# 7. Verify fix
ls -la /mnt/vm-rescue/bin  # Should show: bin -> usr/bin
ls -la /mnt/vm-rescue/lib  # Should show: lib -> usr/lib
ls /mnt/vm-rescue/bin/bash # Should exist
ls /mnt/vm-rescue/bin/systemctl  # Should exist

# 8. Unmount, unmap, start VM
umount /mnt/vm-rescue
rbd unmap /dev/rbdN
qm start VMID

# 9. Wait 30-60s for boot, verify guest agent
qm guest exec VMID -- sh -c "echo BOOT_OK && systemctl is-active rke2-server"
```

### 10.3 RBD Mapping Pitfalls

- **RBD device disappears**: If `rbd map` succeeds but `/dev/rbdN`
  vanishes before mount, re-run `rbd map` (previous mapping was stale).
  Always `sleep 2` after mapping for udev to settle.
- **Partition devices not created**: `lsblk` shows partitions but
  `/dev/rbdNp1` doesn't exist as a block device. Run `partprobe
  /dev/rbdN` or just re-map (unmap + map again).
- **kpartx not available**: PVE nodes may not have `kpartx`. The
  kernel usually creates partition devices automatically after
  `rbd map` + `sleep 2`. If not, `partprobe /dev/rbdN` helps.
- **Don't mount while VM is running**: RBD is exclusive — stop the VM
  first. Mounting a live RBD image will corrupt the filesystem.

### 10.4 Post-Recovery: Restore Correct RKE2 Version

If the tarball extraction was for a NEW version (e.g. v1.35.6) but
the cached runtime images on the node are for the OLD version (e.g.
v1.35.2), RKE2 will fail to start with:
```
failed to pull images: image "rancher/rke2-runtime:v1.35.6-rke2r1": not found
```

**Fix**: Download the MATCHING tarball version and extract correctly:
```bash
# On the broken VM (after symlink repair + reboot):
cd /tmp
wget https://github.com/rancher/rke2/releases/download/v1.35.2%2Brke2r1/rke2.linux-amd64.tar.gz
tar xf rke2.linux-amd64.tar.gz -C /usr/local/  # CORRECT target!
systemctl restart rke2-server
```

Then perform the version upgrade properly via Ansible (Section 9).

---

## Section 11: OS Updates on K8s VMs

### 11.1 Updating Via qm guest exec

K8s VMs don't have SSH keys from the mgmt-runner, so Ansible can't
reach them directly. Use `qm guest exec` through Proxmox:

```bash
# Discover which PVE node hosts each VM first (Section 6.1)
# Then run apt upgrade:
ssh -i ~/.ssh/id_ed25519_proxmox root@PVE_JUMPHOST \
  "ssh root@PVE_NODE 'qm guest exec VMID --timeout 600 -- sh -c \"
    apt-get update && apt-get upgrade -y && apt-get autoremove -y
  \"'"
```

### 11.2 Interrupted dpkg Recovery

If apt upgrade is interrupted (timeout, connection drop), dpkg is left
in a half-configured state. This blocks ALL subsequent apt operations
on that node — including the Ansible playbook's pre_tasks (which install
`ceph-common` and `rbd-nbd`). The Ansible run will fail with:
```
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a'
```

**Simple fix** (if only dpkg lock is held):
```bash
qm guest exec VMID --timeout 300 -- sh -c "dpkg --configure -a && apt-get update && apt-get upgrade -y"
```

**Full fix** (if dpkg AND debconf locks are held by stale processes):
```bash
# Via qm guest exec on the PVE node hosting the VM:
qm guest exec VMID --timeout 120 -- bash -c "
  fuser -k /var/lib/dpkg/lock-frontend 2>/dev/null;
  fuser -k /var/lib/dpkg/lock 2>/dev/null;
  fuser -k /var/cache/debconf/config.dat 2>/dev/null;
  sleep 2;
  rm -f /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock;
  DEBIAN_FRONTEND=noninteractive dpkg --configure -a &&
  apt-get install -y ceph-common rbd-nbd
"
```

**Pitfall: debconf lock** — Even after clearing dpkg locks, `debconf`
has its own lock at `/var/cache/debconf/config.dat`. If a stale process
holds it, `dpkg --configure -a` fails with:
```
debconf: DbDriver "config": /var/cache/debconf/config.dat is locked by another process
```
Must `fuser -k /var/cache/debconf/config.dat` before retrying.

**Pitfall: Interactive prompts hang under qm guest exec** — Without
`DEBIAN_FRONTEND=noninteractive`, dpkg may prompt for config file
conflicts (e.g. `sshd_config` was locally modified) and hang indefinitely
under `qm guest exec` (no TTY). Always set
`DEBIAN_FRONTEND=noninteractive` to auto-keep-existing config.

**Cross-reference**: This must be fixed BEFORE running the Ansible
upgrade playbook (Section 9.3) — the `lablabs.rke2` role's pre_tasks
install `ceph-common` and `rbd-nbd` via apt, which will fail if dpkg
is in an interrupted state. Fix all affected nodes first, then re-run
`ansible-playbook playbook.yml`.

### 11.3 Direct SSH to mgmt-runner

VM200 (mgmt-runner) at 10.0.30.124 accepts direct SSH from the Hermes
host:
```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124
```
This bypasses the Proxmox jumphost chain and is faster for running
kubectl/helm/git commands.

---

## Section 12: Actual VM→PVE-Node Mapping (July 2026)

The tofu state records where VMs were originally placed, but PVE HA
migrates VMs. The ACTUAL mapping (as of 2026-07-12) is:

| VM | Name | PVE-Node | IP |
|----|------|----------|-----|
| 118 | rke2-cp-01 | proxmox5 | 10.0.30.51 |
| 130 | rke2-cp-02 | proxmox3 | 10.0.30.52 |
| 129 | rke2-cp-03 | proxmox2 | 10.0.30.53 |
| 128 | rke2-worker-01 | proxmox | 10.0.30.61 |
| 132 | rke2-worker-02 | proxmox2 | 10.0.30.62 |
| 131 | rke2-worker-03 | proxmox3 | 10.0.30.63 |
| 200 | mgmt-runner | proxmox3 | 10.0.30.124 |

**Always re-discover** before operating — HA may have moved VMs since
this table was written.

---

## Section 13: Helm Chart Upgrades

### 13.1 Chart Inventory & Version Matrix

Before upgrading, inventory all Helm releases and compare against
latest available versions:

```bash
# On VM200 (mgmt-runner):
export KUBECONFIG=/root/.kube/config

# 1. List all Helm releases across namespaces
helm list -A

# 2. Update repo cache
helm repo update

# 3. Check latest available versions
helm search repo argo/argo-cd | head -3
helm search repo external-secrets/external-secrets | head -3
# etc.

# 4. RKE2-bundled charts (already updated with RKE2 upgrade):
helm list -n kube-system | grep rke2-
# These are: cilium, traefik, coredns, metrics-server, snapshot-controller
```

Build a matrix: Component | Current Version | Latest Available |
Update Type (Major/Minor/Patch) | Management Method (Helm-direct vs ArgoCD GitOps).

**RKE2-bundled charts** (Cilium, Traefik, CoreDNS, Metrics Server,
Snapshot Controller) are automatically updated when RKE2 is upgraded —
no separate Helm upgrade needed.

### 13.2 ArgoCD GitOps-Managed Charts (CNPG, Velero)

Charts deployed via ArgoCD Applications are upgraded by changing the
`targetRevision` in the Application manifest in the IaC repo:

```bash
cd /root/iac-homelab
# Find the Application manifest
grep -r "targetRevision" clusters/main/

# Update the chart version (e.g. CNPG 0.22.1 → 0.29.0)
sed -i 's/targetRevision: 0.22.1/targetRevision: 0.29.0/' \
  clusters/main/operators/cloudnativepg.yaml
git add -A && git commit -m "feat: upgrade CNPG chart" && git push
```

**Pitfall: ArgoCD doesn't auto-refresh after chart version bump** —
Even with `selfHeal: true`, ArgoCD may keep the old chart revision
cached. Force a hard refresh:
```bash
kubectl annotate application <app-name> -n argocd \
  argocd.argoproj.io/refresh=hard --overwrite
```

Wait for the new revision to appear in `kubectl get applications -n argocd`,
then monitor the rollout:
```bash
kubectl rollout status deployment/<deployment-name> -n <namespace> --timeout=300s
```

### 13.3 Helm-Direct Charts (ArgoCD, External Secrets, Ceph CSI)

Charts installed directly via `helm install` are upgraded with
`helm upgrade`:

```bash
# Save current values before upgrading
helm get values <release-name> -n <namespace> > /tmp/<release>-values.yaml

# Dry-run first
helm upgrade <release-name> <repo>/<chart> \
  --version <new-version> \
  -n <namespace> \
  --reuse-values \
  --dry-run

# Actual upgrade
helm upgrade <release-name> <repo>/<chart> \
  --version <new-version> \
  -n <namespace> \
  --reuse-values \
  --timeout 300s
```

**Pitfall: `--reuse-values` may not suffice for major chart versions** —
Major chart versions can introduce required fields that don't exist in
old values. If `helm upgrade` fails with a template error (e.g. `nil
pointer evaluating interface {}`), create a complete values file that
includes the new required fields:
```bash
# Instead of --reuse-values, provide explicit values:
helm upgrade <release-name> <repo>/<chart> \
  --version <new-version> \
  -n <namespace> \
  -f /tmp/<release>-v3-values.yaml \
  --timeout 300s
```

### 13.4 ArgoCD v2→v3 Major Upgrade (Breaking Change)

Upgrading the ArgoCD Helm chart from v7.x (app v2.x) to v10.x (app
v3.x) requires adding `redis.networkPolicy.create: false` to the
values file. Without this, the chart fails with:
```
template: argo-cd/templates/redis/networkpolicy.yaml:2:22:
nil pointer evaluating interface {}.create
```

**Safe upgrade path**: First upgrade to the latest v2.x patch
(chart 7.9.1 = app v2.14.11), then jump to v3.x:
```bash
# Step 1: Safe patch to latest v2.x
helm upgrade argocd argo/argo-cd --version 7.9.1 -n argocd --reuse-values

# Step 2: Create v3-compatible values file with redis.networkPolicy
cat > /tmp/argocd-v3-values.yaml << 'EOF'
# ... copy all existing values, then add:
redis:
  networkPolicy:
    create: false
EOF

# Step 3: Major upgrade to v3
helm upgrade argocd argo/argo-cd --version 10.1.3 -n argocd \
  -f /tmp/argocd-v3-values.yaml --timeout 300s
```

**Save values files to the IaC repo** (`epic-2-k8s/helm/<chart>/values.yaml`)
for reproducible installs.

### 13.5 CNPG Replica Rebuild from WAL Gap

When a CNPG PostgreSQL replica falls behind and the primary has
already recycled the needed WAL segments, the replica is permanently
broken:
```
requested WAL segment 000000020000000000000008 has already been removed
```

**Fix**: Delete the broken replica's PVCs and pod — CNPG automatically
rebuilds from a fresh base backup of the primary:
```bash
# 1. Identify the broken replica pod
kubectl get pods -n postgres | grep -v Running
# e.g. postgres-main-2 is in a bad state

# 2. Delete the PVCs (data + wal)
kubectl delete pvc postgres-main-2 postgres-main-2-wal -n postgres

# 3. Delete the pod so CNPG recreates it
kubectl delete pod postgres-main-2 -n postgres

# 4. CNPG creates a new replica (e.g. postgres-main-4) automatically
# Wait for it to join:
kubectl get pods -n postgres -w
# Watch for 3/3 Running
```

**Do this BEFORE upgrading the CNPG operator** — upgrading with a
broken replica is risky. The operator upgrade itself may trigger a
primary restart, and having only 1 healthy replica reduces safety margin.

### 13.6 Ceph CSI RBD Upgrade — Transient API Server Disruption

**Pitfall**: Upgrading the Ceph CSI RBD chart triggers a DaemonSet
rolling update of `ceph-csi-rbd-nodeplugin` pods across ALL nodes,
including control plane nodes. The nodeplugin pods mount/unmount CSI
driver sockets, which can briefly disrupt the kubelet on CP nodes and
cause all 3 API servers to become unreachable for 2-5 minutes.

**Symptoms during the disruption**:
```
The connection to the server 10.0.30.51:6443 was refused
```

**Mitigation**:
- Schedule Ceph CSI upgrades during maintenance windows
- After `helm upgrade`, monitor with `curl -sk https://CP_IP:6443/healthz`
  until it responds (even 401 Unauthorized means the server is up)
- Once API server recovers, verify with `kubectl get pods -n kube-system | grep ceph-csi-rbd`
- New nodeplugin pods may stay in `ContainerCreating` for several
  minutes while pulling the new CSI image — this is normal
- All PVCs remain bound throughout; no data loss occurs

### 13.7 ExternalSecret Perpetual OutOfSync in ArgoCD

**Pitfall**: The External Secrets Operator adds default fields to
ExternalSecret specs that aren't present in the Git YAML:
- `remoteRef.conversionStrategy: "Default"`
- `remoteRef.decodingStrategy: "None"`
- `remoteRef.metadataPolicy: "None"`
- `target.deletionPolicy: "Retain"`
- `target.template.engineVersion: "v2"`
- `target.template.mergePolicy: "Replace"`

ArgoCD sees these as spec drift and marks the ExternalSecret
perpetually `OutOfSync`. The `ignoreDifferences` rule for
`/status` doesn't cover `/spec` differences.

**Fix**: Either add the ESO default fields to the Git YAML (so spec
matches exactly), or add `ignoreDifferences` for the specific JSON
pointers. Adding fields to Git YAML is cleaner:
```yaml
spec:
  data:
    - secretKey: access_key_id
      remoteRef:
        key: velero-s3-backup/S3 Credentials/access_key_id
        conversionStrategy: Default
        decodingStrategy: None
        metadataPolicy: None
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    template:
      engineVersion: v2
      mergePolicy: Replace
```

### 13.8 Accidental .kube/cache Commit

**Pitfall**: Running `kubectl` commands from the IaC repo root creates
`.kube/cache/` with discovery API responses. If `git add -A` is used,
these cache files get committed. Always add `.kube/` to `.gitignore`:
```bash
echo ".kube/" >> .gitignore
```

---

## References

See `references/` directory for session-specific details.

- `references/cold-boot-recovery-2026-07.md` — Session detail: full cluster cold-boot, Cilium LB pool migration from .70-.89 to .200-.250, ArgoCD convergence timeline
- `references/rke2-cluster-topology.md` — Complete topology, IP map, operator inventory, IaC repo structure
- `references/hindsight-k8s-deployment-2026-07.md` — Deploying hindsight-api in K8s: ExternalSecret format, StorageClass, env var ordering, vault mismatch, ArgoCD cache pitfalls
- `references/rke2-upgrade-and-recovery-2026-07.md` — Session detail: RKE2 tarball destruction, Ceph RBD offline recovery, Ansible-based upgrade prerequisites, OS update procedures, openclaw-memory removal via GitOps, Ansible SSH key prerequisite
- `references/helm-chart-upgrades-2026-07.md` — Session detail: full Helm chart upgrade cycle (ArgoCD v2→v3, External Secrets, Ceph CSI, CNPG, Velero), CNPG replica WAL-gap rebuild, Ceph CSI transient API server disruption, ExternalSecret OutOfSync root cause