# RKE2 Cluster Cold-Boot Recovery — 2026-07-12

## Situation

RKE2 cluster (6 VMs: 3 CP + 3 Worker) was offline for 111 days. All VMs
were stopped across the PVE cluster. User requested cluster startup
and K8s evaluation for Docker workload migration.

## Startup Sequence

### 1. VM Discovery

VMs had been HA-migrated to various PVE nodes. Had to scan all nodes
to find where each VM landed:

```
118 (CP-01)    → proxmox5
130 (CP-02)    → proxmox3
129 (CP-03)    → proxmox2
128 (Worker-01) → proxmox5
132 (Worker-02) → proxmox2
131 (Worker-03) → proxmox3
200 (mgmt-runner) → proxmox3
```

### 2. Start Order

1. Started all 6 RKE2 VMs via `qm start` (HA placed them on various nodes)
2. CP nodes began etcd convergence within 30s
3. Workers initially showed `NotReady` — waiting for
   `serving-kubelet.crt` retrieval (503 from API server)
4. Worker-03 became Ready after ~2 min
5. Worker-02 became Ready after ~3 min (slower due to proxy reconnection)
6. All 6 nodes Ready within 5 min

### 3. Convergence Timeline

| T+0 | VMs started |
| T+30s | CPs booting, etcd forming quorum |
| T+1min | Workers connecting via wss://CP:9345/v1-rke2/connect |
| T+2min | First workers Ready (cert retrieval successful) |
| T+3min | All nodes Ready, Cilium agents starting |
| T+5min | ArgoCD, External Secrets, operators beginning reconciliation |
| T+10min | Most pods Running, old replicas Terminating |
| T+15min | Databases recovering (Galera, PostgreSQL) |

### 4. Pod States After Cold Boot

Most pods showed mixed states initially:
- `Running` — new replicas scheduled by deployment controller
- `Terminating` — old replicas from before shutdown (111 days old)
- `Unknown` — node was down when status was last reported
- `CrashLoopBackOff` — ceph-csi nodeplugin (2 of 6) — Ceph connectivity
- `CreateContainerConfigError` — memory-api, qdrant — missing secrets
  (ExternalSecret hadn't synced yet)

All resolved naturally except:
- 3 ExternalSecrets with SecretSyncedError (1Password item issues)
- 2 ceph-csi-rbd-nodeplugin CrashLoopBackOff

## Cilium LB Pool Migration

### Problem

Cilium LB pool was set to `10.0.30.70-10.0.30.89`, overlapping with
Galera/MaxScale VIPs:
- 10.0.30.70 = Keepalived VIP for MaxScale (Galera)
- 10.0.30.71-73 = Galera VM IPs

K8s services had claimed:
- Traefik → 10.0.30.70 (conflicted with MaxScale VIP!)
- PostgreSQL RW → 10.0.30.71 (conflicted with Galera VM!)
- MariaDB MaxScale (K8s) → 10.0.30.72

### Fix Applied

1. Deleted live `CiliumLoadBalancerIPPool` CRD
2. Created new pool with `10.0.30.200-10.0.30.250`
3. Services auto-reassigned:
   - Traefik → 10.0.30.202
   - MariaDB MaxScale (K8s) → 10.0.30.201
   - PostgreSQL RW → 10.0.30.200
4. Updated manifest on all 3 CP nodes:
   `/var/lib/rancher/rke2/server/manifests/cilium-l2-lb.yaml`
5. Fixed IaC repo: `epic-2-k8s/ansible/playbook.yml` — committed + pushed

### Verification

- `10.0.30.70` responds to ping (Keepalived VIP for MaxScale — correct!)
- No K8s LoadBalancer service holds IPs in .70-.89 range
- Cilium LB pool shows 48 IPs available in .200-.250 range

## K8s MariaDB Removal

### Decision

User clarified: Galera lives as **native VMs** (VM300-302 + MaxScale
VM310), not in K8s. The K8s MariaDB Galera (3 pods) + MaxScale (2 pods)
in namespace `mariadb` was redundant and should be removed.

### Removal Steps

1. Removed YAML files from `iac-homelab` repo:
   - `clusters/main/databases/mariadb/` (5 files: cluster.yaml,
     external-secrets.yaml, maxscale.yaml, namespace.yaml, README.md)
   - `clusters/main/operators/mariadb-operator.yaml`
   - `clusters/main/operators/mariadb-operator-crds.yaml`
2. Committed + pushed to Gitea
3. ArgoCD detected change (`databases` → `OutOfSync`) but did NOT
   auto-prune the operator-managed CRs
4. Manual cleanup required:
   - `kubectl delete mariadb mariadb-galera -n mariadb` (CR deletion
     triggers operator finalizer processing)
   - `kubectl delete maxscale mariadb-maxscale -n mariadb` (already
     gone — operator had processed it)
   - `kubectl delete namespace mariadb` (clean)
   - `kubectl delete namespace mariadb-operator` (stuck Terminating
     briefly, cleared after ~15s)
   - `kubectl delete crd ...` (9 mariadb CRDs — ArgoCD didn't prune
     these, had to delete manually)
5. Forced ArgoCD hard-refresh: `kubectl annotate application root -n
   argocd argocd.argoproj.io/refresh-options=hard-refresh --overwrite`

### Result

- Namespaces `mariadb` + `mariadb-operator` deleted
- 9 MariaDB CRDs deleted
- ArgoCD apps `mariadb-operator` + `mariadb-operator-crds` pruned
- `databases` app re-synced (only postgres remains)
- Native Galera VMs (300-302) + MaxScale VM310 untouched
- LB IP 10.0.30.201 freed (was K8s MaxScale, now available in pool)

### Key Lesson

ArgoCD `prune: true` handles standard K8s resources but **does not
reliably prune operator CRs and CRDs** — finalizers require the
operator to be alive, and CRD deletion isn't tracked by ArgoCD's
resource tree. Manual `kubectl delete` of CRs → namespace → CRDs is
required when removing an operator-managed application.

ArgoCD was already installed and configured:
- 8 Applications synced (operators, databases, backups, memory, root)
- Ingress via Traefik (host: `argocd`, routed through Traefik LB IP)
- App-of-Apps pattern pointing to `dominik/iac-homelab` on Gitea
- Auto-sync with `prune: true, selfHeal: true`

Admin password retrieved via:
```bash
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d
```

## IaC Repository

`dominik/iac-homelab` on Gitea (10.0.30.105:3000) contains:
- 7 Epics (management VM, K8s cluster, networking, storage, GitOps, memory, MariaDB)
- OpenTofu for VM provisioning + Helm releases
- Ansible for RKE2 bootstrap
- GitHub Actions CI/CD (self-hosted runner on mgmt-runner VM)
- ArgoCD Application manifests in `clusters/main/apps/`

Separate repo: `dominik/observability-iac` — Loki, Prometheus/Thanos, Grafana
