hermes-skills/devops/proxmox-ve-administration/references/rke2-cluster-cold-boot-2026-07.md

# RKE2 Kubernetes Cluster — Discovery & Cold-Boot Recovery

## TL;DR

An existing RKE2 cluster (3 CP + 3 Workers) was found stopped on Proxmox.
Booting it requires only `qm start` on each VM. The cluster was fully
bootstrapped 111 days prior with ArgoCD, Cilium CNI, Ceph CSI, External
Secrets, CloudNativePG, MariaDB Operator, and Velero.

## Cluster Topology

| VMID | Name | Role | Node (dynamic) | IP | Specs |
|------|------|------|-----------------|-----|-------|
| 118 | rke2-cp-01 | Control Plane | proxmox5 | 10.0.30.51 | 4c / 12GB / 40GB |
| 130 | rke2-cp-02 | Control Plane | proxmox3 | 10.0.30.52 | 4c / 12GB / 40GB |
| 129 | rke2-cp-03 | Control Plane | proxmox2 | 10.0.30.53 | 4c / 12GB / 40GB |
| 128 | rke2-worker-01 | Worker | proxmox5 | 10.0.30.61 | 4c / 12GB / 80GB |
| 132 | rke2-worker-02 | Worker | proxmox2 | 10.0.30.62 | 4c / 12GB / 80GB |
| 131 | rke2-worker-03 | Worker | proxmox3 | 10.0.30.63 | 4c / 12GB / 80GB |

VMs are HA-managed — PVE may place them on different nodes than where
they were originally created. Always scan all nodes for VMs:
```bash
for n in proxmox1 proxmox2 proxmox3 proxmox4 proxmox5 proxmox6 proxmox7; do
  ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
    "ssh -o ConnectTimeout=3 root@$n 'qm list 2>/dev/null | grep rke2'"
done
```

## Management VM

VM 200 (mgmt-runner-01) on proxmox3 has `kubectl` and `helm` installed.
Kubeconfig at `/root/.kube/config` (must use `KUBECONFIG=/root/.kube/config`
explicitly — kubectl doesn't pick it up from default location via qm guest exec).

## Cold-Boot Procedure

### 1. Start CPs First, Then Workers

```bash
# Start all CPs
for vmid in 118 130 129; do
  node=$(find_node_for_vmid $vmid)
  ssh root@$node "qm start $vmid"
done
sleep 15
# Start all workers
for vmid in 128 132 131; do
  node=$(find_node_for_vmid $vmid)
  ssh root@$node "qm start $vmid"
done
```

### 2. Wait for Convergence (5-10 min)

Workers initially show `NotReady` — the rke2-agent needs to retrieve
serving-kubelet certs from the API server (returns 503 until CPs converge):

```
Waiting to retrieve agent configuration;
server is not ready: serving-kubelet.crt: 503 Service Unavailable
```

This is NORMAL during cold boot. Wait 5-10 minutes. Workers transition
to `Ready` once cert exchange completes.

### 3. Verify

```bash
# Via mgmt-runner
ssh root@10.0.20.10 "ssh root@proxmox3 'qm guest exec 200 -- sh -c \
  \"KUBECONFIG=/root/.kube/config kubectl get nodes\"'"
```

All 6 nodes should show `Ready`.

## What's Already Installed

| Component | Namespace | Status |
|-----------|-----------|--------|
| ArgoCD | argocd | Running (App-of-Apps) |
| Cilium CNI | kube-system | Running |
| Ceph CSI (RBD) | kube-system | 2 CrashLoopBackOff on cold boot |
| Traefik Ingress | kube-system | Running (IngressClass) |
| CoreDNS | kube-system | Running |
| Metrics Server | kube-system | Running |
| External Secrets | external-secrets | Running |
| CloudNativePG | cnpg-system | Running (PostgreSQL operator) |
| MariaDB Operator | mariadb-operator | Running |
| MariaDB Galera | mariadb | Starting (needs time) |
| PostgreSQL | postgres | 3-replica cluster |
| Velero | velero | Partially running |
| Memory (Qdrant+Ollama) | openclaw-memory | ContainerCreating |

## ArgoCD GitOps

- **Admin password**: `kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d`
- **Ingress**: `argocd` host via Traefik (https://10.0.30.70 with Host: argocd)
- **Git repo**: `http://10.0.30.105:3000/dominik/iac-homelab.git`
- **Root app path**: `clusters/main/apps` (App-of-Apps pattern)
- **Auto-sync**: `prune: true, selfHeal: true`
- **Applications**: root, operators, databases, mariadb-operator, mariadb-operator-crds, cloudnativepg-operator, velero-operator, backups, memory

## Known Cold-Boot Issues

### Ceph CSI CrashLoopBackOff

Some `ceph-csi-rbd-nodeplugin` pods enter CrashLoopBackOff after cold boot.
They typically recover after a few restart cycles. If they don't:
```bash
kubectl -n kube-system delete pod -l app=csi-rbd-nodeplugin
# Or restart the provisioner
kubectl -n kube-system rollout restart deployment ceph-csi-rbd-provisioner
```

### kube-controller-manager CrashLoopBackOff on CP-01

`kube-controller-manager-rke2-cp-01` may CrashLoop briefly during leader
election. It resolves once a stable leader is elected among the 3 CPs.

### Old Pods Stuck in Terminating

After 111 days offline, many pods show `Terminating` or `Unknown`.
These are ghosts from the previous run. They clear automatically as the
new replicas become ready. Force-delete if stubborn:
```bash
kubectl -n <namespace> delete pod <pod-name> --force --grace-period=0
```

## Kubeconfig Transfer

To set up kubectl on a new management machine:

```bash
# Get kubeconfig from CP-01 (via qm guest exec)
RAW=$(ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
  "ssh root@proxmox5 'qm guest exec 118 -- cat /etc/rancher/rke2/rke2.yaml'" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['out-data'])")

# Replace 127.0.0.1 with CP-01 IP
echo "$RAW" | sed 's/127.0.0.1/10.0.30.51/g' > ~/.kube/config
chmod 600 ~/.kube/config
```

## Migration Candidates (PVE → K8s)

Services suitable for K8s migration (ordered by ease):

| Phase | Services | Current Location |
|-------|----------|-----------------|
| 1 | (Already done) ArgoCD, Operators, DBs | RKE2 cluster |
| 2 | Grafana, Prometheus, Alertmanager, Blackbox | CT 141 (Docker) |
| 3 | Traefik (→ IngressController), Authelia | CT 99999, CT 112 |
| 4 | Gitea, Paperless-ngx, Paperless-GPT, LibreChat | CT 108, 104, 113, 133 |
| 5 | Seafile, Immich | CT 111 (Docker, 11 containers) |
| 6 | Ollama, LiteLLM, Ideogram4 | CT 123, 124, CT 111 |

Services that STAY on PVE (not K8s candidates):
- Home Assistant (USB/Zigbee, HA OS)
- Frigate (GPU passthrough — though could use K8s GPU operator)
- MariaDB Galera VMs (already clustered, K8s adds complexity)
- Hermes Agent (terminal access, persistent state)
- PBS (PVE-integrated)
- Samba shares (kernel-level file serving)
- Dovecot IMAP