# RKE2 Upgrade Attempt & Tarball Recovery — 2026-07-12

## Goal

Update K8s cluster: OS updates on all VMs, RKE2 upgrade v1.35.2→v1.35.6,
Helm chart updates, fix broken components. Scope: K8s only (no PVE hosts).

## Phase 1: OS Updates (COMPLETED)

All 7 VMs updated via `qm guest exec` through Proxmox jumphost chain.

### VM→PVE-Node Discovery

Initial assumption that all VMs were on proxmox3 was WRONG. Had to scan
all PVE nodes to find actual placement:

```
118 (CP-01)    → proxmox5    (NOT proxmox3)
130 (CP-02)    → proxmox3
129 (CP-03)    → proxmox2
128 (Worker-01) → proxmox5   (later moved to proxmox)
132 (Worker-02) → proxmox2
131 (Worker-03) → proxmox3
200 (mgmt-runner) → proxmox3
```

### Update Process

- VM200: Direct SSH (10.0.30.124) — 0 updates remaining
- VM118, VM129: Updated via qm guest exec on respective PVE nodes
- VM132: dpkg interrupted (timeout) — fixed with `dpkg --configure -a`
- All VMs: 0 remaining updates after completion
- Cluster stayed healthy throughout (6/6 nodes Ready)

### Command Pattern

```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
  "ssh -o ConnectTimeout=5 root@PVE_NODE 'qm guest exec VMID --timeout 600 \
   -- sh -c \"apt-get update && apt-get upgrade -y\"'"
```

## Phase 2: RKE2 Upgrade (PARTIAL — TARBALL DISASTER)

### Initial Approach (WRONG)

Attempted manual tarball installation:
1. Downloaded `rke2.linux-amd64.tar.gz` (v1.35.6) to VM118
2. Extracted with `tar xf rke2.linux-amd64.tar.gz -C /` ← **FATAL ERROR**

### Tarball Destruction

The RKE2 tarball contains:
```
bin/      → RKE2 binaries
lib/      → systemd unit files
share/    → RKE2 data files
```

Extracting to `/` on Debian 12 (which uses usrmerge) replaced:
- `/bin` symlink (→ usr/bin) with a real directory containing only Rke2 binaries
- `/lib` symlink (→ usr/lib) with a real directory containing only systemd files
- Created bogus `/share` directory

Result: ALL system binaries inaccessible (`ls`, `sh`, `bash`, `systemctl`,
`mount`). `qm guest exec` returns exit code 29. VM was still running
(kernel + RKE2 processes in memory) but no new processes could start.

### Recovery: Ceph RBD Offline Mount

1. **Stopped VM118** (`qm stop 118`) — safe because 2/3 CP nodes still
   provide etcd quorum
2. **Mapped RBD image** on proxmox5:
   ```bash
   rbd map vm_disks/vm-118-disk-0  # → /dev/rbd1
   sleep 2  # Wait for udev
   ```
3. **Found partitions**: `rbd1p1` (root fs 39.9G), `rbd1p14` (BIOS boot),
   `rbd1p15` (EFI 124M)
4. **Mounted root partition**: `mount /dev/rbd1p1 /mnt/vm118-rescue`
5. **Verified damage**:
   - `/mnt/vm118-rescue/bin` = real directory with only `rke2`, `rke2-killall.sh`, `rke2-uninstall.sh`
   - `/mnt/vm118-rescue/lib` = real directory with only `systemd/`
   - `/mnt/vm118-rescue/usr/bin/bash` = INTACT (original system binaries fine)
6. **Fixed symlinks**:
   ```bash
   # Save RKE2 binaries
   cp /mnt/vm118-rescue/bin/rke2 /mnt/vm118-rescue/usr/local/bin/
   cp /mnt/vm118-rescue/bin/rke2-*.sh /mnt/vm118-rescue/usr/local/bin/
   
   # Save systemd units
   mkdir -p /mnt/vm118-rescue/usr/lib/systemd/system
   cp -a /mnt/vm118-rescue/lib/systemd/system/rke2-*.service /mnt/vm118-rescue/usr/lib/systemd/system/
   cp -a /mnt/vm118-rescue/lib/systemd/system/rke2-*.env /mnt/vm118-rescue/usr/lib/systemd/system/
   
   # Restore symlinks
   rm -rf /mnt/vm118-rescue/bin
   ln -s usr/bin /mnt/vm118-rescue/bin
   rm -rf /mnt/vm118-rescue/lib
   ln -s usr/lib /mnt/vm118-rescue/lib
   rm -rf /mnt/vm118-rescue/share
   ```
7. **Unmounted, unmapped, started VM**:
   ```bash
   umount /mnt/vm118-rescue
   rbd unmap /dev/rbd1
   qm start 118
   ```
8. **Verified**: Guest agent responded after 60s boot. Shell works.
   BUT RKE2 server stuck "activating" — v1.35.6 binary couldn't find
   v1.35.6 runtime images (cached images were v1.35.2).

### Second Fix: Restore Matching Binary

Downloaded v1.35.2 tarball and extracted CORRECTLY:
```bash
wget https://github.com/rancher/rke2/releases/download/v1.35.2%2Brke2r1/rke2.linux-amd64.tar.gz
tar xf rke2.linux-amd64.tar.gz -C /usr/local/  # CORRECT!
rke2 --version  # v1.35.2+rke2r1
systemctl start rke2-server
```

Node came back as Ready v1.35.2+rke2r1. Cluster fully healthy again.

### RBD Pitfalls Encountered

- **Device disappearance**: First `rbd map` created `/dev/rbd1` but it
  vanished before mount. Had to unmap stale mappings and re-map.
- **No kpartx**: PVE node didn't have `kpartx`. Partition devices were
  created by kernel automatically after `sleep 2` post-mapping.
- **partprobe failed**: `partprobe /dev/rbd1` gave "Could not stat device"
  because the device had already disappeared. Re-mapping fixed it.

## Phase 2b: Proper Upgrade Path (PREREQS IDENTIFIED, BLOCKED ON SSH)

### IaC Repo Analysis

The `epic-2-k8s/ansible/playbook.yml` uses `lablabs.rke2` role v1.50.1:
```yaml
rke2_version: v1.35.2+rke2r1  # ← Change this to upgrade
rke2_token: "{{ lookup('env', 'RKE2_TOKEN') }}"
rke2_ha_mode: true
rke2_api_ip: 10.0.30.50
```

Play 2 does rolling restart: Workers (serial:1) → Masters (serial:1).

### Prerequisites Status

1. ✅ Ansible role installed on VM200 (`ansible-galaxy install -r requirements.yml -p /root/.ansible/roles`)
2. ✅ Inventory generated manually (tofu state shows VM IDs, IPs known)
3. ✅ RKE2 token extracted from VM130 config (`/etc/rancher/rke2/config.yaml` → `token:` line)
4. ❌ **SSH access from VM200 to K8s VMs** — `Permission denied (publickey)`
   - mgmt-runner's SSH key NOT in `debian@VMs/.ssh/authorized_keys`
   - Ansible inventory uses `ansible_user: debian`
   - Need to deploy VM200's public key to all 6 K8s VMs before Ansible can run

### Tofu State Details

Tofu state at `/opt/tofu-state/rke2-cluster.tfstate` has VM IDs:
```
rke2-cp-01: vm_id=118, node=proxmox (original placement, HA may move)
rke2-cp-02: vm_id=130, node=proxmox2
rke2-cp-03: vm_id=129, node=proxmox3
rke2-worker-01: vm_id=128, node=proxmox
rke2-worker-02: vm_id=132, node=proxmox2
rke2-worker-03: vm_id=131, node=proxmox3
```

### User Directive

User instructed: "Nutze GitOps!" and "Neue nodes mit neuer version
erstellen. Cluster join. Verify. Workloads migrieren. Alte nodes löschen"

This means: prefer GitOps/Ansible for upgrades, or blue-green with new
VMs if rolling upgrade is too risky. NEVER manual tarball installation.

## Phase 3: openclaw-memory Removal (COMPLETED via GitOps)

### Context

User: "Openclaw memory kann weg. Nutze ich nicht mehr"

The `openclaw-memory` namespace had 3 broken pods:
- `memory-api` — CreateContainerConfigError (missing secret)
- `qdrant-0` — CreateContainerConfigError (missing secret)
- `ollama` — Running but unused

Two ExternalSecrets in `SecretSyncedError` state (wrong 1Password vault).

### Removal Steps (Clean GitOps)

1. **Deleted from Git repo**:
   ```bash
   rm clusters/main/apps/memory.yaml          # ArgoCD Application
   rm -rf clusters/main/memory/                # All manifests
   rm .github/workflows/epic6_prepare-memory-secrets.yml
   git commit -m "chore: remove openclaw-memory (unused, deprecated)"
   git push origin main                        # commit ebc4da5
   ```

2. **Deleted ArgoCD Application** (cascade=foreground):
   ```bash
   kubectl --server=https://10.0.30.52:6443 \
     delete application memory -n argocd --cascade=foreground
   ```

3. **Deleted namespace** (pods cleared, namespace went Terminating → Gone):
   ```bash
   kubectl --server=https://10.0.30.52:6443 delete ns openclaw-memory
   ```

4. **Verified**: Namespace NotFound, 0 non-running pods, ArgoCD apps
   all Healthy except `backups` (OutOfSync, non-critical).

### Key Observations

- ArgoCD Application `--cascade=foreground` cleaned up managed resources
  (deployments, services) but namespace remained — needed manual `kubectl delete ns`.
- No operator CRs involved, so no finalizer issues (unlike MariaDB removal).
- `backups` ArgoCD app remains OutOfSync (pre-existing, not related).

## Phase 2c: SSH Key Deployment & Ansible Launch (COMPLETED)

### Problem

Ansible on VM200 could not SSH to K8s VMs — `Permission denied (publickey)`.
VM200 had no SSH keypair at all (`~/.ssh/` only had `authorized_keys` and
`known_hosts`).

### Fix

1. **Generated SSH key on VM200**:
   ```bash
   ssh root@10.0.30.124 "ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N '' -C 'vm200-ansible'"
   ```

2. **Confirmed actual VM→PVE-Node mapping** (all 6 nodes scanned):
   ```
   118 (CP-01)      → proxmox5
   130 (CP-02)      → proxmox3
   129 (CP-03)      → proxmox2
   128 (Worker-01)  → proxmox5  (later: proxmox)
   131 (Worker-03)  → proxmox3
   132 (Worker-02)  → proxmox2
   ```

3. **Distributed public key to all 6 VMs** via `qm guest exec` — each VM
   got the key appended to `/home/debian/.ssh/authorized_keys` with
   correct ownership (debian:debian) and permissions (700/600).
   All 6 returned OK.

4. **Verified Ansible connectivity**:
   ```bash
   cd /root/iac-homelab/epic-2-k8s/ansible && ansible all -m ping
   ```
   All 6 nodes: SUCCESS => { "ping": "pong" }

### Ansible Playbook Execution

With all prerequisites met, launched the upgrade:

```bash
cd /root/iac-homelab/epic-2-k8s/ansible
export RKE2_TOKEN='EyMCxVDobAkWjgyPDcwvXRJQHgWHoH6qgDcvFhQUkd'
ansible-playbook playbook.yml -v
```

The playbook:
- Play 1: `lablabs.rke2` role applies `rke2_version: v1.35.6+rke2r1` to
  all nodes (writes config, downloads/installs new binaries)
- Play 2: Rolling restart Workers (serial: 1) — each node restarted,
  waits for RKE2 agent active + node Ready
- Play 3: Rolling restart Masters (serial: 1) — same pattern for CP

GitOps trail:
- Commit `6cb1b3c`: `feat(rke2): upgrade v1.35.2 → v1.35.6+rke2r1`
- Commit `ebc4da5`: `chore: remove openclaw-memory (unused, deprecated)`

## Phase 2d: dpkg Lock Blocking Ansible (RESOLVED)

### Problem

First Ansible playbook run failed on 5 of 6 nodes. Only Worker-01
succeeded (it had been updated cleanly earlier). The other 5 failed
at the pre_tasks stage with:

```
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a'
```

Root cause: Previous OS updates via `qm guest exec` left dpkg in a
half-configured state on 5 nodes. The `openssh-server` package was
mid-upgrade (held a config file conflict prompt that hung without a TTY).

### Fix Sequence

Had to fix each node individually via `qm guest exec` (could not use
Ansible ad-hoc commands because Ansible itself triggers apt):

```bash
# For each broken node, on its hosting PVE node:
ssh root@PVE_NODE "qm guest exec VMID --timeout 120 -- bash -c \"
  fuser -k /var/lib/dpkg/lock-frontend 2>/dev/null;
  fuser -k /var/lib/dpkg/lock 2>/dev/null;
  fuser -k /var/cache/debconf/config.dat 2>/dev/null;
  sleep 2;
  rm -f /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock;
  DEBIAN_FRONTEND=noninteractive dpkg --configure -a &&
  apt-get install -y ceph-common rbd-nbd &&
  echo FIXED
\""
```

All 5 nodes fixed (CP-01, CP-02, CP-03, Worker-02, Worker-03). Then
re-ran `ansible-playbook playbook.yml` successfully.

### Key Insight

The `lablabs.rke2` role's pre_tasks install `ceph-common` and `rbd-nbd`
via apt. If any node has dpkg in an interrupted state, the ENTIRE
playbook fails on that node. Must fix dpkg on ALL nodes before running
the upgrade playbook. Use `DEBIAN_FRONTEND=noninteractive` to avoid
config file conflict prompts hanging under `qm guest exec` (no TTY).

## Phase 2e: Upgrade Completion & Transient CP Failure (COMPLETED)

### Result

Second Ansible playbook run succeeded. All 6 nodes upgraded to
v1.35.6+rke2r1. PLAY RECAP showed `failed=1` on CP-03 (VM129) — but
investigation via `qm guest exec 129` confirmed the service was already
active and running v1.35.6+rke2r1. The failure was a timing race:
Ansible's `systemctl restart rke2-server` raced with RKE2's internal
reload from the binary swap. The service had already started before
Ansible tried to restart it, causing a transient "already activating"
error.

### Verification

```bash
kubectl get nodes -o wide
# All 6 nodes: Ready, v1.35.6+rke2r1, containerd 2.2.5-k3s2
kubectl get pods -A | grep -v Running | grep -v Completed
# Empty — 0 broken pods
```

### Lesson: Transient CP restart failures can be benign

If a CP node shows `failed=1` in the PLAY RECAP but is `Ready` and
running the target version, the failure was a race condition between
Ansible's `systemctl restart` and RKE2's internal reload during the
binary swap. No remediation needed — verify cluster state with
`kubectl get nodes` rather than blindly re-running the playbook.

## Remaining Work

1. ✅ OS Updates — DONE
2. ✅ RKE2 Upgrade v1.35.2→v1.35.6 — COMPLETED (commit 6cb1b3c)
3. ✅ Helm Chart Updates — COMPLETED (commits 36eb5c4, acd3231)
   See `references/helm-chart-upgrades-2026-07.md` for full detail.
   ArgoCD v2→v3, External Secrets, Ceph CSI RBD, CNPG, Velero all upgraded.
4. ✅ openclaw-memory — REMOVED via GitOps (commit ebc4da5)
5. ✅ ArgoCD `backups` app OutOfSync — ROOT CAUSE FOUND (cosmetic:
   ESO default fields in live spec not in Git YAML). Fix pending
   user approval (read-only investigation completed).

## Lessons Learned

1. **NEVER extract RKE2 tarball to `/`** — always use `/usr/local/`
2. **Debian 12 usrmerge**: `/bin`, `/lib`, `/sbin` are symlinks, not
   real directories. Any tarball that creates `bin/` or `lib/` at root
   will destroy the system.
3. **Ceph RBD offline mount** is viable for VM filesystem repair when
   guest agent is broken — stop VM, map RBD, mount partition, fix, unmount, start.
4. **Always discover VM→PVE-Node mapping dynamically** — tofu state and
   actual placement diverge due to HA migrations.
5. **Etcd quorum**: With 3 CP nodes, stopping 1 is safe (2/3 quorum).
   Never stop 2 simultaneously.
6. **RKE2 version must match cached runtime images** — a newer binary
   without matching images will loop "activating" forever.
7. **GitOps-first**: User expects cluster changes through IaC repo, not
   manual interventions. Manual SSH/qm-guest-exec should be last resort.
8. **Non-operator app removal via GitOps** is straightforward: delete
   manifests → push → delete ArgoCD App (cascade) → delete namespace.
   No finalizer complications unlike operator-managed resources.
9. **Ansible SSH prerequisite**: mgmt-runner (VM200) needs its SSH public
   key in `debian@VMs/.ssh/authorized_keys` before Ansible can reach K8s nodes.
   Generate keypair on VM200, distribute via `qm guest exec` to all 6 VMs,
   verify with `ansible all -m ping` before running the playbook.
10. **GitOps-first principle reinforced**: User explicitly directed to use
    the IaC repo (`rke2_version` variable in Ansible playbook) for upgrades
    rather than manual operations. All cluster changes should go through
    `dominik/iac-homelab` on Gitea → commit → push → Ansible/ArgoCD.
11. **dpkg interrupted state blocks Ansible**: If OS updates via `qm guest
    exec` leave dpkg half-configured (e.g. openssh-server mid-upgrade with
    a config file conflict prompt hanging without TTY), the Ansible
    playbook's pre_tasks (`apt install ceph-common rbd-nbd`) will fail on
    that node. Fix: `fuser -k` on dpkg AND debconf locks, then
    `DEBIAN_FRONTEND=noninteractive dpkg --configure -a` on each affected
    node BEFORE running the upgrade playbook.
12. **debconf has its own lock**: Beyond `/var/lib/dpkg/lock-frontend` and
    `/var/lib/dpkg/lock`, debconf maintains `/var/cache/debconf/config.dat`.
    A stale process holding this lock causes `dpkg --configure -a` to fail
    even after clearing dpkg locks. Must `fuser -k` all three.
13. **Transient CP restart failures can be benign**: Ansible's
    `systemctl restart rke2-server` can race with RKE2's internal reload
    during the binary swap. If a CP node shows `failed=1` in PLAY RECAP
    but is `Ready` and running the target version, the failure was a
    timing race — no remediation needed. Always verify with
    `kubectl get nodes -o wide` rather than blindly re-running the playbook.