Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
@@ -0,0 +1,162 @@
|
||||
# RKE2 Cluster Cold-Boot Recovery — 2026-07-12
|
||||
|
||||
## Situation
|
||||
|
||||
RKE2 cluster (6 VMs: 3 CP + 3 Worker) was offline for 111 days. All VMs
|
||||
were stopped across the PVE cluster. User requested cluster startup
|
||||
and K8s evaluation for Docker workload migration.
|
||||
|
||||
## Startup Sequence
|
||||
|
||||
### 1. VM Discovery
|
||||
|
||||
VMs had been HA-migrated to various PVE nodes. Had to scan all nodes
|
||||
to find where each VM landed:
|
||||
|
||||
```
|
||||
118 (CP-01) → proxmox5
|
||||
130 (CP-02) → proxmox3
|
||||
129 (CP-03) → proxmox2
|
||||
128 (Worker-01) → proxmox5
|
||||
132 (Worker-02) → proxmox2
|
||||
131 (Worker-03) → proxmox3
|
||||
200 (mgmt-runner) → proxmox3
|
||||
```
|
||||
|
||||
### 2. Start Order
|
||||
|
||||
1. Started all 6 RKE2 VMs via `qm start` (HA placed them on various nodes)
|
||||
2. CP nodes began etcd convergence within 30s
|
||||
3. Workers initially showed `NotReady` — waiting for
|
||||
`serving-kubelet.crt` retrieval (503 from API server)
|
||||
4. Worker-03 became Ready after ~2 min
|
||||
5. Worker-02 became Ready after ~3 min (slower due to proxy reconnection)
|
||||
6. All 6 nodes Ready within 5 min
|
||||
|
||||
### 3. Convergence Timeline
|
||||
|
||||
| T+0 | VMs started |
|
||||
| T+30s | CPs booting, etcd forming quorum |
|
||||
| T+1min | Workers connecting via wss://CP:9345/v1-rke2/connect |
|
||||
| T+2min | First workers Ready (cert retrieval successful) |
|
||||
| T+3min | All nodes Ready, Cilium agents starting |
|
||||
| T+5min | ArgoCD, External Secrets, operators beginning reconciliation |
|
||||
| T+10min | Most pods Running, old replicas Terminating |
|
||||
| T+15min | Databases recovering (Galera, PostgreSQL) |
|
||||
|
||||
### 4. Pod States After Cold Boot
|
||||
|
||||
Most pods showed mixed states initially:
|
||||
- `Running` — new replicas scheduled by deployment controller
|
||||
- `Terminating` — old replicas from before shutdown (111 days old)
|
||||
- `Unknown` — node was down when status was last reported
|
||||
- `CrashLoopBackOff` — ceph-csi nodeplugin (2 of 6) — Ceph connectivity
|
||||
- `CreateContainerConfigError` — memory-api, qdrant — missing secrets
|
||||
(ExternalSecret hadn't synced yet)
|
||||
|
||||
All resolved naturally except:
|
||||
- 3 ExternalSecrets with SecretSyncedError (1Password item issues)
|
||||
- 2 ceph-csi-rbd-nodeplugin CrashLoopBackOff
|
||||
|
||||
## Cilium LB Pool Migration
|
||||
|
||||
### Problem
|
||||
|
||||
Cilium LB pool was set to `10.0.30.70-10.0.30.89`, overlapping with
|
||||
Galera/MaxScale VIPs:
|
||||
- 10.0.30.70 = Keepalived VIP for MaxScale (Galera)
|
||||
- 10.0.30.71-73 = Galera VM IPs
|
||||
|
||||
K8s services had claimed:
|
||||
- Traefik → 10.0.30.70 (conflicted with MaxScale VIP!)
|
||||
- PostgreSQL RW → 10.0.30.71 (conflicted with Galera VM!)
|
||||
- MariaDB MaxScale (K8s) → 10.0.30.72
|
||||
|
||||
### Fix Applied
|
||||
|
||||
1. Deleted live `CiliumLoadBalancerIPPool` CRD
|
||||
2. Created new pool with `10.0.30.200-10.0.30.250`
|
||||
3. Services auto-reassigned:
|
||||
- Traefik → 10.0.30.202
|
||||
- MariaDB MaxScale (K8s) → 10.0.30.201
|
||||
- PostgreSQL RW → 10.0.30.200
|
||||
4. Updated manifest on all 3 CP nodes:
|
||||
`/var/lib/rancher/rke2/server/manifests/cilium-l2-lb.yaml`
|
||||
5. Fixed IaC repo: `epic-2-k8s/ansible/playbook.yml` — committed + pushed
|
||||
|
||||
### Verification
|
||||
|
||||
- `10.0.30.70` responds to ping (Keepalived VIP for MaxScale — correct!)
|
||||
- No K8s LoadBalancer service holds IPs in .70-.89 range
|
||||
- Cilium LB pool shows 48 IPs available in .200-.250 range
|
||||
|
||||
## K8s MariaDB Removal
|
||||
|
||||
### Decision
|
||||
|
||||
User clarified: Galera lives as **native VMs** (VM300-302 + MaxScale
|
||||
VM310), not in K8s. The K8s MariaDB Galera (3 pods) + MaxScale (2 pods)
|
||||
in namespace `mariadb` was redundant and should be removed.
|
||||
|
||||
### Removal Steps
|
||||
|
||||
1. Removed YAML files from `iac-homelab` repo:
|
||||
- `clusters/main/databases/mariadb/` (5 files: cluster.yaml,
|
||||
external-secrets.yaml, maxscale.yaml, namespace.yaml, README.md)
|
||||
- `clusters/main/operators/mariadb-operator.yaml`
|
||||
- `clusters/main/operators/mariadb-operator-crds.yaml`
|
||||
2. Committed + pushed to Gitea
|
||||
3. ArgoCD detected change (`databases` → `OutOfSync`) but did NOT
|
||||
auto-prune the operator-managed CRs
|
||||
4. Manual cleanup required:
|
||||
- `kubectl delete mariadb mariadb-galera -n mariadb` (CR deletion
|
||||
triggers operator finalizer processing)
|
||||
- `kubectl delete maxscale mariadb-maxscale -n mariadb` (already
|
||||
gone — operator had processed it)
|
||||
- `kubectl delete namespace mariadb` (clean)
|
||||
- `kubectl delete namespace mariadb-operator` (stuck Terminating
|
||||
briefly, cleared after ~15s)
|
||||
- `kubectl delete crd ...` (9 mariadb CRDs — ArgoCD didn't prune
|
||||
these, had to delete manually)
|
||||
5. Forced ArgoCD hard-refresh: `kubectl annotate application root -n
|
||||
argocd argocd.argoproj.io/refresh-options=hard-refresh --overwrite`
|
||||
|
||||
### Result
|
||||
|
||||
- Namespaces `mariadb` + `mariadb-operator` deleted
|
||||
- 9 MariaDB CRDs deleted
|
||||
- ArgoCD apps `mariadb-operator` + `mariadb-operator-crds` pruned
|
||||
- `databases` app re-synced (only postgres remains)
|
||||
- Native Galera VMs (300-302) + MaxScale VM310 untouched
|
||||
- LB IP 10.0.30.201 freed (was K8s MaxScale, now available in pool)
|
||||
|
||||
### Key Lesson
|
||||
|
||||
ArgoCD `prune: true` handles standard K8s resources but **does not
|
||||
reliably prune operator CRs and CRDs** — finalizers require the
|
||||
operator to be alive, and CRD deletion isn't tracked by ArgoCD's
|
||||
resource tree. Manual `kubectl delete` of CRs → namespace → CRDs is
|
||||
required when removing an operator-managed application.
|
||||
|
||||
ArgoCD was already installed and configured:
|
||||
- 8 Applications synced (operators, databases, backups, memory, root)
|
||||
- Ingress via Traefik (host: `argocd`, routed through Traefik LB IP)
|
||||
- App-of-Apps pattern pointing to `dominik/iac-homelab` on Gitea
|
||||
- Auto-sync with `prune: true, selfHeal: true`
|
||||
|
||||
Admin password retrieved via:
|
||||
```bash
|
||||
kubectl -n argocd get secret argocd-initial-admin-secret \
|
||||
-o jsonpath='{.data.password}' | base64 -d
|
||||
```
|
||||
|
||||
## IaC Repository
|
||||
|
||||
`dominik/iac-homelab` on Gitea (10.0.30.105:3000) contains:
|
||||
- 7 Epics (management VM, K8s cluster, networking, storage, GitOps, memory, MariaDB)
|
||||
- OpenTofu for VM provisioning + Helm releases
|
||||
- Ansible for RKE2 bootstrap
|
||||
- GitHub Actions CI/CD (self-hosted runner on mgmt-runner VM)
|
||||
- ArgoCD Application manifests in `clusters/main/apps/`
|
||||
|
||||
Separate repo: `dominik/observability-iac` — Loki, Prometheus/Thanos, Grafana
|
||||
@@ -0,0 +1,229 @@
|
||||
# Helm Chart Upgrades — 2026-07-12
|
||||
|
||||
## Goal
|
||||
|
||||
Upgrade all Helm-managed charts on the RKE2 cluster after the v1.35.6
|
||||
upgrade. 5 components needed updates; RKE2-bundled charts were already
|
||||
current.
|
||||
|
||||
## Version Matrix (Final)
|
||||
|
||||
| Component | Old Version | New Version | Method | Status |
|
||||
|-----------|------------|-------------|--------|--------|
|
||||
| External Secrets | 2.2.0 | 2.7.0 | Helm direct | ✅ |
|
||||
| CNPG | 0.22.1 (app 1.24.1) | 0.29.0 (app 1.30.0) | ArgoCD GitOps | ✅ |
|
||||
| Velero | 8.1.0 (app 1.15.0) | 12.1.0 (app 1.18.1) | ArgoCD GitOps | ✅ |
|
||||
| Ceph CSI RBD | 3.10.1 | 3.17.0 | Helm direct | ✅ |
|
||||
| ArgoCD | 7.8.13 (app v2.14.7) | 10.1.3 (app v3.4.5) | Helm direct | ✅ |
|
||||
| Cilium | rke2-cilium-1.19.402 | — | RKE2-bundled | Already current |
|
||||
| Traefik | rke2-traefik-40.1.003 | — | RKE2-bundled | Already current |
|
||||
| CoreDNS | rke2-coredns-1.46.002 | — | RKE2-bundled | Already current |
|
||||
| Metrics Server | rke2-metrics-server-3.13.100 | — | RKE2-bundled | Already current |
|
||||
| Snapshot Ctrl | rke2-snapshot-controller-4.2.006 | — | RKE2-bundled | Already current |
|
||||
|
||||
## Upgrade Order (Safest → Riskiest)
|
||||
|
||||
1. External Secrets 2.2.0 → 2.7.0 (Minor, Helm direct, minimal values)
|
||||
2. CNPG 0.22.1 → 0.29.0 (Major, ArgoCD GitOps, **required replica fix first**)
|
||||
3. Velero 8.1.0 → 12.1.0 (Major, ArgoCD GitOps)
|
||||
4. Ceph CSI RBD 3.10.1 → 3.17.0 (Major, Helm direct, **causes transient API outage**)
|
||||
5. ArgoCD v2.14.7 → v3.4.5 (MAJOR v2→v3, Helm direct, **breaking change**)
|
||||
|
||||
## Detail: External Secrets (2.2.0 → 2.7.0)
|
||||
|
||||
Straightforward `helm upgrade --reuse-values`. Values were minimal
|
||||
(`installCRDs: true`). No breaking changes. All ExternalSecrets
|
||||
continued syncing. Two pre-existing broken ExternalSecrets
|
||||
(argocd-repo-credentials, postgres-main-db) were already broken for
|
||||
110+ days — unrelated to the upgrade.
|
||||
|
||||
## Detail: CNPG (0.22.1 → 0.29.0) — Replica Rebuild Required
|
||||
|
||||
### Pre-Existing Issue Found
|
||||
|
||||
`postgres-main-2` replica was broken for 109 days:
|
||||
```
|
||||
requested WAL segment 000000020000000000000008 has already been removed
|
||||
WAL file not found in the recovery object store
|
||||
```
|
||||
Primary had recycled WAL segments the replica still needed. Cluster
|
||||
was running with 2/3 Ready (primary + 1 replica), backups operational.
|
||||
|
||||
### Replica Rebuild
|
||||
|
||||
```bash
|
||||
# Delete broken replica PVCs + pod
|
||||
kubectl delete pvc postgres-main-2 postgres-main-2-wal -n postgres
|
||||
kubectl delete pod postgres-main-2 -n postgres
|
||||
|
||||
# CNPG automatically created postgres-main-4 as replacement
|
||||
# Wait for 3/3 Ready:
|
||||
kubectl get pods -n postgres -w
|
||||
```
|
||||
|
||||
### Operator Upgrade via GitOps
|
||||
|
||||
```bash
|
||||
# Changed targetRevision in ArgoCD Application manifest
|
||||
cd /root/iac-homelab
|
||||
sed -i 's/targetRevision: 0.22.1/targetRevision: 0.29.0/' \
|
||||
clusters/main/operators/cloudnativepg.yaml
|
||||
git add -A && git commit -m "feat: upgrade CNPG 0.22.1 → 0.29.0" && git push
|
||||
```
|
||||
|
||||
**Pitfall**: ArgoCD didn't auto-detect the chart version change despite
|
||||
`selfHeal: true`. Required hard refresh:
|
||||
```bash
|
||||
kubectl annotate application cloudnativepg-operator -n argocd \
|
||||
argocd.argoproj.io/refresh=hard --overwrite
|
||||
```
|
||||
|
||||
After refresh, ArgoCD pulled new chart, created new operator pod with
|
||||
image v1.30.0. Postgres cluster briefly went 2/3 during reconciliation
|
||||
(operator triggered primary restart), then recovered to 3/3 Healthy.
|
||||
|
||||
## Detail: Velero (8.1.0 → 12.1.0)
|
||||
|
||||
Same GitOps pattern as CNPG — changed `targetRevision` in ArgoCD
|
||||
Application manifest, pushed, hard-refreshed ArgoCD. All Velero
|
||||
components rolled out: server + 6 node-agents (one per K8s node).
|
||||
BSL (Backup Storage Location) remained Available throughout. Backup
|
||||
schedules (daily-full-cluster, weekly-full-cluster) continued working.
|
||||
|
||||
## Detail: Ceph CSI RBD (3.10.1 → 3.17.0) — API Server Disruption
|
||||
|
||||
### Upgrade
|
||||
|
||||
```bash
|
||||
helm upgrade ceph-csi-rbd cephcsi/ceph-csi-rbd \
|
||||
--version 3.17.0 -n kube-system --reuse-values --timeout 120s
|
||||
```
|
||||
|
||||
### Transient API Server Outage
|
||||
|
||||
The nodeplugin DaemonSet rolling update disrupted kubelet on CP nodes.
|
||||
All 3 API servers became unreachable:
|
||||
```
|
||||
The connection to the server 10.0.30.51:6443 was refused
|
||||
```
|
||||
|
||||
Recovery took ~5 minutes. Monitored with:
|
||||
```bash
|
||||
curl -sk --connect-timeout 5 https://10.0.30.51:6443/healthz
|
||||
# 401 Unauthorized = server is UP (expected without credentials)
|
||||
# Connection refused = server is DOWN
|
||||
```
|
||||
|
||||
After recovery, provisioner pods (2 replicas) were 7/7 Running with
|
||||
image `quay.io/cephcsi/cephcsi:v3.17.0`. One nodeplugin pod stayed in
|
||||
`ContainerCreating` for several minutes (image pull delay on CP-01).
|
||||
All 7 PVCs remained bound throughout — no data impact.
|
||||
|
||||
### Root Cause
|
||||
|
||||
The Ceph CSI nodeplugin DaemonSet runs on EVERY node including CP
|
||||
nodes. Rolling the DaemonSet causes the CSI driver socket to be
|
||||
briefly unavailable, which can block kubelet operations on CP nodes,
|
||||
temporarily preventing the API server from serving requests.
|
||||
|
||||
## Detail: ArgoCD v2.14.7 → v3.4.5 (MAJOR Upgrade)
|
||||
|
||||
### Safe Path: Two-Step Upgrade
|
||||
|
||||
Step 1: Patch to latest v2.x (low risk):
|
||||
```bash
|
||||
helm upgrade argocd argo/argo-cd --version 7.9.1 -n argocd --reuse-values
|
||||
# v2.14.7 → v2.14.11
|
||||
```
|
||||
|
||||
Step 2: Major v2→v3 upgrade with explicit values:
|
||||
```bash
|
||||
# v3 chart requires redis.networkPolicy.create field (breaking change)
|
||||
# --reuse-values FAILS with:
|
||||
# nil pointer evaluating interface {}.create
|
||||
# Must provide explicit values file with redis.networkPolicy.create: false
|
||||
|
||||
helm upgrade argocd argo/argo-cd --version 10.1.3 -n argocd \
|
||||
-f /tmp/argocd-v3-values.yaml --timeout 300s
|
||||
```
|
||||
|
||||
### v3 Values File
|
||||
|
||||
Key addition vs v2 values:
|
||||
```yaml
|
||||
redis:
|
||||
networkPolicy:
|
||||
create: false
|
||||
```
|
||||
|
||||
All other values carried over unchanged (server.insecure, ingress,
|
||||
replicas, affinity).
|
||||
|
||||
### Post-Upgrade Verification
|
||||
|
||||
- All ArgoCD pods Running (server ×2, repo-server ×2,
|
||||
application-controller, applicationset-controller, dex-server,
|
||||
notifications-controller, redis)
|
||||
- Image: `quay.io/argoproj/argocd:v3.4.5`
|
||||
- All applications Synced (except `backups` — pre-existing
|
||||
ExternalSecret OutOfSync, see §13.7)
|
||||
|
||||
## Detail: ArgoCD `backups` App OutOfSync (Task 4)
|
||||
|
||||
### Root Cause
|
||||
|
||||
Single resource OutOfSync: `ExternalSecret/velero-s3-credentials` in
|
||||
`velero` namespace. ESO adds default fields to the live spec that
|
||||
aren't in the Git YAML:
|
||||
- `conversionStrategy: "Default"`
|
||||
- `decodingStrategy: "None"`
|
||||
- `metadataPolicy: "None"`
|
||||
- `deletionPolicy: "Retain"`
|
||||
- `engineVersion: "v2"`
|
||||
- `mergePolicy: "Replace"`
|
||||
|
||||
ArgoCD's `ignoreDifferences` only covers `/status`, not `/spec`.
|
||||
|
||||
### Impact
|
||||
|
||||
Cosmetic — ExternalSecret functions correctly (`secret synced`,
|
||||
Velero backups operational). No data or functionality impact.
|
||||
|
||||
### Fix (Not Yet Applied — Read-Only Investigation)
|
||||
|
||||
Add ESO default fields to Git YAML at
|
||||
`clusters/main/backups/velero/external-secret.yaml` so spec matches
|
||||
exactly. Alternative: expand `ignoreDifferences` to cover specific
|
||||
JSON pointers (less clean).
|
||||
|
||||
## GitOps Trail
|
||||
|
||||
Commits pushed to `dominik/iac-homelab` on Gitea:
|
||||
- `36eb5c4` — `feat(helm): upgrade all managed charts` (values files +
|
||||
ArgoCD Application targetRevision changes)
|
||||
- `acd3231` — `chore: remove .kube cache from repo, add to .gitignore`
|
||||
(cleanup of accidentally committed `.kube/cache/` directory)
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Always check CNPG cluster health before upgrading the operator** —
|
||||
a broken replica (WAL gap) should be rebuilt first (delete PVC + pod,
|
||||
CNPG auto-creates fresh replica from base backup).
|
||||
2. **ArgoCD GitOps chart bumps need hard refresh** — changing
|
||||
`targetRevision` in the Application manifest and pushing doesn't
|
||||
guarantee ArgoCD pulls the new chart. Annotate with
|
||||
`argocd.argoproj.io/refresh=hard` to force.
|
||||
3. **Ceph CSI RBD DaemonSet upgrades disrupt API servers** — the
|
||||
nodeplugin DaemonSet rolls across ALL nodes including CP nodes,
|
||||
briefly disrupting kubelet and making API servers unreachable for
|
||||
2-5 minutes. Schedule during maintenance windows.
|
||||
4. **ArgoCD v2→v3 chart requires redis.networkPolicy.create** —
|
||||
`--reuse-values` fails with nil pointer error. Must provide
|
||||
explicit values file with `redis.networkPolicy.create: false`.
|
||||
5. **ExternalSecret perpetual OutOfSync** — ESO adds default fields to
|
||||
live spec not present in Git YAML. Fix by adding those fields to the
|
||||
Git source YAML.
|
||||
6. **`.kube/cache` gets committed by `git add -A`** — always add
|
||||
`.kube/` to `.gitignore` before running kubectl from a repo root.
|
||||
7. **Save Helm values to IaC repo** — `helm get values` → save to
|
||||
`epic-2-k8s/helm/<chart>/values.yaml` for reproducible installs.
|
||||
@@ -0,0 +1,95 @@
|
||||
# Hindsight K8s Deployment — Session Detail (2026-07-12)
|
||||
|
||||
## Context
|
||||
Hindsight daemon on Hermes host (Debian 12, GLIBC 2.36) couldn't start
|
||||
because the Rust binary needs GLIBC 2.39. Decision: deploy hindsight-api
|
||||
(Python PyPI package) as a K8s Deployment in the RKE2 cluster.
|
||||
|
||||
## What was done
|
||||
|
||||
1. Discovered `hindsight-api` is a Python package on PyPI (v0.8.4),
|
||||
NOT the Rust binary. The Rust binary is only the CLI client.
|
||||
2. Created K8s manifests:
|
||||
- Namespace: `hindsight`
|
||||
- StatefulSet: `hindsight-postgres` (image: `pgvector/pgvector:pg16`)
|
||||
- Deployment: `hindsight-api` (image: `python:3.11-slim`, venv + pip install)
|
||||
- Services, ExternalSecrets, ArgoCD Application
|
||||
3. Created 1Password items in vault "Kubernetes ESO":
|
||||
- `noris-api-key` (API Credential) → credential = noris API key
|
||||
- `hindsight-db-password` (Password) → password = random string
|
||||
4. Committed to `dominik/iac-homelab` repo, pushed to Gitea
|
||||
5. ArgoCD auto-synced the new Application
|
||||
|
||||
## Issues hit and fixes (chronological)
|
||||
|
||||
### Issue 1: ExternalSecret API version
|
||||
- **Error**: `"The Kubernetes API could not find version 'v1beta1'"`
|
||||
- **Fix**: Changed `apiVersion` from `external-secrets.io/v1beta1` to `external-secrets.io/v1`
|
||||
|
||||
### Issue 2: StorageClass name
|
||||
- **Error**: PVC stuck `Pending` with no PV created
|
||||
- **Fix**: Changed `ceph-rbd` → `ceph-hdd-replica`
|
||||
|
||||
### Issue 3: ExternalSecret key format
|
||||
- **Error**: `"secret reference has invalid format"`
|
||||
- **Fix**: Combined into `key: item-name/field-name` format, removed `property:` field
|
||||
|
||||
### Issue 4: 1Password vault mismatch
|
||||
- **Error**: `"no vault matched the secret reference query"`
|
||||
- **Cause**: Items created in vault "Hermes" via local `op` CLI, but K8s
|
||||
ClusterSecretStore uses vault "Kubernetes ESO" (`334ykdtj5kar3jlpcrztjvx2fu`)
|
||||
- **Fix**: Used K8s service account token to create items in correct vault:
|
||||
```bash
|
||||
K8S_TOKEN=$(kubectl get secret onepassword-token -n external-secrets \
|
||||
-o jsonpath='{.data.token}' | base64 -d)
|
||||
OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op item create \
|
||||
--category="API Credential" --title="noris-api-key" \
|
||||
--vault="334ykdtj5kar3jlpcrztjvx2fu" credential=VALUE
|
||||
```
|
||||
- **Key learning**: K8s service account token CAN create items (not read-only)
|
||||
|
||||
### Issue 5: K8s env var interpolation order
|
||||
- **Fix**: Moved `HINDSIGHT_DB_PASSWORD` to be the first env var (before `DATABASE_URL`)
|
||||
|
||||
### Issue 6: ArgoCD stale manifest cache
|
||||
- **Fix**: Delete Application and recreate, or force hard refresh via annotation
|
||||
|
||||
### Issue 7: StatefulSet PVC immutability
|
||||
- **Fix**: Delete STS + PVC manually, let ArgoCD recreate with corrected manifest
|
||||
|
||||
### Issue 8: PostgreSQL lost+found conflict on PVC
|
||||
- **Error**: Postgres CrashLoopBackOff — initdb refuses non-empty data directory
|
||||
- **Cause**: Ceph PVC has `lost+found` directory at root
|
||||
- **Fix**: Set `PGDATA=/var/lib/postgresql/data/pgdata` (subdirectory)
|
||||
|
||||
### Issue 9: pip --target doesn't install console_scripts
|
||||
- **Error**: Main container: `/bin/sh: 1: hindsight-api: not found`
|
||||
- **Cause**: `pip install --target=/opt/packages` copies modules but NOT bin/ scripts
|
||||
- **Fix**: Use `python -m venv /opt/venv` instead — creates full venv with bin/
|
||||
|
||||
### Issue 10: CUDA torch (526MB) causes OOMKill
|
||||
- **Error**: Init container OOMKilled during pip install
|
||||
- **Cause**: Default torch wheel includes CUDA libraries (526MB+)
|
||||
- **Fix**: Install CPU torch first with `--index-url https://download.pytorch.org/whl/cpu`,
|
||||
then main package with `--extra-index-url` for same CPU index
|
||||
|
||||
### Issue 11: kubectl rollout restart doesn't apply manifest changes
|
||||
- **Error**: After `rollout restart`, new pod still uses old spec
|
||||
- **Cause**: `rollout restart` only restarts with current deployment spec, doesn't pull new manifests
|
||||
- **Fix**: `kubectl apply -f <manifest>` first, THEN `rollout restart` if needed
|
||||
|
||||
### Issue 12: ArgoCD doesn't recreate manually deleted resources
|
||||
- **Error**: After `kubectl delete externalsecret`, ArgoCD reports "successfully synced"
|
||||
but doesn't recreate the ES
|
||||
- **Fix**: `kubectl apply -f <manifest>` manually
|
||||
|
||||
## State at session end
|
||||
- Postgres pod: 1/1 Running ✅
|
||||
- ExternalSecrets: Synced, Secrets created ✅
|
||||
- LB Service: 10.0.30.201:9177 assigned ✅
|
||||
- API pod: Init container pip install (venv + CPU torch) takes 5-10min
|
||||
- Latest commit `8424566` uses venv approach
|
||||
- Needs `kubectl apply -f` on VM200 to pick up the venv fix
|
||||
- Previous `rollout restart` didn't apply the manifest change
|
||||
- **Next step**: `kubectl apply -f clusters/main/hindsight/api-deployment.yaml` on VM200,
|
||||
wait ~10min for init container, verify pod becomes Ready, then update Hermes config
|
||||
@@ -0,0 +1,106 @@
|
||||
# RKE2 Cluster Topology & Inventory
|
||||
|
||||
## Cluster Nodes
|
||||
|
||||
| VMID | Name | Role | IP | Specs | RKE2 Service |
|
||||
|------|------|------|----|-------|-------------|
|
||||
| 118 | rke2-cp-01 | control-plane,etcd | 10.0.30.51 | 4c/12GB/40GB | rke2-server |
|
||||
| 130 | rke2-cp-02 | control-plane,etcd | 10.0.30.52 | 4c/12GB/40GB | rke2-server |
|
||||
| 129 | rke2-cp-03 | control-plane,etcd | 10.0.30.53 | 4c/12GB/40GB | rke2-server |
|
||||
| 128 | rke2-worker-01 | worker | 10.0.30.61 | 4c/12GB/80GB | rke2-agent |
|
||||
| 132 | rke2-worker-02 | worker | 10.0.30.62 | 4c/12GB/80GB | rke2-agent |
|
||||
| 131 | rke2-worker-03 | worker | 10.0.30.63 | 4c/12GB/80GB | rke2-agent |
|
||||
|
||||
- **RKE2 API VIP**: 10.0.30.50 (Keepalived on CP nodes)
|
||||
- **RKE2 version**: v1.35.2+rke2r1
|
||||
- **Container runtime**: containerd 2.1.5-k3s1
|
||||
- **OS**: Debian 12 (bookworm), kernel 6.1.0-44-amd64
|
||||
|
||||
## Management VM
|
||||
|
||||
| VMID | Name | IP | Specs |
|
||||
|------|------|----|-------|
|
||||
| 200 | mgmt-runner-01 | 10.0.30.X (DHCP) | 4c/4GB/30GB |
|
||||
|
||||
- Has `kubectl`, `helm`, `git`, GitHub Actions runner
|
||||
- Kubeconfig at `/root/.kube/config` (must set `KUBECONFIG=` explicitly)
|
||||
- GitHub Actions runner: `actions.runner.dsgrafiniert-iac-homelab.mgmt-runner-01`
|
||||
- OpenTofu state files in `/opt/tofu-state/`
|
||||
|
||||
## LoadBalancer IP Assignments
|
||||
|
||||
| Service | Namespace | LB IP | Ports |
|
||||
|---------|-----------|-------|-------|
|
||||
| Traefik Ingress | kube-system | 10.0.30.202 | 80, 443 |
|
||||
| PostgreSQL RW (CNPG) | postgres | 10.0.30.200 | 5432 |
|
||||
|
||||
**Cilium LB Pool**: 10.0.30.200 - 10.0.30.250 (48 IPs available)
|
||||
|
||||
**Removed**: K8s MariaDB MaxScale (was 10.0.30.201, namespace deleted
|
||||
2026-07-12). MariaDB Galera runs as native VMs, not in K8s.
|
||||
|
||||
**Non-K8s services on same subnet** (must not overlap with LB pool):
|
||||
- 10.0.30.50 = RKE2 API VIP (Keepalived)
|
||||
- 10.0.30.70 = MaxScale VIP (Keepalived, standalone Galera VMs)
|
||||
- 10.0.30.71-73 = Galera VMs (VM300-302)
|
||||
- 10.0.30.99 = Docker host (CT121)
|
||||
- 10.0.30.104 = Frigate (CT120)
|
||||
- 10.0.30.105 = Gitea (CT108)
|
||||
- 10.0.30.10 = Home Assistant (VM106)
|
||||
|
||||
## Operators & Add-ons
|
||||
|
||||
| Component | Namespace | Status |
|
||||
|-----------|-----------|--------|
|
||||
| Cilium CNI | kube-system | Running (eBPF, L2 announcements) |
|
||||
| Traefik Ingress | kube-system | Running (RKE2-bundled) |
|
||||
| CoreDNS | kube-system | Running (RKE2-bundled) |
|
||||
| Metrics Server | kube-system | Running |
|
||||
| External Secrets Operator | external-secrets | Running (1Password backend) |
|
||||
| CloudNativePG | cnpg-system | Running (PostgreSQL operator) |
|
||||
| Velero | velero | Running (S3 backup) |
|
||||
| Ceph CSI | kube-system | Running (RBD storage, external Ceph) |
|
||||
| ArgoCD | argocd | Running (GitOps, App-of-Apps) |
|
||||
|
||||
**Removed** (2026-07-12): MariaDB Operator + CRDs — namespace
|
||||
`mariadb` and `mariadb-operator` deleted. Galera runs as native VMs.
|
||||
|
||||
## ArgoCD Applications
|
||||
|
||||
| App | Sync Wave | Status | Description |
|
||||
|-----|-----------|--------|-------------|
|
||||
| root | — | Synced/Healthy | Root App-of-Apps |
|
||||
| operators | 0 | Synced/Healthy | CNPG, Velero operators |
|
||||
| databases | 1 | Synced/Healthy | PostgreSQL cluster |
|
||||
| backups | 2 | Synced/Healthy | Velero schedules |
|
||||
| memory | 3 | Unknown/Degraded | Qdrant + Ollama + Memory API |
|
||||
| cloudnativepg-operator | — | Synced/Healthy | CNPG operator |
|
||||
| velero-operator | — | Synced/Healthy | Velero operator |
|
||||
|
||||
**Removed** (2026-07-12): `mariadb-operator`, `mariadb-operator-crds`
|
||||
apps — deleted from repo, CRs cleaned up manually, namespaces + CRDs purged.
|
||||
|
||||
## IaC Repositories (Gitea at 10.0.30.105:3000)
|
||||
|
||||
| Repo | Content |
|
||||
|------|---------|
|
||||
| dominik/iac-homelab | Main IaC: 7 epics, OpenTofu, Ansible, ArgoCD apps |
|
||||
| dominik/observability-iac | Observability: Loki, Prometheus/Thanos, Grafana |
|
||||
| d.schoen/terraform-proxmox-k8s | Older Terraform-Proxmox-K8s work |
|
||||
| d.schoen/packer-proxmox | VM templates with Packer |
|
||||
|
||||
## PVE Node Map
|
||||
|
||||
| Node | IP | Notes |
|
||||
|------|----|-------|
|
||||
| proxmox1 | 10.0.20.10 | Cluster master (API endpoint) |
|
||||
| proxmox2 | 10.0.20.20 | Hosts CP-03, Worker-02 |
|
||||
| proxmox3 | 10.0.20.30 | Hosts CP-02, Worker-03, mgmt-runner |
|
||||
| proxmox4 | 10.0.20.40 | Hosts MaxScale VM, Galera VM |
|
||||
| proxmox5 | 10.0.20.50 | Hosts CP-01, Worker-01 |
|
||||
| proxmox6 | 10.0.20.60 | Hosts Gitea, Authelia, Paperless |
|
||||
| proxmox7 | 10.0.20.70 | Hosts monitoring, Omada, Ollama |
|
||||
| n5pro | 10.0.20.91 | AMD GPU, Frigate, Seafile, Immich |
|
||||
|
||||
**Note**: VM placements change with HA migration. Always discover
|
||||
current placement before operating (scan all nodes with `qm list`).
|
||||
@@ -0,0 +1,411 @@
|
||||
# RKE2 Upgrade Attempt & Tarball Recovery — 2026-07-12
|
||||
|
||||
## Goal
|
||||
|
||||
Update K8s cluster: OS updates on all VMs, RKE2 upgrade v1.35.2→v1.35.6,
|
||||
Helm chart updates, fix broken components. Scope: K8s only (no PVE hosts).
|
||||
|
||||
## Phase 1: OS Updates (COMPLETED)
|
||||
|
||||
All 7 VMs updated via `qm guest exec` through Proxmox jumphost chain.
|
||||
|
||||
### VM→PVE-Node Discovery
|
||||
|
||||
Initial assumption that all VMs were on proxmox3 was WRONG. Had to scan
|
||||
all PVE nodes to find actual placement:
|
||||
|
||||
```
|
||||
118 (CP-01) → proxmox5 (NOT proxmox3)
|
||||
130 (CP-02) → proxmox3
|
||||
129 (CP-03) → proxmox2
|
||||
128 (Worker-01) → proxmox5 (later moved to proxmox)
|
||||
132 (Worker-02) → proxmox2
|
||||
131 (Worker-03) → proxmox3
|
||||
200 (mgmt-runner) → proxmox3
|
||||
```
|
||||
|
||||
### Update Process
|
||||
|
||||
- VM200: Direct SSH (10.0.30.124) — 0 updates remaining
|
||||
- VM118, VM129: Updated via qm guest exec on respective PVE nodes
|
||||
- VM132: dpkg interrupted (timeout) — fixed with `dpkg --configure -a`
|
||||
- All VMs: 0 remaining updates after completion
|
||||
- Cluster stayed healthy throughout (6/6 nodes Ready)
|
||||
|
||||
### Command Pattern
|
||||
|
||||
```bash
|
||||
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.20.10 \
|
||||
"ssh -o ConnectTimeout=5 root@PVE_NODE 'qm guest exec VMID --timeout 600 \
|
||||
-- sh -c \"apt-get update && apt-get upgrade -y\"'"
|
||||
```
|
||||
|
||||
## Phase 2: RKE2 Upgrade (PARTIAL — TARBALL DISASTER)
|
||||
|
||||
### Initial Approach (WRONG)
|
||||
|
||||
Attempted manual tarball installation:
|
||||
1. Downloaded `rke2.linux-amd64.tar.gz` (v1.35.6) to VM118
|
||||
2. Extracted with `tar xf rke2.linux-amd64.tar.gz -C /` ← **FATAL ERROR**
|
||||
|
||||
### Tarball Destruction
|
||||
|
||||
The RKE2 tarball contains:
|
||||
```
|
||||
bin/ → RKE2 binaries
|
||||
lib/ → systemd unit files
|
||||
share/ → RKE2 data files
|
||||
```
|
||||
|
||||
Extracting to `/` on Debian 12 (which uses usrmerge) replaced:
|
||||
- `/bin` symlink (→ usr/bin) with a real directory containing only Rke2 binaries
|
||||
- `/lib` symlink (→ usr/lib) with a real directory containing only systemd files
|
||||
- Created bogus `/share` directory
|
||||
|
||||
Result: ALL system binaries inaccessible (`ls`, `sh`, `bash`, `systemctl`,
|
||||
`mount`). `qm guest exec` returns exit code 29. VM was still running
|
||||
(kernel + RKE2 processes in memory) but no new processes could start.
|
||||
|
||||
### Recovery: Ceph RBD Offline Mount
|
||||
|
||||
1. **Stopped VM118** (`qm stop 118`) — safe because 2/3 CP nodes still
|
||||
provide etcd quorum
|
||||
2. **Mapped RBD image** on proxmox5:
|
||||
```bash
|
||||
rbd map vm_disks/vm-118-disk-0 # → /dev/rbd1
|
||||
sleep 2 # Wait for udev
|
||||
```
|
||||
3. **Found partitions**: `rbd1p1` (root fs 39.9G), `rbd1p14` (BIOS boot),
|
||||
`rbd1p15` (EFI 124M)
|
||||
4. **Mounted root partition**: `mount /dev/rbd1p1 /mnt/vm118-rescue`
|
||||
5. **Verified damage**:
|
||||
- `/mnt/vm118-rescue/bin` = real directory with only `rke2`, `rke2-killall.sh`, `rke2-uninstall.sh`
|
||||
- `/mnt/vm118-rescue/lib` = real directory with only `systemd/`
|
||||
- `/mnt/vm118-rescue/usr/bin/bash` = INTACT (original system binaries fine)
|
||||
6. **Fixed symlinks**:
|
||||
```bash
|
||||
# Save RKE2 binaries
|
||||
cp /mnt/vm118-rescue/bin/rke2 /mnt/vm118-rescue/usr/local/bin/
|
||||
cp /mnt/vm118-rescue/bin/rke2-*.sh /mnt/vm118-rescue/usr/local/bin/
|
||||
|
||||
# Save systemd units
|
||||
mkdir -p /mnt/vm118-rescue/usr/lib/systemd/system
|
||||
cp -a /mnt/vm118-rescue/lib/systemd/system/rke2-*.service /mnt/vm118-rescue/usr/lib/systemd/system/
|
||||
cp -a /mnt/vm118-rescue/lib/systemd/system/rke2-*.env /mnt/vm118-rescue/usr/lib/systemd/system/
|
||||
|
||||
# Restore symlinks
|
||||
rm -rf /mnt/vm118-rescue/bin
|
||||
ln -s usr/bin /mnt/vm118-rescue/bin
|
||||
rm -rf /mnt/vm118-rescue/lib
|
||||
ln -s usr/lib /mnt/vm118-rescue/lib
|
||||
rm -rf /mnt/vm118-rescue/share
|
||||
```
|
||||
7. **Unmounted, unmapped, started VM**:
|
||||
```bash
|
||||
umount /mnt/vm118-rescue
|
||||
rbd unmap /dev/rbd1
|
||||
qm start 118
|
||||
```
|
||||
8. **Verified**: Guest agent responded after 60s boot. Shell works.
|
||||
BUT RKE2 server stuck "activating" — v1.35.6 binary couldn't find
|
||||
v1.35.6 runtime images (cached images were v1.35.2).
|
||||
|
||||
### Second Fix: Restore Matching Binary
|
||||
|
||||
Downloaded v1.35.2 tarball and extracted CORRECTLY:
|
||||
```bash
|
||||
wget https://github.com/rancher/rke2/releases/download/v1.35.2%2Brke2r1/rke2.linux-amd64.tar.gz
|
||||
tar xf rke2.linux-amd64.tar.gz -C /usr/local/ # CORRECT!
|
||||
rke2 --version # v1.35.2+rke2r1
|
||||
systemctl start rke2-server
|
||||
```
|
||||
|
||||
Node came back as Ready v1.35.2+rke2r1. Cluster fully healthy again.
|
||||
|
||||
### RBD Pitfalls Encountered
|
||||
|
||||
- **Device disappearance**: First `rbd map` created `/dev/rbd1` but it
|
||||
vanished before mount. Had to unmap stale mappings and re-map.
|
||||
- **No kpartx**: PVE node didn't have `kpartx`. Partition devices were
|
||||
created by kernel automatically after `sleep 2` post-mapping.
|
||||
- **partprobe failed**: `partprobe /dev/rbd1` gave "Could not stat device"
|
||||
because the device had already disappeared. Re-mapping fixed it.
|
||||
|
||||
## Phase 2b: Proper Upgrade Path (PREREQS IDENTIFIED, BLOCKED ON SSH)
|
||||
|
||||
### IaC Repo Analysis
|
||||
|
||||
The `epic-2-k8s/ansible/playbook.yml` uses `lablabs.rke2` role v1.50.1:
|
||||
```yaml
|
||||
rke2_version: v1.35.2+rke2r1 # ← Change this to upgrade
|
||||
rke2_token: "{{ lookup('env', 'RKE2_TOKEN') }}"
|
||||
rke2_ha_mode: true
|
||||
rke2_api_ip: 10.0.30.50
|
||||
```
|
||||
|
||||
Play 2 does rolling restart: Workers (serial:1) → Masters (serial:1).
|
||||
|
||||
### Prerequisites Status
|
||||
|
||||
1. ✅ Ansible role installed on VM200 (`ansible-galaxy install -r requirements.yml -p /root/.ansible/roles`)
|
||||
2. ✅ Inventory generated manually (tofu state shows VM IDs, IPs known)
|
||||
3. ✅ RKE2 token extracted from VM130 config (`/etc/rancher/rke2/config.yaml` → `token:` line)
|
||||
4. ❌ **SSH access from VM200 to K8s VMs** — `Permission denied (publickey)`
|
||||
- mgmt-runner's SSH key NOT in `debian@VMs/.ssh/authorized_keys`
|
||||
- Ansible inventory uses `ansible_user: debian`
|
||||
- Need to deploy VM200's public key to all 6 K8s VMs before Ansible can run
|
||||
|
||||
### Tofu State Details
|
||||
|
||||
Tofu state at `/opt/tofu-state/rke2-cluster.tfstate` has VM IDs:
|
||||
```
|
||||
rke2-cp-01: vm_id=118, node=proxmox (original placement, HA may move)
|
||||
rke2-cp-02: vm_id=130, node=proxmox2
|
||||
rke2-cp-03: vm_id=129, node=proxmox3
|
||||
rke2-worker-01: vm_id=128, node=proxmox
|
||||
rke2-worker-02: vm_id=132, node=proxmox2
|
||||
rke2-worker-03: vm_id=131, node=proxmox3
|
||||
```
|
||||
|
||||
### User Directive
|
||||
|
||||
User instructed: "Nutze GitOps!" and "Neue nodes mit neuer version
|
||||
erstellen. Cluster join. Verify. Workloads migrieren. Alte nodes löschen"
|
||||
|
||||
This means: prefer GitOps/Ansible for upgrades, or blue-green with new
|
||||
VMs if rolling upgrade is too risky. NEVER manual tarball installation.
|
||||
|
||||
## Phase 3: openclaw-memory Removal (COMPLETED via GitOps)
|
||||
|
||||
### Context
|
||||
|
||||
User: "Openclaw memory kann weg. Nutze ich nicht mehr"
|
||||
|
||||
The `openclaw-memory` namespace had 3 broken pods:
|
||||
- `memory-api` — CreateContainerConfigError (missing secret)
|
||||
- `qdrant-0` — CreateContainerConfigError (missing secret)
|
||||
- `ollama` — Running but unused
|
||||
|
||||
Two ExternalSecrets in `SecretSyncedError` state (wrong 1Password vault).
|
||||
|
||||
### Removal Steps (Clean GitOps)
|
||||
|
||||
1. **Deleted from Git repo**:
|
||||
```bash
|
||||
rm clusters/main/apps/memory.yaml # ArgoCD Application
|
||||
rm -rf clusters/main/memory/ # All manifests
|
||||
rm .github/workflows/epic6_prepare-memory-secrets.yml
|
||||
git commit -m "chore: remove openclaw-memory (unused, deprecated)"
|
||||
git push origin main # commit ebc4da5
|
||||
```
|
||||
|
||||
2. **Deleted ArgoCD Application** (cascade=foreground):
|
||||
```bash
|
||||
kubectl --server=https://10.0.30.52:6443 \
|
||||
delete application memory -n argocd --cascade=foreground
|
||||
```
|
||||
|
||||
3. **Deleted namespace** (pods cleared, namespace went Terminating → Gone):
|
||||
```bash
|
||||
kubectl --server=https://10.0.30.52:6443 delete ns openclaw-memory
|
||||
```
|
||||
|
||||
4. **Verified**: Namespace NotFound, 0 non-running pods, ArgoCD apps
|
||||
all Healthy except `backups` (OutOfSync, non-critical).
|
||||
|
||||
### Key Observations
|
||||
|
||||
- ArgoCD Application `--cascade=foreground` cleaned up managed resources
|
||||
(deployments, services) but namespace remained — needed manual `kubectl delete ns`.
|
||||
- No operator CRs involved, so no finalizer issues (unlike MariaDB removal).
|
||||
- `backups` ArgoCD app remains OutOfSync (pre-existing, not related).
|
||||
|
||||
## Phase 2c: SSH Key Deployment & Ansible Launch (COMPLETED)
|
||||
|
||||
### Problem
|
||||
|
||||
Ansible on VM200 could not SSH to K8s VMs — `Permission denied (publickey)`.
|
||||
VM200 had no SSH keypair at all (`~/.ssh/` only had `authorized_keys` and
|
||||
`known_hosts`).
|
||||
|
||||
### Fix
|
||||
|
||||
1. **Generated SSH key on VM200**:
|
||||
```bash
|
||||
ssh root@10.0.30.124 "ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N '' -C 'vm200-ansible'"
|
||||
```
|
||||
|
||||
2. **Confirmed actual VM→PVE-Node mapping** (all 6 nodes scanned):
|
||||
```
|
||||
118 (CP-01) → proxmox5
|
||||
130 (CP-02) → proxmox3
|
||||
129 (CP-03) → proxmox2
|
||||
128 (Worker-01) → proxmox5 (later: proxmox)
|
||||
131 (Worker-03) → proxmox3
|
||||
132 (Worker-02) → proxmox2
|
||||
```
|
||||
|
||||
3. **Distributed public key to all 6 VMs** via `qm guest exec` — each VM
|
||||
got the key appended to `/home/debian/.ssh/authorized_keys` with
|
||||
correct ownership (debian:debian) and permissions (700/600).
|
||||
All 6 returned OK.
|
||||
|
||||
4. **Verified Ansible connectivity**:
|
||||
```bash
|
||||
cd /root/iac-homelab/epic-2-k8s/ansible && ansible all -m ping
|
||||
```
|
||||
All 6 nodes: SUCCESS => { "ping": "pong" }
|
||||
|
||||
### Ansible Playbook Execution
|
||||
|
||||
With all prerequisites met, launched the upgrade:
|
||||
|
||||
```bash
|
||||
cd /root/iac-homelab/epic-2-k8s/ansible
|
||||
export RKE2_TOKEN='EyMCxVDobAkWjgyPDcwvXRJQHgWHoH6qgDcvFhQUkd'
|
||||
ansible-playbook playbook.yml -v
|
||||
```
|
||||
|
||||
The playbook:
|
||||
- Play 1: `lablabs.rke2` role applies `rke2_version: v1.35.6+rke2r1` to
|
||||
all nodes (writes config, downloads/installs new binaries)
|
||||
- Play 2: Rolling restart Workers (serial: 1) — each node restarted,
|
||||
waits for RKE2 agent active + node Ready
|
||||
- Play 3: Rolling restart Masters (serial: 1) — same pattern for CP
|
||||
|
||||
GitOps trail:
|
||||
- Commit `6cb1b3c`: `feat(rke2): upgrade v1.35.2 → v1.35.6+rke2r1`
|
||||
- Commit `ebc4da5`: `chore: remove openclaw-memory (unused, deprecated)`
|
||||
|
||||
## Phase 2d: dpkg Lock Blocking Ansible (RESOLVED)
|
||||
|
||||
### Problem
|
||||
|
||||
First Ansible playbook run failed on 5 of 6 nodes. Only Worker-01
|
||||
succeeded (it had been updated cleanly earlier). The other 5 failed
|
||||
at the pre_tasks stage with:
|
||||
|
||||
```
|
||||
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a'
|
||||
```
|
||||
|
||||
Root cause: Previous OS updates via `qm guest exec` left dpkg in a
|
||||
half-configured state on 5 nodes. The `openssh-server` package was
|
||||
mid-upgrade (held a config file conflict prompt that hung without a TTY).
|
||||
|
||||
### Fix Sequence
|
||||
|
||||
Had to fix each node individually via `qm guest exec` (could not use
|
||||
Ansible ad-hoc commands because Ansible itself triggers apt):
|
||||
|
||||
```bash
|
||||
# For each broken node, on its hosting PVE node:
|
||||
ssh root@PVE_NODE "qm guest exec VMID --timeout 120 -- bash -c \"
|
||||
fuser -k /var/lib/dpkg/lock-frontend 2>/dev/null;
|
||||
fuser -k /var/lib/dpkg/lock 2>/dev/null;
|
||||
fuser -k /var/cache/debconf/config.dat 2>/dev/null;
|
||||
sleep 2;
|
||||
rm -f /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock;
|
||||
DEBIAN_FRONTEND=noninteractive dpkg --configure -a &&
|
||||
apt-get install -y ceph-common rbd-nbd &&
|
||||
echo FIXED
|
||||
\""
|
||||
```
|
||||
|
||||
All 5 nodes fixed (CP-01, CP-02, CP-03, Worker-02, Worker-03). Then
|
||||
re-ran `ansible-playbook playbook.yml` successfully.
|
||||
|
||||
### Key Insight
|
||||
|
||||
The `lablabs.rke2` role's pre_tasks install `ceph-common` and `rbd-nbd`
|
||||
via apt. If any node has dpkg in an interrupted state, the ENTIRE
|
||||
playbook fails on that node. Must fix dpkg on ALL nodes before running
|
||||
the upgrade playbook. Use `DEBIAN_FRONTEND=noninteractive` to avoid
|
||||
config file conflict prompts hanging under `qm guest exec` (no TTY).
|
||||
|
||||
## Phase 2e: Upgrade Completion & Transient CP Failure (COMPLETED)
|
||||
|
||||
### Result
|
||||
|
||||
Second Ansible playbook run succeeded. All 6 nodes upgraded to
|
||||
v1.35.6+rke2r1. PLAY RECAP showed `failed=1` on CP-03 (VM129) — but
|
||||
investigation via `qm guest exec 129` confirmed the service was already
|
||||
active and running v1.35.6+rke2r1. The failure was a timing race:
|
||||
Ansible's `systemctl restart rke2-server` raced with RKE2's internal
|
||||
reload from the binary swap. The service had already started before
|
||||
Ansible tried to restart it, causing a transient "already activating"
|
||||
error.
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
kubectl get nodes -o wide
|
||||
# All 6 nodes: Ready, v1.35.6+rke2r1, containerd 2.2.5-k3s2
|
||||
kubectl get pods -A | grep -v Running | grep -v Completed
|
||||
# Empty — 0 broken pods
|
||||
```
|
||||
|
||||
### Lesson: Transient CP restart failures can be benign
|
||||
|
||||
If a CP node shows `failed=1` in the PLAY RECAP but is `Ready` and
|
||||
running the target version, the failure was a race condition between
|
||||
Ansible's `systemctl restart` and RKE2's internal reload during the
|
||||
binary swap. No remediation needed — verify cluster state with
|
||||
`kubectl get nodes` rather than blindly re-running the playbook.
|
||||
|
||||
## Remaining Work
|
||||
|
||||
1. ✅ OS Updates — DONE
|
||||
2. ✅ RKE2 Upgrade v1.35.2→v1.35.6 — COMPLETED (commit 6cb1b3c)
|
||||
3. ✅ Helm Chart Updates — COMPLETED (commits 36eb5c4, acd3231)
|
||||
See `references/helm-chart-upgrades-2026-07.md` for full detail.
|
||||
ArgoCD v2→v3, External Secrets, Ceph CSI RBD, CNPG, Velero all upgraded.
|
||||
4. ✅ openclaw-memory — REMOVED via GitOps (commit ebc4da5)
|
||||
5. ✅ ArgoCD `backups` app OutOfSync — ROOT CAUSE FOUND (cosmetic:
|
||||
ESO default fields in live spec not in Git YAML). Fix pending
|
||||
user approval (read-only investigation completed).
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **NEVER extract RKE2 tarball to `/`** — always use `/usr/local/`
|
||||
2. **Debian 12 usrmerge**: `/bin`, `/lib`, `/sbin` are symlinks, not
|
||||
real directories. Any tarball that creates `bin/` or `lib/` at root
|
||||
will destroy the system.
|
||||
3. **Ceph RBD offline mount** is viable for VM filesystem repair when
|
||||
guest agent is broken — stop VM, map RBD, mount partition, fix, unmount, start.
|
||||
4. **Always discover VM→PVE-Node mapping dynamically** — tofu state and
|
||||
actual placement diverge due to HA migrations.
|
||||
5. **Etcd quorum**: With 3 CP nodes, stopping 1 is safe (2/3 quorum).
|
||||
Never stop 2 simultaneously.
|
||||
6. **RKE2 version must match cached runtime images** — a newer binary
|
||||
without matching images will loop "activating" forever.
|
||||
7. **GitOps-first**: User expects cluster changes through IaC repo, not
|
||||
manual interventions. Manual SSH/qm-guest-exec should be last resort.
|
||||
8. **Non-operator app removal via GitOps** is straightforward: delete
|
||||
manifests → push → delete ArgoCD App (cascade) → delete namespace.
|
||||
No finalizer complications unlike operator-managed resources.
|
||||
9. **Ansible SSH prerequisite**: mgmt-runner (VM200) needs its SSH public
|
||||
key in `debian@VMs/.ssh/authorized_keys` before Ansible can reach K8s nodes.
|
||||
Generate keypair on VM200, distribute via `qm guest exec` to all 6 VMs,
|
||||
verify with `ansible all -m ping` before running the playbook.
|
||||
10. **GitOps-first principle reinforced**: User explicitly directed to use
|
||||
the IaC repo (`rke2_version` variable in Ansible playbook) for upgrades
|
||||
rather than manual operations. All cluster changes should go through
|
||||
`dominik/iac-homelab` on Gitea → commit → push → Ansible/ArgoCD.
|
||||
11. **dpkg interrupted state blocks Ansible**: If OS updates via `qm guest
|
||||
exec` leave dpkg half-configured (e.g. openssh-server mid-upgrade with
|
||||
a config file conflict prompt hanging without TTY), the Ansible
|
||||
playbook's pre_tasks (`apt install ceph-common rbd-nbd`) will fail on
|
||||
that node. Fix: `fuser -k` on dpkg AND debconf locks, then
|
||||
`DEBIAN_FRONTEND=noninteractive dpkg --configure -a` on each affected
|
||||
node BEFORE running the upgrade playbook.
|
||||
12. **debconf has its own lock**: Beyond `/var/lib/dpkg/lock-frontend` and
|
||||
`/var/lib/dpkg/lock`, debconf maintains `/var/cache/debconf/config.dat`.
|
||||
A stale process holding this lock causes `dpkg --configure -a` to fail
|
||||
even after clearing dpkg locks. Must `fuser -k` all three.
|
||||
13. **Transient CP restart failures can be benign**: Ansible's
|
||||
`systemctl restart rke2-server` can race with RKE2's internal reload
|
||||
during the binary swap. If a CP node shows `failed=1` in PLAY RECAP
|
||||
but is `Ready` and running the target version, the failure was a
|
||||
timing race — no remediation needed. Always verify with
|
||||
`kubectl get nodes -o wide` rather than blindly re-running the playbook.
|
||||
Reference in New Issue
Block a user