Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
@@ -0,0 +1,229 @@
|
||||
# Helm Chart Upgrades — 2026-07-12
|
||||
|
||||
## Goal
|
||||
|
||||
Upgrade all Helm-managed charts on the RKE2 cluster after the v1.35.6
|
||||
upgrade. 5 components needed updates; RKE2-bundled charts were already
|
||||
current.
|
||||
|
||||
## Version Matrix (Final)
|
||||
|
||||
| Component | Old Version | New Version | Method | Status |
|
||||
|-----------|------------|-------------|--------|--------|
|
||||
| External Secrets | 2.2.0 | 2.7.0 | Helm direct | ✅ |
|
||||
| CNPG | 0.22.1 (app 1.24.1) | 0.29.0 (app 1.30.0) | ArgoCD GitOps | ✅ |
|
||||
| Velero | 8.1.0 (app 1.15.0) | 12.1.0 (app 1.18.1) | ArgoCD GitOps | ✅ |
|
||||
| Ceph CSI RBD | 3.10.1 | 3.17.0 | Helm direct | ✅ |
|
||||
| ArgoCD | 7.8.13 (app v2.14.7) | 10.1.3 (app v3.4.5) | Helm direct | ✅ |
|
||||
| Cilium | rke2-cilium-1.19.402 | — | RKE2-bundled | Already current |
|
||||
| Traefik | rke2-traefik-40.1.003 | — | RKE2-bundled | Already current |
|
||||
| CoreDNS | rke2-coredns-1.46.002 | — | RKE2-bundled | Already current |
|
||||
| Metrics Server | rke2-metrics-server-3.13.100 | — | RKE2-bundled | Already current |
|
||||
| Snapshot Ctrl | rke2-snapshot-controller-4.2.006 | — | RKE2-bundled | Already current |
|
||||
|
||||
## Upgrade Order (Safest → Riskiest)
|
||||
|
||||
1. External Secrets 2.2.0 → 2.7.0 (Minor, Helm direct, minimal values)
|
||||
2. CNPG 0.22.1 → 0.29.0 (Major, ArgoCD GitOps, **required replica fix first**)
|
||||
3. Velero 8.1.0 → 12.1.0 (Major, ArgoCD GitOps)
|
||||
4. Ceph CSI RBD 3.10.1 → 3.17.0 (Major, Helm direct, **causes transient API outage**)
|
||||
5. ArgoCD v2.14.7 → v3.4.5 (MAJOR v2→v3, Helm direct, **breaking change**)
|
||||
|
||||
## Detail: External Secrets (2.2.0 → 2.7.0)
|
||||
|
||||
Straightforward `helm upgrade --reuse-values`. Values were minimal
|
||||
(`installCRDs: true`). No breaking changes. All ExternalSecrets
|
||||
continued syncing. Two pre-existing broken ExternalSecrets
|
||||
(argocd-repo-credentials, postgres-main-db) were already broken for
|
||||
110+ days — unrelated to the upgrade.
|
||||
|
||||
## Detail: CNPG (0.22.1 → 0.29.0) — Replica Rebuild Required
|
||||
|
||||
### Pre-Existing Issue Found
|
||||
|
||||
`postgres-main-2` replica was broken for 109 days:
|
||||
```
|
||||
requested WAL segment 000000020000000000000008 has already been removed
|
||||
WAL file not found in the recovery object store
|
||||
```
|
||||
Primary had recycled WAL segments the replica still needed. Cluster
|
||||
was running with 2/3 Ready (primary + 1 replica), backups operational.
|
||||
|
||||
### Replica Rebuild
|
||||
|
||||
```bash
|
||||
# Delete broken replica PVCs + pod
|
||||
kubectl delete pvc postgres-main-2 postgres-main-2-wal -n postgres
|
||||
kubectl delete pod postgres-main-2 -n postgres
|
||||
|
||||
# CNPG automatically created postgres-main-4 as replacement
|
||||
# Wait for 3/3 Ready:
|
||||
kubectl get pods -n postgres -w
|
||||
```
|
||||
|
||||
### Operator Upgrade via GitOps
|
||||
|
||||
```bash
|
||||
# Changed targetRevision in ArgoCD Application manifest
|
||||
cd /root/iac-homelab
|
||||
sed -i 's/targetRevision: 0.22.1/targetRevision: 0.29.0/' \
|
||||
clusters/main/operators/cloudnativepg.yaml
|
||||
git add -A && git commit -m "feat: upgrade CNPG 0.22.1 → 0.29.0" && git push
|
||||
```
|
||||
|
||||
**Pitfall**: ArgoCD didn't auto-detect the chart version change despite
|
||||
`selfHeal: true`. Required hard refresh:
|
||||
```bash
|
||||
kubectl annotate application cloudnativepg-operator -n argocd \
|
||||
argocd.argoproj.io/refresh=hard --overwrite
|
||||
```
|
||||
|
||||
After refresh, ArgoCD pulled new chart, created new operator pod with
|
||||
image v1.30.0. Postgres cluster briefly went 2/3 during reconciliation
|
||||
(operator triggered primary restart), then recovered to 3/3 Healthy.
|
||||
|
||||
## Detail: Velero (8.1.0 → 12.1.0)
|
||||
|
||||
Same GitOps pattern as CNPG — changed `targetRevision` in ArgoCD
|
||||
Application manifest, pushed, hard-refreshed ArgoCD. All Velero
|
||||
components rolled out: server + 6 node-agents (one per K8s node).
|
||||
BSL (Backup Storage Location) remained Available throughout. Backup
|
||||
schedules (daily-full-cluster, weekly-full-cluster) continued working.
|
||||
|
||||
## Detail: Ceph CSI RBD (3.10.1 → 3.17.0) — API Server Disruption
|
||||
|
||||
### Upgrade
|
||||
|
||||
```bash
|
||||
helm upgrade ceph-csi-rbd cephcsi/ceph-csi-rbd \
|
||||
--version 3.17.0 -n kube-system --reuse-values --timeout 120s
|
||||
```
|
||||
|
||||
### Transient API Server Outage
|
||||
|
||||
The nodeplugin DaemonSet rolling update disrupted kubelet on CP nodes.
|
||||
All 3 API servers became unreachable:
|
||||
```
|
||||
The connection to the server 10.0.30.51:6443 was refused
|
||||
```
|
||||
|
||||
Recovery took ~5 minutes. Monitored with:
|
||||
```bash
|
||||
curl -sk --connect-timeout 5 https://10.0.30.51:6443/healthz
|
||||
# 401 Unauthorized = server is UP (expected without credentials)
|
||||
# Connection refused = server is DOWN
|
||||
```
|
||||
|
||||
After recovery, provisioner pods (2 replicas) were 7/7 Running with
|
||||
image `quay.io/cephcsi/cephcsi:v3.17.0`. One nodeplugin pod stayed in
|
||||
`ContainerCreating` for several minutes (image pull delay on CP-01).
|
||||
All 7 PVCs remained bound throughout — no data impact.
|
||||
|
||||
### Root Cause
|
||||
|
||||
The Ceph CSI nodeplugin DaemonSet runs on EVERY node including CP
|
||||
nodes. Rolling the DaemonSet causes the CSI driver socket to be
|
||||
briefly unavailable, which can block kubelet operations on CP nodes,
|
||||
temporarily preventing the API server from serving requests.
|
||||
|
||||
## Detail: ArgoCD v2.14.7 → v3.4.5 (MAJOR Upgrade)
|
||||
|
||||
### Safe Path: Two-Step Upgrade
|
||||
|
||||
Step 1: Patch to latest v2.x (low risk):
|
||||
```bash
|
||||
helm upgrade argocd argo/argo-cd --version 7.9.1 -n argocd --reuse-values
|
||||
# v2.14.7 → v2.14.11
|
||||
```
|
||||
|
||||
Step 2: Major v2→v3 upgrade with explicit values:
|
||||
```bash
|
||||
# v3 chart requires redis.networkPolicy.create field (breaking change)
|
||||
# --reuse-values FAILS with:
|
||||
# nil pointer evaluating interface {}.create
|
||||
# Must provide explicit values file with redis.networkPolicy.create: false
|
||||
|
||||
helm upgrade argocd argo/argo-cd --version 10.1.3 -n argocd \
|
||||
-f /tmp/argocd-v3-values.yaml --timeout 300s
|
||||
```
|
||||
|
||||
### v3 Values File
|
||||
|
||||
Key addition vs v2 values:
|
||||
```yaml
|
||||
redis:
|
||||
networkPolicy:
|
||||
create: false
|
||||
```
|
||||
|
||||
All other values carried over unchanged (server.insecure, ingress,
|
||||
replicas, affinity).
|
||||
|
||||
### Post-Upgrade Verification
|
||||
|
||||
- All ArgoCD pods Running (server ×2, repo-server ×2,
|
||||
application-controller, applicationset-controller, dex-server,
|
||||
notifications-controller, redis)
|
||||
- Image: `quay.io/argoproj/argocd:v3.4.5`
|
||||
- All applications Synced (except `backups` — pre-existing
|
||||
ExternalSecret OutOfSync, see §13.7)
|
||||
|
||||
## Detail: ArgoCD `backups` App OutOfSync (Task 4)
|
||||
|
||||
### Root Cause
|
||||
|
||||
Single resource OutOfSync: `ExternalSecret/velero-s3-credentials` in
|
||||
`velero` namespace. ESO adds default fields to the live spec that
|
||||
aren't in the Git YAML:
|
||||
- `conversionStrategy: "Default"`
|
||||
- `decodingStrategy: "None"`
|
||||
- `metadataPolicy: "None"`
|
||||
- `deletionPolicy: "Retain"`
|
||||
- `engineVersion: "v2"`
|
||||
- `mergePolicy: "Replace"`
|
||||
|
||||
ArgoCD's `ignoreDifferences` only covers `/status`, not `/spec`.
|
||||
|
||||
### Impact
|
||||
|
||||
Cosmetic — ExternalSecret functions correctly (`secret synced`,
|
||||
Velero backups operational). No data or functionality impact.
|
||||
|
||||
### Fix (Not Yet Applied — Read-Only Investigation)
|
||||
|
||||
Add ESO default fields to Git YAML at
|
||||
`clusters/main/backups/velero/external-secret.yaml` so spec matches
|
||||
exactly. Alternative: expand `ignoreDifferences` to cover specific
|
||||
JSON pointers (less clean).
|
||||
|
||||
## GitOps Trail
|
||||
|
||||
Commits pushed to `dominik/iac-homelab` on Gitea:
|
||||
- `36eb5c4` — `feat(helm): upgrade all managed charts` (values files +
|
||||
ArgoCD Application targetRevision changes)
|
||||
- `acd3231` — `chore: remove .kube cache from repo, add to .gitignore`
|
||||
(cleanup of accidentally committed `.kube/cache/` directory)
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Always check CNPG cluster health before upgrading the operator** —
|
||||
a broken replica (WAL gap) should be rebuilt first (delete PVC + pod,
|
||||
CNPG auto-creates fresh replica from base backup).
|
||||
2. **ArgoCD GitOps chart bumps need hard refresh** — changing
|
||||
`targetRevision` in the Application manifest and pushing doesn't
|
||||
guarantee ArgoCD pulls the new chart. Annotate with
|
||||
`argocd.argoproj.io/refresh=hard` to force.
|
||||
3. **Ceph CSI RBD DaemonSet upgrades disrupt API servers** — the
|
||||
nodeplugin DaemonSet rolls across ALL nodes including CP nodes,
|
||||
briefly disrupting kubelet and making API servers unreachable for
|
||||
2-5 minutes. Schedule during maintenance windows.
|
||||
4. **ArgoCD v2→v3 chart requires redis.networkPolicy.create** —
|
||||
`--reuse-values` fails with nil pointer error. Must provide
|
||||
explicit values file with `redis.networkPolicy.create: false`.
|
||||
5. **ExternalSecret perpetual OutOfSync** — ESO adds default fields to
|
||||
live spec not present in Git YAML. Fix by adding those fields to the
|
||||
Git source YAML.
|
||||
6. **`.kube/cache` gets committed by `git add -A`** — always add
|
||||
`.kube/` to `.gitignore` before running kubectl from a repo root.
|
||||
7. **Save Helm values to IaC repo** — `helm get values` → save to
|
||||
`epic-2-k8s/helm/<chart>/values.yaml` for reproducible installs.
|
||||
Reference in New Issue
Block a user