hermes-skills/devops/rke2-cluster-administration/references/hindsight-k8s-deployment-2026-07.md

Hindsight K8s Deployment — Session Detail (2026-07-12)

Context

Hindsight daemon on Hermes host (Debian 12, GLIBC 2.36) couldn't start because the Rust binary needs GLIBC 2.39. Decision: deploy hindsight-api (Python PyPI package) as a K8s Deployment in the RKE2 cluster.

What was done

1. Discovered hindsight-api is a Python package on PyPI (v0.8.4), NOT the Rust binary. The Rust binary is only the CLI client.
2. Created K8s manifests:
   • Namespace: hindsight
   • StatefulSet: hindsight-postgres (image: pgvector/pgvector:pg16)
   • Deployment: hindsight-api (image: python:3.11-slim, venv + pip install)
   • Services, ExternalSecrets, ArgoCD Application
3. Created 1Password items in vault "Kubernetes ESO":
   • noris-api-key (API Credential) → credential = noris API key
   • hindsight-db-password (Password) → password = random string
4. Committed to dominik/iac-homelab repo, pushed to Gitea
5. ArgoCD auto-synced the new Application

Issues hit and fixes (chronological)

Issue 1: ExternalSecret API version

• Error: "The Kubernetes API could not find version 'v1beta1'"
• Fix: Changed apiVersion from external-secrets.io/v1beta1 to external-secrets.io/v1

Issue 2: StorageClass name

• Error: PVC stuck Pending with no PV created
• Fix: Changed ceph-rbd → ceph-hdd-replica

Issue 3: ExternalSecret key format

• Error: "secret reference has invalid format"
• Fix: Combined into key: item-name/field-name format, removed property: field

Issue 4: 1Password vault mismatch

• Error: "no vault matched the secret reference query"
• Cause: Items created in vault "Hermes" via local op CLI, but K8s ClusterSecretStore uses vault "Kubernetes ESO" (334ykdtj5kar3jlpcrztjvx2fu)
• Fix: Used K8s service account token to create items in correct vault:

  K8S_TOKEN=$(kubectl get secret onepassword-token -n external-secrets \
    -o jsonpath='{.data.token}' | base64 -d)
  OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op item create \
    --category="API Credential" --title="noris-api-key" \
    --vault="334ykdtj5kar3jlpcrztjvx2fu" credential=VALUE
  

• Key learning: K8s service account token CAN create items (not read-only)

Issue 5: K8s env var interpolation order

• Fix: Moved HINDSIGHT_DB_PASSWORD to be the first env var (before DATABASE_URL)

Issue 6: ArgoCD stale manifest cache

• Fix: Delete Application and recreate, or force hard refresh via annotation

Issue 7: StatefulSet PVC immutability

• Fix: Delete STS + PVC manually, let ArgoCD recreate with corrected manifest

Issue 8: PostgreSQL lost+found conflict on PVC

• Error: Postgres CrashLoopBackOff — initdb refuses non-empty data directory
• Cause: Ceph PVC has lost+found directory at root
• Fix: Set PGDATA=/var/lib/postgresql/data/pgdata (subdirectory)

Issue 9: pip --target doesn't install console_scripts

• Error: Main container: /bin/sh: 1: hindsight-api: not found
• Cause: pip install --target=/opt/packages copies modules but NOT bin/ scripts
• Fix: Use python -m venv /opt/venv instead — creates full venv with bin/

Issue 10: CUDA torch (526MB) causes OOMKill

• Error: Init container OOMKilled during pip install
• Cause: Default torch wheel includes CUDA libraries (526MB+)
• Fix: Install CPU torch first with --index-url https://download.pytorch.org/whl/cpu, then main package with --extra-index-url for same CPU index

Issue 11: kubectl rollout restart doesn't apply manifest changes

• Error: After rollout restart, new pod still uses old spec
• Cause: rollout restart only restarts with current deployment spec, doesn't pull new manifests
• Fix: kubectl apply -f <manifest> first, THEN rollout restart if needed

Issue 12: ArgoCD doesn't recreate manually deleted resources

• Error: After kubectl delete externalsecret, ArgoCD reports "successfully synced" but doesn't recreate the ES
• Fix: kubectl apply -f <manifest> manually

State at session end

• Postgres pod: 1/1 Running ✅
• ExternalSecrets: Synced, Secrets created ✅
• LB Service: 10.0.30.201:9177 assigned ✅
• API pod: Init container pip install (venv + CPU torch) takes 5-10min
  • Latest commit 8424566 uses venv approach
  • Needs kubectl apply -f on VM200 to pick up the venv fix
  • Previous rollout restart didn't apply the manifest change
• Next step: kubectl apply -f clusters/main/hindsight/api-deployment.yaml on VM200, wait ~10min for init container, verify pod becomes Ready, then update Hermes config