# Hindsight K8s Deployment — Session Detail (2026-07-12)

## Context
Hindsight daemon on Hermes host (Debian 12, GLIBC 2.36) couldn't start
because the Rust binary needs GLIBC 2.39. Decision: deploy hindsight-api
(Python PyPI package) as a K8s Deployment in the RKE2 cluster.

## What was done

1. Discovered `hindsight-api` is a Python package on PyPI (v0.8.4),
   NOT the Rust binary. The Rust binary is only the CLI client.
2. Created K8s manifests:
   - Namespace: `hindsight`
   - StatefulSet: `hindsight-postgres` (image: `pgvector/pgvector:pg16`)
   - Deployment: `hindsight-api` (image: `python:3.11-slim`, venv + pip install)
   - Services, ExternalSecrets, ArgoCD Application
3. Created 1Password items in vault "Kubernetes ESO":
   - `noris-api-key` (API Credential) → credential = noris API key
   - `hindsight-db-password` (Password) → password = random string
4. Committed to `dominik/iac-homelab` repo, pushed to Gitea
5. ArgoCD auto-synced the new Application

## Issues hit and fixes (chronological)

### Issue 1: ExternalSecret API version
- **Error**: `"The Kubernetes API could not find version 'v1beta1'"`
- **Fix**: Changed `apiVersion` from `external-secrets.io/v1beta1` to `external-secrets.io/v1`

### Issue 2: StorageClass name
- **Error**: PVC stuck `Pending` with no PV created
- **Fix**: Changed `ceph-rbd` → `ceph-hdd-replica`

### Issue 3: ExternalSecret key format
- **Error**: `"secret reference has invalid format"`
- **Fix**: Combined into `key: item-name/field-name` format, removed `property:` field

### Issue 4: 1Password vault mismatch
- **Error**: `"no vault matched the secret reference query"`
- **Cause**: Items created in vault "Hermes" via local `op` CLI, but K8s
  ClusterSecretStore uses vault "Kubernetes ESO" (`334ykdtj5kar3jlpcrztjvx2fu`)
- **Fix**: Used K8s service account token to create items in correct vault:
  ```bash
  K8S_TOKEN=$(kubectl get secret onepassword-token -n external-secrets \
    -o jsonpath='{.data.token}' | base64 -d)
  OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op item create \
    --category="API Credential" --title="noris-api-key" \
    --vault="334ykdtj5kar3jlpcrztjvx2fu" credential=VALUE
  ```
- **Key learning**: K8s service account token CAN create items (not read-only)

### Issue 5: K8s env var interpolation order
- **Fix**: Moved `HINDSIGHT_DB_PASSWORD` to be the first env var (before `DATABASE_URL`)

### Issue 6: ArgoCD stale manifest cache
- **Fix**: Delete Application and recreate, or force hard refresh via annotation

### Issue 7: StatefulSet PVC immutability
- **Fix**: Delete STS + PVC manually, let ArgoCD recreate with corrected manifest

### Issue 8: PostgreSQL lost+found conflict on PVC
- **Error**: Postgres CrashLoopBackOff — initdb refuses non-empty data directory
- **Cause**: Ceph PVC has `lost+found` directory at root
- **Fix**: Set `PGDATA=/var/lib/postgresql/data/pgdata` (subdirectory)

### Issue 9: pip --target doesn't install console_scripts
- **Error**: Main container: `/bin/sh: 1: hindsight-api: not found`
- **Cause**: `pip install --target=/opt/packages` copies modules but NOT bin/ scripts
- **Fix**: Use `python -m venv /opt/venv` instead — creates full venv with bin/

### Issue 10: CUDA torch (526MB) causes OOMKill
- **Error**: Init container OOMKilled during pip install
- **Cause**: Default torch wheel includes CUDA libraries (526MB+)
- **Fix**: Install CPU torch first with `--index-url https://download.pytorch.org/whl/cpu`,
  then main package with `--extra-index-url` for same CPU index

### Issue 11: kubectl rollout restart doesn't apply manifest changes
- **Error**: After `rollout restart`, new pod still uses old spec
- **Cause**: `rollout restart` only restarts with current deployment spec, doesn't pull new manifests
- **Fix**: `kubectl apply -f <manifest>` first, THEN `rollout restart` if needed

### Issue 12: ArgoCD doesn't recreate manually deleted resources
- **Error**: After `kubectl delete externalsecret`, ArgoCD reports "successfully synced"
  but doesn't recreate the ES
- **Fix**: `kubectl apply -f <manifest>` manually

## State at session end
- Postgres pod: 1/1 Running ✅
- ExternalSecrets: Synced, Secrets created ✅
- LB Service: 10.0.30.201:9177 assigned ✅
- API pod: Init container pip install (venv + CPU torch) takes 5-10min
  - Latest commit `8424566` uses venv approach
  - Needs `kubectl apply -f` on VM200 to pick up the venv fix
  - Previous `rollout restart` didn't apply the manifest change
- **Next step**: `kubectl apply -f clusters/main/hindsight/api-deployment.yaml` on VM200,
  wait ~10min for init container, verify pod becomes Ready, then update Hermes config