Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
@@ -0,0 +1,95 @@
|
||||
# Hindsight K8s Deployment — Session Detail (2026-07-12)
|
||||
|
||||
## Context
|
||||
Hindsight daemon on Hermes host (Debian 12, GLIBC 2.36) couldn't start
|
||||
because the Rust binary needs GLIBC 2.39. Decision: deploy hindsight-api
|
||||
(Python PyPI package) as a K8s Deployment in the RKE2 cluster.
|
||||
|
||||
## What was done
|
||||
|
||||
1. Discovered `hindsight-api` is a Python package on PyPI (v0.8.4),
|
||||
NOT the Rust binary. The Rust binary is only the CLI client.
|
||||
2. Created K8s manifests:
|
||||
- Namespace: `hindsight`
|
||||
- StatefulSet: `hindsight-postgres` (image: `pgvector/pgvector:pg16`)
|
||||
- Deployment: `hindsight-api` (image: `python:3.11-slim`, venv + pip install)
|
||||
- Services, ExternalSecrets, ArgoCD Application
|
||||
3. Created 1Password items in vault "Kubernetes ESO":
|
||||
- `noris-api-key` (API Credential) → credential = noris API key
|
||||
- `hindsight-db-password` (Password) → password = random string
|
||||
4. Committed to `dominik/iac-homelab` repo, pushed to Gitea
|
||||
5. ArgoCD auto-synced the new Application
|
||||
|
||||
## Issues hit and fixes (chronological)
|
||||
|
||||
### Issue 1: ExternalSecret API version
|
||||
- **Error**: `"The Kubernetes API could not find version 'v1beta1'"`
|
||||
- **Fix**: Changed `apiVersion` from `external-secrets.io/v1beta1` to `external-secrets.io/v1`
|
||||
|
||||
### Issue 2: StorageClass name
|
||||
- **Error**: PVC stuck `Pending` with no PV created
|
||||
- **Fix**: Changed `ceph-rbd` → `ceph-hdd-replica`
|
||||
|
||||
### Issue 3: ExternalSecret key format
|
||||
- **Error**: `"secret reference has invalid format"`
|
||||
- **Fix**: Combined into `key: item-name/field-name` format, removed `property:` field
|
||||
|
||||
### Issue 4: 1Password vault mismatch
|
||||
- **Error**: `"no vault matched the secret reference query"`
|
||||
- **Cause**: Items created in vault "Hermes" via local `op` CLI, but K8s
|
||||
ClusterSecretStore uses vault "Kubernetes ESO" (`334ykdtj5kar3jlpcrztjvx2fu`)
|
||||
- **Fix**: Used K8s service account token to create items in correct vault:
|
||||
```bash
|
||||
K8S_TOKEN=$(kubectl get secret onepassword-token -n external-secrets \
|
||||
-o jsonpath='{.data.token}' | base64 -d)
|
||||
OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op item create \
|
||||
--category="API Credential" --title="noris-api-key" \
|
||||
--vault="334ykdtj5kar3jlpcrztjvx2fu" credential=VALUE
|
||||
```
|
||||
- **Key learning**: K8s service account token CAN create items (not read-only)
|
||||
|
||||
### Issue 5: K8s env var interpolation order
|
||||
- **Fix**: Moved `HINDSIGHT_DB_PASSWORD` to be the first env var (before `DATABASE_URL`)
|
||||
|
||||
### Issue 6: ArgoCD stale manifest cache
|
||||
- **Fix**: Delete Application and recreate, or force hard refresh via annotation
|
||||
|
||||
### Issue 7: StatefulSet PVC immutability
|
||||
- **Fix**: Delete STS + PVC manually, let ArgoCD recreate with corrected manifest
|
||||
|
||||
### Issue 8: PostgreSQL lost+found conflict on PVC
|
||||
- **Error**: Postgres CrashLoopBackOff — initdb refuses non-empty data directory
|
||||
- **Cause**: Ceph PVC has `lost+found` directory at root
|
||||
- **Fix**: Set `PGDATA=/var/lib/postgresql/data/pgdata` (subdirectory)
|
||||
|
||||
### Issue 9: pip --target doesn't install console_scripts
|
||||
- **Error**: Main container: `/bin/sh: 1: hindsight-api: not found`
|
||||
- **Cause**: `pip install --target=/opt/packages` copies modules but NOT bin/ scripts
|
||||
- **Fix**: Use `python -m venv /opt/venv` instead — creates full venv with bin/
|
||||
|
||||
### Issue 10: CUDA torch (526MB) causes OOMKill
|
||||
- **Error**: Init container OOMKilled during pip install
|
||||
- **Cause**: Default torch wheel includes CUDA libraries (526MB+)
|
||||
- **Fix**: Install CPU torch first with `--index-url https://download.pytorch.org/whl/cpu`,
|
||||
then main package with `--extra-index-url` for same CPU index
|
||||
|
||||
### Issue 11: kubectl rollout restart doesn't apply manifest changes
|
||||
- **Error**: After `rollout restart`, new pod still uses old spec
|
||||
- **Cause**: `rollout restart` only restarts with current deployment spec, doesn't pull new manifests
|
||||
- **Fix**: `kubectl apply -f <manifest>` first, THEN `rollout restart` if needed
|
||||
|
||||
### Issue 12: ArgoCD doesn't recreate manually deleted resources
|
||||
- **Error**: After `kubectl delete externalsecret`, ArgoCD reports "successfully synced"
|
||||
but doesn't recreate the ES
|
||||
- **Fix**: `kubectl apply -f <manifest>` manually
|
||||
|
||||
## State at session end
|
||||
- Postgres pod: 1/1 Running ✅
|
||||
- ExternalSecrets: Synced, Secrets created ✅
|
||||
- LB Service: 10.0.30.201:9177 assigned ✅
|
||||
- API pod: Init container pip install (venv + CPU torch) takes 5-10min
|
||||
- Latest commit `8424566` uses venv approach
|
||||
- Needs `kubectl apply -f` on VM200 to pick up the venv fix
|
||||
- Previous `rollout restart` didn't apply the manifest change
|
||||
- **Next step**: `kubectl apply -f clusters/main/hindsight/api-deployment.yaml` on VM200,
|
||||
wait ~10min for init container, verify pod becomes Ready, then update Hermes config
|
||||
Reference in New Issue
Block a user