# Hindsight K8s Deployment Reference

## Architecture

```
K8s Namespace: hindsight
├── StatefulSet: hindsight-postgres (pgvector/pgvector:pg16)
│   ├── PVC: 10Gi, storageClassName: ceph-hdd-replica
│   ├── PGDATA: /var/lib/postgresql/data/pgdata (subdirectory!)
│   └── Service: hindsight-postgres:5432 (ClusterIP)
├── Deployment: hindsight-api (python:3.11-slim)
│   ├── initContainer: python -m venv /opt/venv + pip install
│   ├── Command: /opt/venv/bin/hindsight-api --host 0.0.0.0 --port 8888
│   └── Service: hindsight-api:9177 → 8888 (LoadBalancer)
├── ExternalSecret: hindsight-noris-key ← 1Password "noris-api-key/credential"
└── ExternalSecret: hindsight-db ← 1Password "hindsight-db-password/password"
```

## Key Decisions

### Why K8s instead of local daemon?
- Local hindsight binary (Rust) requires GLIBC ≥2.39, host has 2.36
- `hindsight-api` is a Python package on PyPI — runs anywhere with Python 3.11+
- K8s cluster already has Ceph storage, External Secrets, Cilium LB

### Why pgvector/pgvector:pg16 image?
- Pre-built pgvector extension baked into the Postgres image
- No need to compile pgvector against a specific PG version
- PG16 matches the local instance version (18.1.0 also works but pg16 is stable)

### Why venv instead of pip --target?
- `pip install --target=/opt/packages` copies Python modules but does NOT
  create bin/ entry points (console_scripts). The `hindsight-api` binary
  will be missing and the main container fails with `command not found`.
- `python -m venv /opt/venv` creates a full virtual environment with
  bin/ scripts. The main container runs `/opt/venv/bin/hindsight-api`.

### Why CPU-only torch?
- `hindsight-api` depends on `sentence-transformers` which depends on `torch`
- Default torch wheel is 526MB+ with CUDA libraries — causes OOMKill in
  init containers with 2Gi memory limit
- CPU torch is ~192MB and sufficient for inference

### Env var ordering (CRITICAL)
K8s `$(VAR_NAME)` interpolation processes env vars top-to-bottom.
The referenced secret MUST appear before the var that uses it:
```yaml
env:
  - name: HINDSIGHT_DB_PASSWORD        # ← must be first
    valueFrom:
      secretKeyRef: ...
  - name: HINDSIGHT_API_DATABASE_URL   # ← references $(HINDSIGHT_DB_PASSWORD)
    value: "postgresql://hindsight:$(HINDSIGHT_DB_PASSWORD)@hindsight-postgres:5432/hindsight"
```

## ExternalSecret Format

1Password items for K8s ESO must be in vault "Kubernetes ESO"
(`334ykdtj5kar3jlpcrztjvx2fu`), NOT the local "Hermes" vault.

Key format: `item-title/field-name` (no `property` field, no `op://` prefix):
```yaml
remoteRef:
  key: noris-api-key/credential          # API Credential category
  key: hindsight-db-password/password    # Password category
```

The K8s service account token CAN create items (not read-only):
```bash
K8S_TOKEN=$(kubectl get secret onepassword-token -n external-secrets \
  -o jsonpath='{.data.token}' | base64 -d)
OP_SERVICE_ACCOUNT_TOKEN="$K8S_TOKEN" op item create \
  --category="API Credential" --title="noris-api-key" \
  --vault="334ykdtj5kar3jlpcrztjvx2fu" credential=VALUE
```

## Init Container Pattern (venv + CPU torch)

```yaml
initContainers:
  - name: install-deps
    image: python:3.11-slim
    command: ["/bin/sh", "-c"]
    args:
      - |
        python -m venv /opt/venv &&
        /opt/venv/bin/pip install --no-cache-dir \
          torch --index-url https://download.pytorch.org/whl/cpu &&
        /opt/venv/bin/pip install --no-cache-dir \
          hindsight-api==0.8.4 psycopg2-binary \
          --extra-index-url https://download.pytorch.org/whl/cpu &&
        echo "Dependencies installed successfully"
    volumeMounts:
      - name: venv
        mountPath: /opt/venv
containers:
  - name: hindsight-api
    image: python:3.11-slim
    command: ["/bin/sh", "-c"]
    args: ["/opt/venv/bin/hindsight-api --host 0.0.0.0 --port 8888"]
    volumeMounts:
      - name: venv
        mountPath: /opt/venv
        readOnly: true
volumes:
  - name: venv
    emptyDir:
      sizeLimit: 3Gi
```

**Timing**: The full pip install (torch CPU + hindsight-api + 100+ deps)
takes 5-10 minutes. The pod stays in `Init:0/1` during this time.
Don't confuse this with a crash — check init container logs.

## IaC Repo Layout

```
iac-homelab/
├── clusters/main/apps/hindsight.yaml          # ArgoCD Application (sync-wave: "2")
└── clusters/main/hindsight/
    ├── namespace.yaml
    ├── postgres.yaml                          # StatefulSet (pgvector/pgvector:pg16)
    │                                          #   PGDATA=/var/lib/postgresql/data/pgdata
    ├── postgres-service.yaml
    ├── api-deployment.yaml                     # hindsight-api (venv + pip in initContainer)
    ├── api-service.yaml                        # LoadBalancer :9177→:8888
    ├── external-secrets.yaml                  # noris-api-key
    └── external-secrets-db.yaml               # hindsight-db-password
```

## Deployment Steps

1. Create 1Password items in vault "Kubernetes ESO":
   - `noris-api-key` (API Credential) → credential = noris API key
   - `hindsight-db-password` (Password) → password = random 32-char string
2. Add manifests to iac-homelab repo, commit + push to Gitea
3. ArgoCD auto-discovers via App-of-Apps, creates namespace + resources
4. If ExternalSecrets fail with "no vault matched" → items in wrong vault
5. If PVC stuck Pending → wrong StorageClass (use `ceph-hdd-replica`)
6. If pods CreateContainerConfigError → ExternalSecrets not synced yet
7. If Postgres CrashLoopBackOff → missing PGDATA subdirectory (lost+found conflict)
8. If API "command not found" → used pip --target instead of venv
9. If API OOMKilled during init → torch CUDA version, use CPU index-url
10. Once pods Running, get LB IP: `kubectl get svc -n hindsight hindsight-api`
11. Update Hermes config.yaml: `hindsight.api_url: http://<LB-IP>:9177`

## Applying Manifest Changes

**CRITICAL**: `kubectl rollout restart` does NOT apply new manifest
changes — it only restarts pods with the current spec. After pushing
a manifest fix, always `kubectl apply -f <manifest>` first:
```bash
cd /root/iac-homelab && git pull
kubectl apply -f clusters/main/hindsight/api-deployment.yaml
# THEN restart if needed:
kubectl rollout restart deployment/hindsight-api -n hindsight
```

**ArgoCD does not recreate manually deleted resources**: If you
`kubectl delete externalsecret` to force re-sync, ArgoCD may not
recreate it. Apply the manifest manually with `kubectl apply -f`.

## Verification

```bash
# Health
curl http://<LB-IP>:9177/health

# Recall test
curl -X POST http://<LB-IP>:9177/v1/default/banks/hermes/memories/recall \
  -H "Content-Type: application/json" \
  -d '{"query": "test", "budget": "low"}'

# Check DB has data
kubectl exec -n hindsight hindsight-postgres-0 -- \
  psql -U hindsight -d hindsight -c \
  "SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;"
```

## Migrating Data from Local Embedded PostgreSQL

If migrating from a local `hindsight-embed` deployment (where the embedded
PostgreSQL at `~/.pg0/` has accumulated data), see
`references/k8s-data-migration.md` for the full dump-and-restore procedure.

**Key shortcut**: Direct SSH to VM200 at `10.0.30.124` eliminates the
complex `proxmox1 → proxmox3 → qm guest exec` chain. The dump can be
piped directly:
```bash
cat /tmp/hindsight_migration.sql | ssh -i ~/.ssh/id_ed25519_proxmox \
  root@10.0.30.124 "cat > /root/hindsight_migration.sql"
```

Then restore via `kubectl exec -i` with `DROP SCHEMA CASCADE` first.

## Hermes config.json Changes (Switching to K8s API)

After K8s deployment is running, update `~/.hermes/hindsight/config.json`
to point the Hermes plugin at the K8s service instead of the local daemon:

```json
{
  "mode": "local_external",
  "api_url": "http://10.0.30.201:9177",
  "bank_id": "hermes",
  "recall_budget": "mid",
  "auto_retain": true,
  "auto_recall": true,
  "retain_every_n_turns": 2,
  "memory_mode": "hybrid",
  "llm_provider": "openai_compatible",
  "llm_base_url": "https://ai.noris.de/v1",
  "llm_model": "vllm/release/gemma-4-31b-it"
}
```

Key changes from local_embedded mode:
- `"mode": "local_external"` — tells the plugin to connect to an external
  API server instead of starting a local daemon. Skips `_check_local_runtime()`
  entirely, so no GLIBC/binary dependency on the Hermes host.
- `"api_url": "http://<LB-IP>:9177"` — the K8s LoadBalancer endpoint.
  Falls back to `HINDSIGHT_API_URL` env var if not set in config.

**CRITICAL: Hermes restart required.** The plugin caches `self._mode` and
`self._api_url` at `__init__` time (line ~1258 and ~1283 of
`plugins/memory/hindsight/__init__.py`). Changing `config.json` mid-session
has NO effect — `hindsight_recall` and `hindsight_retain` will still try
to start the local daemon and fail with "Failed to start daemon for profile
'hermes'". The REST API itself works immediately (verify via curl); the
Hermes plugin tools only work after a process restart.

## K8s Port Env Var Collision

**PITFALL**: Kubernetes auto-injects `<SERVICE_NAME>_PORT` and
`<SERVICE_NAME>_PORT_<PORT>_TCP*` env vars from Services. When the Service
is named `hindsight-api` exposing port 9177, K8s injects:
```
HINDSIGHT_API_PORT_9177_TCP=tcp://10.43.160.208:9177
HINDSIGHT_API_PORT_9177_TCP_PORT=9177
HINDSIGHT_API_PORT_9177_TCP_ADDR=10.43.160.208
HINDSIGHT_API_SERVICE_HOST=10.43.160.208
HINDSIGHT_API_SERVICE_PORT=9177
```
The `hindsight-api` server reads `HINDSIGHT_API_PORT` to determine its
listen port. If K8s injects a conflicting value, the server may bind to
the wrong port or crash. **Fix**: Explicitly set `HINDSIGHT_API_PORT=8888`
in the container env vars (before any K8s-injected vars):
```yaml
env:
  - name: HINDSIGHT_API_PORT
    value: "8888"
```

## LLM Model Name Pitfall (noris)

**CRITICAL**: The LLM model name in the K8s deployment env vars MUST
include the `release/` prefix: `vllm/release/gemma-4-31b-it`, NOT
`vllm/gemma-4-31b-it`. Without `release/`, noris returns:
```json
{"error": {"error": "no keys found that support model: gemma-4-31b-it"}}
```
This affects the LLM (fact extraction) model, not just the embedding model.
The same pitfall applies to embeddings: `vllm/release/harrier-oss-v1-0.6b`.

## Retain Endpoint Format

The `/v1/default/banks/{bank_id}/memories` POST endpoint requires:
```json
{
  "items": [
    {
      "content": "fact text here",
      "context": "optional context",
      "tags": ["tag1", "tag2"]
    }
  ],
  "async": false
}
```
NOT a flat `{"content": "..."}` — the `items` array is required.

## Verification Checklist

```bash
# 1. Health
curl -s http://<LB-IP>:9177/health
# Expected: {"status":"healthy","database":"connected"}

# 2. Retain (test fact extraction + embeddings)
curl -s -X POST http://<LB-IP>:9177/v1/default/banks/hermes/memories \
  -H "Content-Type: application/json" \
  -d '{"items":[{"content":"test fact","context":"verification"}],"async":false}'
# Expected: {"success":true,"bank_id":"hermes","items_count":1,...}

# 3. Recall (semantic search)
curl -s -X POST http://<LB-IP>:9177/v1/default/banks/hermes/memories/recall \
  -H "Content-Type: application/json" \
  -d '{"query":"test fact","budget":"low"}'
# Expected: {"results":[{"id":"...","text":"test fact",...}]}

# 4. Reflect (synthesis)
curl -s -X POST http://<LB-IP>:9177/v1/default/banks/hermes/reflect \
  -H "Content-Type: application/json" \
  -d '{"query":"What was deployed?","budget":"mid"}'

# 5. Check DB has data
kubectl exec -n hindsight hindsight-postgres-0 -- \
  psql -U hindsight -d hindsight -c \
  "SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;"
```

## Known Issues (as of 2026-07-12)

- `hindsight-api` package pulls `claude-agent-sdk` (81MB) and `mlx`
  as dependencies even though they're unused in this deployment
- Init container pip install takes 5-10 minutes (100+ packages)
- `hindsight-api` entry point is `hindsight-api = hindsight_api.main:main`
  (defined in `hindsight_api_slim` package's `entry_points.txt`)
