---
name: hindsight
description: "Class-level skill for the Hindsight memory system: PostgreSQL diagnostics, pgvector/NumPy fixes, embedding daemon configuration, noris provider setup, operation & best practices (auto-retain/recall tuning, three-tier memory strategy, dedup), meta-memory automation (monthly cleanup, nightly dream cycle, weekly insight digest cronjobs), compound-learning bridge, and direct REST API access on port 9177."
version: 2.2.0
tags: [hindsight, debugging, embedding, memory, postgresql, pgvector]
related_skills: [compound-learning]
---

## Overview

Hindsight is an **optional overlay** on top of Hermes Agent's built-in SQLite memory (`~/.hermes/state.db`). The actual persistent memory always lives in SQLite; Hindsight adds semantic search via embeddings but is NOT required for basic memory.

This umbrella skill unifies diagnostics for:
- **Hindsight Memory & PostgreSQL** — daemon startup, migration, GLIBC, CPU instruction issues
- **Embedding Daemon** — GGUF model failures, venv paths, API startup, fallback models

## Quick Status Check

```bash
# === K8s deployment (current) ===
# 1. Check API health via LoadBalancer
curl -s http://10.0.30.201:9177/health
# Expected: {"status":"healthy","database":"connected"}

# 2. Check pods (direct SSH to VM200 — no proxmox chain needed)
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl get pods -n hindsight"

# 3. Check Hermes plugin config
cat ~/.hermes/hindsight/config.json
# Should show: mode=local_external, api_url=http://10.0.30.201:9177

# 4. Test recall via REST API
curl -s -X POST http://10.0.30.201:9177/v1/default/banks/hermes/memories/recall \
  -H "Content-Type: application/json" \
  -d '{"query":"test","budget":"low"}'

# 5. Check the actual memory database (ALWAYS works, even without Hindsight)
sqlite3 ~/.hermes/state.db "SELECT id, source, title, started_at FROM sessions ORDER BY started_at;"

# === Legacy local daemon (deprecated, GLIBC-blocked) ===
# 6. Check if PostgreSQL is running (local)
ps aux | grep postgres | grep -v grep

# 7. Check Hindsight daemon log (local)
tail -30 ~/.hindsight/profiles/hermes.log

# 8. Check PostgreSQL instance JSON (local)
cat ~/.pg0/instances/hindsight-embed-hermes/instance.json

# 9. Verify daemon health (local — will fail on Debian 12 due to GLIBC 2.39)
curl -s http://127.0.0.1:9177/health
```

---

## Section 1: Hindsight Memory & PostgreSQL Diagnostics

### 1.1 Database Migration Failed (Daemon Won't Start)
If the Hindsight app can't migrate its tables into PostgreSQL, the built-in SQLite memory still works.

**Fix:**
1. Check PostgreSQL connectivity: `pg_isready -h localhost -p 5432`
2. Check `tail ~/.hindsight/profiles/hermes.log`
3. PostgreSQL instance: `~/.pg0/instances/hindsight-embed-hermes/`
4. If migration is stuck, drop and recreate the `hindsight` database

### 1.2 pgvector.so GLIBC Version Mismatch
**Error:** `could not load library vector.so: ... GLIBC_X.YY not found`

**Fix:** Build pgvector against the bundled PostgreSQL:
```bash
cd /tmp && rm -rf pgvector && git clone https://github.com/pgvector/pgvector.git && cd pgvector
make PG_CONFIG=/home/debian/.pg0/installation/18.1.0/bin/pg_config
cp vector.so ~/.pg0/installation/18.1.0/lib/vector.so
ldd ~/.pg0/installation/18.1.0/lib/vector.so | grep not || echo "Compatible"
```

### 1.3 NumPy AVX2 / CPU Instruction Mismatch (VM Environments)
**Error:** `RuntimeError: NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support: (X86_V2)`

**Fix:** Replace the UV-cached NumPy with a compatible build:
```bash
pip3 install --no-build-isolation --no-deps --target=/tmp/numpy_install numpy==1.26.4
UV_SITE=~/.cache/uv/archive-v0/$(ls ~/.cache/uv/ | head -1 | sed 's/archive-v0\///')/lib/python3.11/site-packages
rm -rf $UV_SITE/numpy $UV_SITE/numpy-*.dist-info $UV_SITE/numpy.libs
cp -a /tmp/numpy_install/numpy $UV_SITE/numpy
cp -a /tmp/numpy_install/numpy.libs $UV_SITE/numpy.libs
cp -a /tmp/numpy_install/numpy-1.26.4.dist-info $UV_SITE/numpy-1.26.4.dist-info
# Test: $UV_SITE/../../bin/python3 -c "import numpy; print(numpy.__version__)"
```

### 1.5 Hindsight Binary GLIBC Mismatch (Daemon Won't Start at All)
**Error:** `~/.local/bin/hindsight: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.39' not found`

The hindsight binary (ELF, dynamically linked) requires GLIBC ≥2.39 (Debian Trixie/Ubuntu 24.04) but the host runs Debian 12 Bookworm with GLIBC 2.36. The binary crashes immediately — no daemon, no port 9177, no auto-retain, no auto-recall.

**Symptom chain:** Daemon can't start → `hindsight_recall` returns "Failed to start daemon" → `hindsight_reflect` returns empty results → weekly digest cronjob produces "Keine signifikanten Patterns erkannt" with zero substance → PostgreSQL DB tables exist but all have 0 rows.

**Diagnosis:**
```bash
~/.local/bin/hindsight serve --port 9177
# Immediate crash with GLIBC_2.39 not found
ldd --version  # shows 2.36 on Debian 12
file ~/.local/bin/hindsight  # ELF 64-bit, dynamically linked
```

**Verification — check if DB is populated despite daemon failure:**
```bash
PG_BIN=~/.pg0/installation/18.1.0/bin
$PG_BIN/psql -h 127.0.0.1 -p 5432 -U postgres -d hindsight -c \
  "SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;"
# All zeros = daemon never ran, nothing was ever retained
```

**Workaround options** (none ideal on Debian 12 without Docker/podman):
1. Run hindsight inside a K8s pod (Debian Trixie base image) connecting to the local embedding PostgreSQL via NodePort
2. Compile hindsight from source against GLIBC 2.36
3. Upgrade host to Debian Trixie (invasive)
4. Use `proot` or `chroot` with a newer glibc

**Recommended: Deploy in K8s** — The `hindsight-api` Python package
(PyPI: `hindsight-api==0.8.4`) is the actual API server. The Rust
binary at `~/.local/bin/hindsight` is only a CLI client — it connects
to the API server, it doesn't serve one. Deploy `hindsight-api` as a
K8s Deployment with a pgvector-enabled Postgres StatefulSet. See
`references/k8s-deployment.md` for the full manifest template and
deployment walkthrough.

**Key K8s deployment pitfalls** (see `references/k8s-deployment.md`
for full detail):
- Use `python -m venv` in init container, NOT `pip install --target=`
  (target doesn't install console_scripts — binary will be missing)
- Install CPU-only torch first: `--index-url https://download.pytorch.org/whl/cpu`
  then main package with `--extra-index-url` (prevents 526MB CUDA download)
- Set `PGDATA=/var/lib/postgresql/data/pgdata` to avoid `lost+found`
  conflict on Ceph PVCs
- `kubectl rollout restart` does NOT apply manifest changes — always
  `kubectl apply -f` first, then restart if needed

**Key insight:** The weekly digest cronjob will happily produce empty output ("Keine signifikanten Patterns") when the daemon is down — it doesn't fail loudly. Always check daemon health first when digests are suspiciously empty.

**Key insight:** `hindsight-api` (Python, PyPI) ≠ `hindsight` (Rust
binary). The Python package is the server; the Rust binary is the CLI
client. For K8s deployment, only the Python package is needed — no
GLIBC concerns since it runs in a standard Python container.

### 1.6 Profile Not Configured
Use `hindsight-embed configure` to write the env file. See **Working Configuration** below for current values.

```bash
source /home/debian/hermes-agent/venv/bin/activate
NORIS_KEY=$(python3 -c "import yaml; print(yaml.safe_load(open('/home/debian/.hermes/config.yaml'))['providers']['noris']['api_key'])")
hindsight-embed configure --profile hermes --port 9177 \
  --env HINDSIGHT_API_LLM_PROVIDER=openai \
  --env HINDSIGHT_API_LLM_BASE_URL=https://ai.noris.de/v1 \
  --env HINDSIGHT_API_LLM_MODEL=vllm/release/gemma-4-31b-it \
  --env HINDSIGHT_API_LLM_API_KEY="$NORIS_KEY" \
  --env HINDSIGHT_API_LOG_LEVEL=info \
  --env HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai \
  --env HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL=https://ai.noris.de/v1 \
  --env HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=vllm/release/harrier-oss-v1-0.6b \
  --env HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY="$NORIS_KEY"
hindsight-embed daemon start --profile hermes
```

**Pitfall:** Editing `~/.hindsight/profiles/hermes.env` by hand does NOT reliably work — the daemon may cache old values. Always use `hindsight-embed configure` to regenerate the file, then kill any lingering daemon processes before restarting.

See also: `references/daemon-startup-troubleshooting.md` — error-to-fix diagnostic flowchart for common daemon startup failures.
See also: `references/k8s-deployment.md` — full K8s deployment guide (manifests, env vars, ExternalSecret format, verification).
See also: `references/k8s-data-migration.md` — migrating memory data from local embedded PostgreSQL to K8s Postgres (pg_dump → kubectl exec restore).

### 1.7 Switching to Remote/K8s API (local_external mode)

Once Hindsight is deployed in K8s (or any remote server), switch the Hermes
plugin from `local_embedded` to `local_external` mode in
`~/.hermes/hindsight/config.json`:

```json
{
  "mode": "local_external",
  "api_url": "http://10.0.30.201:9177",
  ...
}
```

- `local_external` skips `_check_local_runtime()` entirely — no GLIBC/binary
  dependency on the Hermes host. The plugin connects directly to the remote API.
- `api_url` can also be set via `HINDSIGHT_API_URL` env var.
- The plugin reads config at `__init__` time and caches `self._mode` +
  `self._api_url`. **Changing config.json mid-session has NO effect** —
  `hindsight_recall`/`hindsight_retain` will still try the old mode and fail
  with "Failed to start daemon for profile 'hermes'". A Hermes process restart
  is required for the plugin to pick up the new config.
- The REST API itself works immediately after config change — verify via curl
  while waiting for the restart window.

### Working Configuration (noris, verified 2026-07-12)
```bash
# ~/.hindsight/profiles/hermes.env  (local daemon mode)
# OR K8s deployment env vars        (K8s mode)
HINDSIGHT_API_LLM_PROVIDER=openai
HINDSIGHT_API_LLM_API_KEY=<noris key from config.yaml providers.noris.api_key>
HINDSIGHT_API_LLM_MODEL=vllm/release/gemma-4-31b-it
HINDSIGHT_API_LLM_BASE_URL=https://ai.noris.de/v1
HINDSIGHT_API_LOG_LEVEL=info
HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL=https://ai.noris.de/v1
HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=vllm/release/harrier-oss-v1-0.6b
HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY=<noris key>
```
- LLM (fact extraction): vllm/release/gemma-4-31b-it via noris
- Embeddings: vllm/release/harrier-oss-v1-0.6b (1024 dim) via noris
- CRITICAL: BOTH LLM and embedding model names must include `release/` prefix — `vllm/gemma-4-31b-it` (without `release/`) returns 400 "no keys found that support model". The `vllm/` prefix alone is insufficient. This applies to ALL noris vLLM models.
- CRITICAL: API key must be the real noris key from `config.yaml` (`providers.noris.api_key`), NOT `dummy`. `dummy` causes 401 AuthenticationError.
- CRITICAL: OpenAI SDK >=2.44.0 required (2.32.0 has parsing bug with noris `extra_fields` in response)
- UV cache: also upgrade openai in `/home/debian/.cache/uv/archive-v0/*/lib/python3.11/site-packages/`
- Startup takes ~30s (reranker model download + embedding init)

### 1.5 Memory Not Injected at Session Start
This is a built-in memory (`memory` tool → `state.db`) issue, NOT Hindsight. Check `auto_retain`/`auto_recall` in config, or use `session_search` to find missing memories.

---

## Section 2: Embedding Daemon Fixes

### 2.1 GGUF Model Fails with CPU Instruction Error (AVX512/AVX2)
When the embedding daemon fails to load a GGUF model due to missing CPU instructions.

**Diagnose:**
```bash
journalctl --user -u hindsight-daemon -n 50 --no-pager
cat ~/.hindsight/profiles/hermes.log
```

**Fix venv path:** The `daemon_embed_manager.py` uses 5 `.parent` traversals from the package to find `hindsight-api`. Verify:
```python
# From: .../site-packages/hindsight_embed/daemon_embed_manager.py
venv_api = Path(__file__).parent.parent.parent.parent.parent / "bin" / "hindsight-api"
```

**Start API manually with proper environment:**
```bash
KEY=$(sed -n 's/^HINDSIGHT_API_LLM_API_KEY=//p' ~/.hindsight/profiles/hermes.env)
cd /home/debian/hermes-agent && source venv/bin/activate
HINDSIGHT_API_LLM_PROVIDER=openai \
HINDSIGHT_API_LLM_BASE_URL=http://10.0.30.99:8080/v1 \
HINDSIGHT_API_LLM_MODEL=llama/model.gguf \
HINDSIGHT_API_LLM_API_KEY="$KEY" \
HINDSIGHT_API_LOG_LEVEL=info \
hindsight-api --port 8080
```

### 2.2 Fallback to Sentence-Transformers
If GGUF won't work, configure `hindsight-api` to use `BAAI/bge-small-en-v1.5`:
- Set `HINDSIGHT_API_EMBEDDINGS_PROVIDER=local`
- Model downloads automatically on first use

> **Important:** `hindsight-api` is NOT an OpenAI-compatible `/v1/embeddings` endpoint — it's a full Hindsight API server with its own endpoints. Per-profile daemons use port offsets (hermes profile = port 9177).

---

## Section 3: Operation & Best Practices

### 3.1 Auto-Behavior — What Fires When

Source: `plugins/memory/hindsight/__init__.py`

| Function | Auto? | Trigger | Mechanism |
|---|---|---|---|
| **retain** | yes (`auto_retain: true`) | after every turn | `sync_turn(user, assistant)` → background writer thread → fact extraction → embeddings → PostgreSQL |
| **recall** | yes (`auto_recall: true`) | before every turn | `queue_prefetch(query)` → background thread → semantic search → result injected as context |
| **reflect** | no | manual only | `hindsight_reflect` tool — synthesizes across ALL memories, expensive |

- `retain_every_n_turns` (default: 1) — controls frequency. Set to 2+ to reduce volume.
- `recall_prefetch_method` (default: `"recall"`) — can be set to `"reflect"` for deeper auto-prefetch, but NOT recommended (too expensive per-turn).
- `recall_budget` (default: `"mid"`) — `low`/`mid`/`high` controls how many memories are retrieved.

### 3.2 PITFALL: Duplicate Facts from Auto+Manual Retain

When `auto_retain: true` AND you also call `hindsight_retain` manually for the same facts, Hindsight stores duplicates (same fact 3-8×). This bloats recall results and wastes embedding compute.

**Mitigation:**
- Trust auto-retain for routine conversation facts.
- Use manual `hindsight_retain` ONLY for deliberately structured facts that auto-extraction might miss (explicit TODOs, credential references, architectural decisions).
- Consider `retain_every_n_turns: 2` to halve auto-retain volume.
- If duplicates accumulated, query PostgreSQL directly to identify and remove them.

### 3.3 Three-Tier Memory Strategy (MEMORY.md + Hindsight + docs/solutions/)

| Layer | Scope | Capacity | Retrieval |
|---|---|---|---|
| MEMORY.md | Hot facts, TODOs, quick refs | ~2,200 chars | Injected every turn (always visible) |
| Hindsight | Unlimited detail, history, context | Unbounded (PostgreSQL) | Semantic search on demand (auto-recall) |
| `docs/solutions/` | Structured problem-solution docs | Unbounded (files) | `search_files` (keyword) + `hindsight_recall` (semantic, via Phase 4.5 sync) |

Rules:
- Don't duplicate between layers — pick ONE layer per fact.
- Short pointers/summaries → MEMORY.md. Full detail → Hindsight.
- Structured problem-solutions (with root cause, fix, lesson, prevention) → `docs/solutions/` + Hindsight summary (compound-learning Phase 4.5).
- When MEMORY.md fills up (>90%), migrate detail entries to Hindsight, keep only 1-line cross-references.
- Recurring procedures/workflows → Skills (not Hindsight, not MEMORY.md, not docs/solutions/).
- Credentials → 1Password (never Hindsight, never docs/solutions/).

**Compound Learning ↔ Hindsight bridge:** The `compound-learning` skill (Phase 4.5) syncs a compact summary of each solution doc to Hindsight via `hindsight_retain`. Grounding Scans in `brainstorming`, `plan`, and `writing-plans` query `hindsight_recall` to find solutions semantically — even when keywords don't match. The weekly digest surfaces solution patterns automatically.

**Self-Improvement Loop:** compound-learning Phase 7 routes learnings that reveal system gaps to `skill_manage(action='patch')`, `memory`, or instruction-file patches. Each SI suggestion is tracked via `hindsight_retain` with tag `["self-improvement"]`. The nightly dream cycle detects learnings that contradict existing skills. The weekly digest audits open vs. acted-on SI items. This closes the loop: learn → document → improve the system → better next run.

### 3.4 recall vs reflect — When to Use Which

- **`recall`** — specific factual lookup ("ComfyUI IP?", "NUT UPS master address?"). Fast, returns ranked matches.
- **`reflect`** — synthesis/analysis across memories ("What was fixed today?", "How do these systems relate?"). Slower, reasons over all stored memories.
- Use `reflect` deliberately for complex questions, never reflexively.

---

## Section 4: Meta-Memory Automation (Layer 5)

Without automation, the memory stack rots: duplicates accumulate, stale facts linger, contradictions go unnoticed. Three cronjobs implement the meta-memory control plane.

See also: `scripts/hindsight_meta_memory.py`, `scripts/hindsight_nightly_dream.py`, `scripts/hindsight_weekly_digest.py`

### 4.1 Monthly Cleanup (`0 3 1 * *`)
- **Script**: `scripts/hindsight_meta_memory.py` (also at `~/.hermes/scripts/`)
- **No LLM needed** — pure Python + psql, 0 tokens
- 3 dedup rounds: exact text → normalized text → Jaccard similarity >0.65 (keep longest)
- Reports stale memories (>6 months, 0 accesses)
- **Silent when clean** (empty stdout = no message delivered)

### 4.2 Nightly Dream Cycle (`0 3 * * *`)
- **Script**: `scripts/hindsight_nightly_dream.py` (also at `~/.hermes/scripts/`)
- Uses Hindsight REST API `/reflect` endpoint (curl, no Python SDK needed)
- Checks: contradictions, blocked/stale TODOs, potentially outdated facts, **self-improvement suggestions** (learnings that contradict existing skills)
- Budget: `mid` for contradiction + SI check, `low` for others
- **Silent when nothing notable**

### 4.3 Weekly Insight Digest (`0 8 * * 0`)
- **Script**: `scripts/hindsight_weekly_digest.py` (also at `~/.hermes/scripts/`)
- Uses Hindsight REST API `/reflect` with curated prompts
- Delivers: recurring patterns, infra changes, open TODOs, recommendations, **solution patterns** (compound-learning), **self-improvement audit** (open vs. acted-on SI items)
- Budget: `high` for pattern detection, `mid` for others
- **Always delivers** (weekly digest is always relevant)

### 4.4 Creating Script-Only Cronjobs
Use `no_agent: true` and `script: <filename>` in the cronjob config. The script runs directly — no LLM call, no token cost. Output goes to the delivery channel. Empty stdout = no message sent.

---

## Section 5: Direct REST API Access

The Hindsight daemon exposes a REST API on port 9177. Scripts can call it directly via curl — no Python SDK, no LLM agent needed.

See also: `references/hindsight-api-endpoints.md` — full endpoint reference.

### Key Endpoints

```bash
# Health check
curl -s http://127.0.0.1:9177/health

# Reflect (synthesize across all memories)
curl -s -X POST http://127.0.0.1:9177/v1/default/banks/hermes/reflect \
  -H "Content-Type: application/json" \
  -d '{"query": "Welche Infrastrukturprobleme sind offen?", "budget": "mid"}'

# Recall (semantic search)
curl -s -X POST http://127.0.0.1:9177/v1/default/banks/hermes/memories/recall \
  -H "Content-Type: application/json" \
  -d '{"query": "Midea AC credentials", "budget": "low"}'

# List all memories
curl -s http://127.0.0.1:9177/v1/default/banks/hermes/memories/list

# Bank stats
curl -s http://127.0.0.1:9177/v1/default/banks/hermes/stats
```

### Direct PostgreSQL Access

```bash
PSQL=~/.pg0/installation/18.1.0/bin/psql
$PSQL -h 127.0.0.1 -p 5432 -U hindsight -d hindsight -c "SELECT COUNT(*) FROM memory_units WHERE bank_id='hermes';"
```

Key tables: `memory_units` (facts + embeddings), `entities`, `documents`, `chunks`.

### PITFALL: recall API returns a list, not an object
The `/memories/recall` endpoint returns a JSON array of results, not `{"text": "..."}` like `/reflect`. Handle both formats in scripts:
```python
data = json.loads(response)
if isinstance(data, list):
    text = "\n".join(item.get("text", str(item)) for item in data)
else:
    text = data.get("text", "")
```

---

## Architecture Reference

```
Built-in Memory (always works):
  ~/.hermes/state.db (SQLite)
  → Sessions, Messages, FTS index
  → Injected at session start via memory tool

Hindsight (K8s deployment — current, verified 2026-07-12):
  K8s namespace: hindsight
  ├─ hindsight-api (python:3.11-slim, venv + pip in initContainer)
  │  → REST API on :8888, LB at 10.0.30.201:9177
  │  → LLM: vllm/release/gemma-4-31b-it (fact extraction)
  │  → Embeddings: vllm/release/harrier-oss-v1-0.6b (via noris)
  └─ hindsight-postgres-0 (pgvector/pgvector:pg16)
     → PVC: ceph-hdd-replica, PGDATA=/var/lib/postgresql/data/pgdata
  ~/.hermes/hindsight/config.json (mode=local_external, api_url=...)
  → Semantic search + entity resolution
  → Plugin caches config at __init__; restart required after changes

Hindsight (legacy local daemon — deprecated, GLIBC 2.39 blocker):
  ~/.pg0/instances/hindsight-embed-hermes/ (PostgreSQL 18.1.0)
  ~/.hindsight/profiles/hermes.env (API keys)
  ~/.hindsight/profiles/hermes.log (daemon logs)
  → Superseded by K8s deployment; kept for reference only
```

## Key Files

See also: `references/config-options.md` — full config key reference from source code analysis.

| File | Purpose |
|---|---|
| `~/.hermes/state.db` | **Main database** — all sessions, messages, FTS |
| `~/.hermes/hindsight/config.json` | Hindsight config (LLM provider, bank, mode) |
| `~/.hindsight/profiles/hermes.env` | Hindsight API credentials |
| `~/.hindsight/profiles/hermes.log` | Hindsight daemon log |
| `~/.pg0/instances/hindsight-embed-hermes/` | PostgreSQL instance (data, config, logs) |
| `~/.pg0/instances/hindsight-embed-hermes/instance.json` | DB credentials |

## Session Summary Query

```bash
sqlite3 ~/.hermes/state.db "
  SELECT s.id, s.source, datetime(s.started_at, 'unixepoch', 'localtime') as started,
         s.title, COUNT(m.id) as messages
  FROM sessions s
  LEFT JOIN messages m ON m.session_id = s.id
  GROUP BY s.id
  ORDER BY s.started_at;
"
```
