# Hindsight Data Migration: Local Embedded PostgreSQL → K8s

## When to Use This

Migrating memory data from the local embedded PostgreSQL (used by the
deprecated local daemon) to the K8s PostgreSQL StatefulSet. The local
daemon can't run on Debian 12 (GLIBC 2.39 requirement), but its database
may contain months of accumulated memories, entities, and links.

## Prerequisites

- Local embedded PostgreSQL running at `~/.pg0/instances/hindsight-embed-hermes/`
  (check: `ps aux | grep postgres | grep pg0`)
- Portable PostgreSQL binaries at `~/.pg0/installation/18.1.0/bin/`
- K8s hindsight-postgres-0 pod running and healthy
- Direct SSH access to VM200 (mgmt-runner): `ssh root@10.0.30.124`

## Step-by-Step

### 1. Verify the Source Database Has Data

```bash
PG_BIN=~/.pg0/installation/18.1.0/bin
PGPASSWORD=hindsight $PG_BIN/psql -h 127.0.0.1 -p 5432 -U hindsight -d hindsight -c "
  SELECT 'memory_units' as tbl, count(*) as cnt FROM memory_units
  UNION ALL SELECT 'entities', count(*) FROM entities
  UNION ALL SELECT 'documents', count(*) FROM documents
  UNION ALL SELECT 'memory_links', count(*) FROM memory_links;
"
```

If the local PostgreSQL isn't running but the data dir exists, start it:
```bash
# Remove stale PID file if needed
rm ~/.pg0/instances/hindsight-embed-hermes/data/postmaster.pid
# Clear stale shared memory if needed
ipcrm -m $(ipcs -m | awk '/debian/{print $2; exit}')
# Start
PGDATA=~/.pg0/instances/hindsight-embed-hermes/data
$PG_BIN/pg_ctl -D "$PGDATA" -l /tmp/pg_old.log start
```

### 2. Dump the Database

```bash
PGPASSWORD=hindsight ~/.pg0/installation/18.1.0/bin/pg_dump \
  -h 127.0.0.1 -p 5432 -U hindsight -d hindsight \
  --format=plain --no-owner --no-privileges \
  -f /tmp/hindsight_migration.sql
```

Expected size: ~13MB for 200 memory_units + 131 entities + 281 documents +
3996 memory_links. Embeddings (1024-dim vectors) are included in the dump.

### 3. Transfer the Dump to VM200

**Direct SSH works** — no need for the complex proxmox1 → proxmox3 → qm
guest exec chain:

```bash
cat /tmp/hindsight_migration.sql | ssh -i ~/.ssh/id_ed25519_proxmox \
  root@10.0.30.124 "cat > /root/hindsight_migration.sql && wc -c /root/hindsight_migration.sql"
```

### 4. Restore to K8s PostgreSQL

SSH to VM200 and pipe the dump through kubectl exec:

```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 '
  export KUBECONFIG=/root/.kube/config

  # Drop existing schema (fresh K8s DB has empty tables from Alembic migrations)
  kubectl exec -i hindsight-postgres-0 -n hindsight -- psql -U hindsight -d hindsight << "SQL"
  DROP SCHEMA public CASCADE;
  CREATE SCHEMA public;
  SQL

  # Restore from dump
  cat /root/hindsight_migration.sql | kubectl exec -i hindsight-postgres-0 -n hindsight \
    -- psql -U hindsight -d hindsight 2>&1 | tail -20
'
```

**Why DROP SCHEMA CASCADE?** The K8s Postgres already ran Alembic migrations
on first boot, creating tables and indexes. Restoring a full dump on top
causes duplicate-object errors. Dropping the schema first ensures a clean
restore. The dump recreates all tables, indexes, extensions (pgvector,
pg_trgm), and the HNSW vector index.

### 5. Verify Row Counts

```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 '
  export KUBECONFIG=/root/.kube/config
  kubectl exec hindsight-postgres-0 -n hindsight -- psql -U hindsight -d hindsight -c "
    SELECT '"'"'memory_units'"'"' as tbl, count(*) as cnt FROM memory_units
    UNION ALL SELECT '"'"'entities'"'"', count(*) FROM entities
    UNION ALL SELECT '"'"'documents'"'"', count(*) FROM documents
    UNION ALL SELECT '"'"'memory_links'"'"', count(*) FROM memory_links;
  "
'
```

### 6. Verify Embeddings Survived

```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 '
  export KUBECONFIG=/root/.kube/config
  kubectl exec hindsight-postgres-0 -n hindsight -- psql -U hindsight -d hindsight -c "
    SELECT count(*) as total, count(embedding) as has_embedding
    FROM memory_units;
  "
'
```

All embeddings should be non-null. Vector dimensions should be 1024
(harrier-oss-v1-0.6b model).

### 7. Restart the API Pod

The API pod's connection pool may hold stale connections to the pre-restore
database. Restart to pick up the restored data:

```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 '
  export KUBECONFIG=/root/.kube/config
  kubectl rollout restart deployment/hindsight-api -n hindsight
'
```

Note: Init container re-runs (~10 min for venv + pip install) since emptyDir
is ephemeral. The old pod stays Running until the new one is Ready.

### 8. Verify Recall Works

```bash
curl -s -X POST http://10.0.30.201:9177/v1/default/banks/hermes/memories/recall \
  -H "Content-Type: application/json" \
  -d '{"query": "infrastructure proxmox kubernetes", "budget": "low"}' \
  | python3 -c "
    import sys, json
    data = json.load(sys.stdin)
    results = data.get('results', [])
    print(f'Results: {len(results)}')
    for r in results[:3]:
        scores = r.get('scores', {})
        print(f'  final={scores.get(\"final\", 0):.3f} semantic={scores.get(\"semantic\", 0):.3f} | {r.get(\"text\", \"\")[:100]}')
  "
```

## Recall API Response Format

The `/memories/recall` endpoint returns results with a `scores` **dict**,
not a scalar `score`:

```json
{
  "results": [
    {
      "id": "uuid",
      "text": "memory text...",
      "type": "experience",
      "entities": ["entity1", "entity2"],
      "occurred_start": "2026-06-27T...",
      "tags": ["tag1"],
      "scores": {
        "final": 0.918,
        "reranker": 0.824,
        "semantic": 0.585,
        "keyword": 0.100
      }
    }
  ]
}
```

Access scores via `result["scores"]["final"]`, not `result["score"]`.

## Pitfalls

### Stale postmaster.pid
If the local PostgreSQL was killed uncleanly, `postmaster.pid` remains and
blocks restart. Delete it: `rm ~/.pg0/instances/hindsight-embed-hermes/data/postmaster.pid`

### Stale Shared Memory Segment
Shared memory segments may persist after the process dies. Check with
`ipcs -m` and remove with `ipcrm -m <shmid>`.

### qm guest exec Quoting Nightmare
Piping data through `ssh → proxmox1 → proxmox3 → qm guest exec 200` involves
4 levels of quote nesting and is nearly impossible for binary data. Use
**direct SSH to VM200** at `10.0.30.124` instead — it's on the same subnet
and accepts the same SSH key.

### HTTP Server Firewall Blocked
Starting a Python HTTP server on the Hermes host (10.0.30.99) to transfer
the dump does NOT work — connections from VM200 are refused despite the
server binding to 0.0.0.0. Use SSH pipe instead (step 3 above).
