# Daemon Startup Troubleshooting Flowchart

Diagnostic sequence for when `hindsight-embed daemon start --profile hermes` fails.
Work top-to-bottom — each error has a distinct cause and fix.

## Quick Diagnosis

```bash
# 1. Check if daemon is already running
ps aux | grep hindsight-embed | grep -v grep

# 2. Check the log (most recent error at the bottom)
tail -40 ~/.hindsight/profiles/hermes.log

# 3. Check port
ss -tlnp | grep 9177

# 4. Check env file
cat ~/.hindsight/profiles/hermes.env
```

## Error → Cause → Fix Matrix

### Error 1: `RuntimeError: Cannot change embedding dimension from 1024 to 384`

**Cause:** The daemon is loading a different embedding model than what was used to
create the existing embeddings in PostgreSQL. The DB has 1024-dim vectors (from
`vllm/release/harrier-oss-v1-0.6b`), but the daemon fell back to a 384-dim default
model because it couldn't resolve the configured model.

**Root cause is usually one of the errors below** (401 or 400) causing a silent
fallback to the default embedding model. Fix the underlying auth/model error first.

**Last resort** (if you genuinely want to switch models and discard old embeddings):
```sql
-- Connect to PostgreSQL and wipe embeddings
DELETE FROM public.memory_units;
-- Then restart the daemon
```

### Error 2: `openai.AuthenticationError: 401 — virtual_key_required`

**Cause:** The API key in `hermes.env` is `dummy` or missing.

**Fix:**
```bash
# Get the real key from Hermes config
NORIS_KEY=$(python3 -c "import yaml; print(yaml.safe_load(open('/home/debian/.hermes/config.yaml'))['providers']['noris']['api_key'])")

# Reconfigure with the real key
hindsight-embed configure --profile hermes --port 9177 \
  --env HINDSIGHT_API_LLM_PROVIDER=openai \
  --env HINDSIGHT_API_LLM_BASE_URL=https://ai.noris.de/v1 \
  --env HINDSIGHT_API_LLM_MODEL=vllm/gemma-4-31b-it \
  --env HINDSIGHT_API_LLM_API_KEY="$NORIS_KEY" \
  --env HINDSIGHT_API_LOG_LEVEL=info \
  --env HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai \
  --env HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL=https://ai.noris.de/v1 \
  --env HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=vllm/release/harrier-oss-v1-0.6b \
  --env HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY="$NORIS_KEY"
```

**Pitfall:** Hand-editing `hermes.env` with `write_file` or `echo` may not take effect
because the daemon process may cache old env values. Always use `hindsight-embed configure`.

### Error 3: `BadRequestError: 400 — no keys found that support model: harrier-oss-v1-0.6b`

**Cause:** The model name is missing the `release/` prefix. noris vLLM routes models
by namespace — `vllm/harrier-oss-v1-0.6b` resolves to a different (non-functional)
endpoint than `vllm/release/harrier-oss-v1-0.6b`.

**Fix:** Change the embedding model to `vllm/release/harrier-oss-v1-0.6b` (with the
`release/` prefix). Use `hindsight-embed configure` (see Error 2 fix above).

**Verification:**
```bash
cd /home/debian/hermes-agent && source venv/bin/activate
python3 -c "
from openai import OpenAI
env = dict(l.strip().split('=',1) for l in open('/home/debian/.hindsight/profiles/hermes.env') if '=' in l)
c = OpenAI(base_url=env['HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL'], api_key=env['HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY'])
r = c.embeddings.create(model='vllm/release/harrier-oss-v1-0.6b', input='test')
print(f'OK dim={len(r.data[0].embedding)}')
"
```

### Error 4: `openai.BadRequestError` with `extra_fields` parsing error

**Cause:** OpenAI SDK version < 2.44.0 can't parse noris response format.

**Fix:**
```bash
cd /home/debian/hermes-agent && source venv/bin/activate
pip install --upgrade openai>=2.44.0
# Also update UV cache copies
find ~/.cache/uv/ -path "*/site-packages/openai" -exec rm -rf {} + 2>/dev/null
pip install --upgrade openai>=2.44.0
```

### Error 5: `could not load library vector.so: GLIBC_X.YY not found`

**Cause:** pgvector compiled against newer GLIBC than the system has.

**Fix:** See SKILL.md Section 1.2 — rebuild pgvector against bundled PostgreSQL.

### Error 6: `NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support`

**Cause:** NumPy binary incompatible with VM CPU instructions.

**Fix:** See SKILL.md Section 1.3 — replace UV-cached NumPy with compatible build.

## Restart Procedure

After any fix, always:

```bash
# 1. Kill ALL lingering daemon processes
pkill -f hindsight-embed 2>/dev/null
pkill -f hindsight-api 2>/dev/null
sleep 2

# 2. Verify no processes remain
ps aux | grep hindsight | grep -v grep | grep -v postgres

# 3. Start fresh (background, daemon mode)
cd /home/debian/hermes-agent && source venv/bin/activate
hindsight-embed daemon start --profile hermes &

# 4. Wait ~30s (reranker download + embedding init)
sleep 30

# 5. Verify health
curl -s http://127.0.0.1:9177/health
# Expected: {"status":"healthy","database":"connected"}
```

## Startup Timeline

The daemon takes ~30 seconds to fully start:

| Seconds | What happens |
|---|---|
| 0-5 | Process spawns, reads env, connects to PostgreSQL |
| 5-15 | Downloads/loads local reranker model (`cross-encoder/ms-marco-MiniLM-L-6-v2`) |
| 15-25 | Initializes embedding client (test call to noris) |
| 25-30 | Uvicorn binds port 9177, worker poller starts |

**Pitfall:** Checking health before 30s gives a false negative. The process is alive
(state `S` = sleeping) but hasn't bound the port yet.

## Model Availability Check

Before configuring, verify which embedding models are available on noris:

```bash
cd /home/debian/hermes-agent && source venv/bin/activate
python3 -c "
from openai import OpenAI
cfg = __import__('yaml').safe_load(open('/home/debian/.hermes/config.yaml'))
c = OpenAI(base_url='https://ai.noris.de/v1', api_key=cfg['providers']['noris']['api_key'])
for m in c.models.list().data:
    if any(k in m.id.lower() for k in ['embed','harrier','bge','e5']):
        print(m.id)
"
```

Known working models (as of 2026-07-10):
- `vllm/release/harrier-oss-v1-0.6b` — 1024 dim ✅
- `vllm/release/qwen3-embedding-8b` — 4096 dim ✅ (would require DB wipe — different dim)
- `vllm/harrier-oss-v1-0.6b` — ❌ 400 error (missing `release/` prefix)
