# Self-Improvement Loop Architecture

The closed-loop system connecting compound-learning, Hindsight, and skill/memory self-improvement.

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                     PROBLEM SOLVED                          │
└──────────────────────────┬──────────────────────────────────┘
                           ▼
┌──────────────────────────────────────────────────────────────┐
│  compound-learning                                          │
│  ├─ Phase 1-3: Identify, classify, overlap-check             │
│  ├─ Phase 4: Write solution doc → docs/solutions/{type}/     │
│  ├─ Phase 4.5: Hindsight sync (hindsight_retain)             │
│  │   └─ Summary + tags + source path → PostgreSQL            │
│  ├─ Phase 5-6: CONCEPTS.md, discoverability                  │
│  └─ Phase 7: Self-Improvement Routing                        │
│      ├─ Skill gap? → skill_manage(patch) [ASK USER]          │
│      ├─ Memory gap? → memory(add)                            │
│      ├─ Instruction file gap? → patch(file) [ASK USER]       │
│      └─ Track: hindsight_retain(tags=["self-improvement"])   │
└──────────────────────────┬──────────────────────────────────┘
                           ▼
┌──────────────────────────────────────────────────────────────┐
│  GROUNDING SCANS (next session)                              │
│  ├─ brainstorming Step 1.5: search_files + hindsight_recall   │
│  ├─ plan Step 1.5: search_files + hindsight_recall            │
│  └─ writing-plans Step 1.5: search_files + hindsight_recall   │
│      └─ Semantic search finds solutions even when keywords    │
│         don't match (e.g. "Galera bootstrap" → "SST Recovery")│
└──────────────────────────┬──────────────────────────────────┘
                           ▼
┌──────────────────────────────────────────────────────────────┐
│  META-MEMORY AUTOMATION (cronjobs)                           │
│                                                              │
│  Nightly Dream (0 3 * * *)                                  │
│  ├─ Contradiction detection                                  │
│  ├─ Blocked/stale TODOs                                      │
│  ├─ Outdated facts                                           │
│  └─ Self-improvement: learnings vs. existing skills           │
│                                                              │
│  Weekly Digest (0 8 * * 0)                                   │
│  ├─ Recurring patterns                                       │
│  ├─ Infra changes                                            │
│  ├─ Open TODOs                                               │
│  ├─ Recommendations                                          │
│  ├─ Solution patterns (compound-learning)                    │
│  └─ Self-improvement audit (open vs. acted-on)               │
│                                                              │
│  Monthly Cleanup (0 3 1 * *)                                 │
│  └─ Dedup + decay report                                     │
└──────────────────────────────────────────────────────────────┘
```

## Three-Tier Memory Strategy

| Layer | Scope | Capacity | Retrieval |
|---|---|---|---|
| MEMORY.md | Hot facts, TODOs, quick refs | ~2,200 chars | Injected every turn (always visible) |
| Hindsight (PostgreSQL) | Unlimited detail, history, context | Unbounded | Semantic search (auto-recall) |
| `docs/solutions/` | Structured problem-solution docs | Unbounded (files) | `search_files` (keyword) + `hindsight_recall` (semantic via Phase 4.5) |

**Routing rule:** One fact, one layer. Don't duplicate.
- Short pointers → MEMORY.md
- Full detail → Hindsight
- Structured solutions (root cause + fix + lesson + prevention) → `docs/solutions/` + Hindsight summary
- Recurring procedures → Skills
- Credentials → 1Password

## Phase 4.5: Hindsight Sync

After writing a solution doc, sync a compact summary to Hindsight:

```python
hindsight_retain(
    content=f"SOLUTION [{problem_type}]: {problem_summary}. "
            f"Root cause: {root_cause}. Fix: {fix}. "
            f"Lesson: {lesson}. Tags: {tags}. "
            f"Source: docs/solutions/{type}/{date}-{slug}.md",
    tags=["solution"] + tags
)
```

Graceful degradation: if Hindsight daemon is down (`curl -s http://127.0.0.1:9177/health` fails),
the file in `docs/solutions/` is sufficient on its own.

## Phase 7: Self-Improvement Routing

Decision tree for when a learning reveals a system gap:

| Learning implies... | Action | Tool | Mode |
|---|---|---|---|
| Skill has wrong/missing/outdated step | Patch skill | `skill_manage(action='patch')` | Ask user |
| No skill for recurring workflow | Propose new skill | `skill_manage(action='create')` | Ask user |
| Durable environment fact | Update MEMORY.md | `memory` tool | Just do it |
| Tool quirk/workaround | Update MEMORY.md | `memory` tool | Just do it |
| Convention change (AGENTS.md/CLAUDE.md) | Patch instruction file | `patch` tool | Ask user |

Tracking: every SI suggestion is stored via `hindsight_retain(tags=["self-improvement"])` so the
weekly digest can audit what was acted on vs. what's still open.

## Files Modified (Integration Points)

| File | Change |
|---|---|
| `compound-learning/SKILL.md` | Phase 4.5 (Hindsight sync), Phase 7 (SI routing), updated integrations + frontmatter |
| `brainstorming/SKILL.md` | Step 1.5 Grounding Scan: + `hindsight_recall` |
| `plan/SKILL.md` | Step 1.5 Grounding Scan: + `hindsight_recall` |
| `writing-plans/SKILL.md` | Step 1.5 Grounding Scan: + `hindsight_recall` |
| `optimize-loops/SKILL.md` | Phase 5: note about Phase 4.5 Hindsight sync |
| `hindsight/SKILL.md` | Three-tier strategy, SI loop documentation, cronjob descriptions updated |
| `hindsight/scripts/hindsight_nightly_dream.py` | Check #4: self-improvement suggestions |
| `hindsight/scripts/hindsight_weekly_digest.py` | Section #5: solution patterns, #6: SI audit |