Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,230 @@
+---
+name: optimize-loops
+description: "Run metric-driven optimization loops for measurable outcomes. Use when improving search relevance, clustering quality, build performance, prompt quality, or scored behavior through experiments."
+version: 1.0.0
+author: Hermes Agent (merged from Every Inc compound-engineering ce-optimize)
+license: MIT
+metadata:
+  hermes:
+    tags: [optimization, experimentation, metrics, iteration, compound-engineering]
+    related_skills: [plan, subagent-driven-development, requesting-code-review]
+---
+
+# Iterative Optimization Loops
+
+Run metric-driven iterative optimization. Define a goal, build measurement scaffolding, then run experiments that converge toward the best solution.
+
+**Core principle:** Don't guess what's better — measure it. Every optimization claim needs a number.
+
+## When to Use
+
+- Improving measurable outcomes (build time, latency, relevance scores)
+- Tuning prompts, configurations, or algorithms where quality is scored
+- Comparing approaches against a metric
+- Systematic A/B testing of implementations
+
+**Skip for:** one-off changes, subjective quality without a scoring method, unmeasurable goals.
+
+## The Process
+
+### Phase 0: Setup
+
+#### 0.1 Define the Optimization Target
+
+What are you optimizing? Choose the type:
+
+| Type | When to Use | Examples |
+|---|---|---|
+| **Hard metric** | Objective, scalar, clear "better" direction | Build time, latency, memory, test pass rate, bundle size |
+| **Judge metric** | Quality requires semantic judgment | Search relevance, clustering quality, summarization, UX copy |
+
+**If qualitative:** strongly recommend `judge` type. Hard metrics alone optimize proxy numbers without checking actual quality.
+
+Three-tier approach for judge metrics:
+1. **Degenerate gates** (hard, cheap): catch obviously broken solutions
+2. **LLM-as-judge** (actual target): sample outputs, score against rubric
+3. **Diagnostics** (logged): distribution stats for understanding
+
+#### 0.2 Create the Spec
+
+Define:
+- **Goal:** What to optimize and direction (minimize/maximize)
+- **Metric:** Exact measurement method
+- **Baseline:** Current performance (measure before optimizing!)
+- **Constraints:** Budget, time, dependencies
+- **Stopping criteria:** Max iterations, target threshold, diminishing returns
+
+Save the spec to `.context/optimize/<spec-name>/spec.yaml`.
+
+### Phase 1: Establish Baseline
+
+Measure the current state BEFORE any changes:
+
+```bash
+# Run the metric on the current implementation
+python scripts/measure_baseline.py
+```
+
+Record the baseline. This is your comparison point.
+
+**Checkpoint:** Write baseline to disk immediately. The conversation is NOT durable storage.
+
+### Phase 2: Generate Hypotheses
+
+Brainstorm 3-5 optimization approaches:
+
+1. [Approach A] — why it might help, predicted impact
+2. [Approach B] — why it might help, predicted impact
+3. [Approach C] — why it might help, predicted impact
+
+Rank by: expected impact × probability of working ÷ cost to test
+
+### Phase 3: Run Experiments
+
+For each hypothesis:
+
+#### 3.1 Implement the Change
+Make the change in an isolated branch or worktree.
+
+#### 3.2 Measure
+Run the metric against the changed implementation.
+
+#### 3.3 Record
+Write the result to disk IMMEDIATELY:
+
+```yaml
+# .context/optimize/<spec-name>/experiment-log.yaml
+- experiment: 1
+  hypothesis: "Cache repeated DB queries"
+  change: "Added Redis cache layer for getUserById"
+  metric_value: 245  # ms, was 380ms baseline
+  improvement: 35.5%
+  verdict: improved
+  timestamp: 2026-07-10T18:30:00
+```
+
+#### 3.4 Compare
+Did it improve over baseline? Over the best so far?
+
+### Phase 4: Iterate
+
+After each batch of experiments:
+
+1. Read the experiment log from disk (not from memory)
+2. Identify what worked and what didn't
+3. Generate new hypotheses informed by results
+4. Run the next batch
+
+**Persistence discipline:**
+- Write each result to disk IMMEDIATELY after measurement
+- Re-read from disk at every phase boundary
+- The experiment log is append-only during Phase 3
+- Never present results to the user without writing to disk first
+
+### Phase 5: Final Summary
+
+After stopping criteria are met:
+
+```markdown
+## Optimization Results
+
+**Goal:** Reduce API latency
+**Baseline:** 380ms average
+**Best result:** 165ms (57% improvement)
+
+### Winning Approach
+[Cached Redis layer for hot queries + connection pooling]
+
+### Experiment History
+| # | Approach | Result | Improvement |
+|---|---|---|---|
+| 1 | Redis cache | 245ms | 35.5% |
+| 2 | Connection pool | 310ms | 18.4% |
+| 3 | Cache + Pool | 165ms | 56.6% |
+| 4 | Query optimization | 290ms | 23.7% |
+
+### Recommendation
+Deploy approach #3 (cache + pool). Estimated impact: 57% latency reduction.
+```
+
+Save the final summary to `docs/solutions/optimization-YYYY-MM-DD-<topic>.md` and invoke `compound-learning` to capture insights (which will also sync to Hindsight via Phase 4.5).
+
+## Using delegate_task for Parallel Experiments
+
+For independent optimization approaches, run experiments in parallel:
+
+```python
+delegate_task(tasks=[
+    {
+        "goal": "Experiment A: Add Redis caching to getUserById. Measure latency.",
+        "context": "Baseline: 380ms. Implement caching, run benchmark, report result.",
+        "toolsets": ["terminal", "file"]
+    },
+    {
+        "goal": "Experiment B: Add connection pooling. Measure latency.",
+        "context": "Baseline: 380ms. Implement pooling, run benchmark, report result.",
+        "toolsets": ["terminal", "file"]
+    },
+])
+```
+
+## LLM-as-Judge Mode
+
+When the metric is qualitative, use a judge prompt:
+
+```python
+delegate_task(
+    goal="""You are a quality judge. Score the following outputs against the rubric.
+
+Rubric:
+- Relevance (0-10): How well does the result match the query intent?
+- Accuracy (0-10): Is the information correct?
+- Completeness (0-10): Does it cover the expected scope?
+
+Score each output independently. Return JSON:
+{
+  "outputs": [
+    {"id": 1, "relevance": 8, "accuracy": 9, "completeness": 7, "total": 24},
+    ...
+  ]
+}
+
+Outputs to score:
+[INSERT SAMPLED OUTPUTS]
+"""
+)
+```
+
+## Pitfalls
+
+- **Don't optimize without a baseline** — you can't know if you improved
+- **Don't trust in-memory state** — write to disk, re-read at boundaries
+- **Don't run too many iterations** — set stopping criteria upfront
+- **Don't use hard metrics for qualitative goals** — proxy optimization produces degenerate solutions
+- **Don't skip degenerate gates** — "all items in 1 cluster" can score well on a bad metric
+- **Don't batch results in memory** — write each result immediately after measurement
+
+## Integration with Other Skills
+
+- **plan** — create an optimization plan first for complex multi-step optimizations
+- **subagent-driven-development** — use for implementing each experiment
+- **compound-learning** — capture optimization insights after completion
+- **requesting-code-review** — review the winning implementation before deploying
+
+## Hermes Agent Integration
+
+- `terminal` — run measurements and benchmarks
+- `write_file` / `read_file` — persistence (experiment log, spec)
+- `delegate_task` — parallel experiments and LLM-as-judge
+- `search_files` — find existing optimization patterns
+
+## Remember
+
+```
+Measure before optimizing
+Write results to disk immediately
+Baseline → Hypothesize → Experiment → Record → Iterate
+Use judge metrics for qualitative goals
+Stop when diminishing returns
+Capture the learning
+```