Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,98 @@
+# Capacity Curve Analysis – When to Stop Scaling Concurrency
+
+## The Core Question
+
+"Shall we test conc=200?" The answer depends on extrapolating the existing capacity curve, not on gut feeling.
+
+## Extrapolation Rule
+
+Given TTFT p99 values at conc=10, 50, 100, estimate conc=200 using a **linear degradation model** (the simplest that fits observed data):
+
+```
+TTFT_p99(conc) ≈ base + k × conc
+
+Where k is derived from the steepest interval:
+k = (TTFT_p99(conc=100) - TTFT_p99(conc=50)) / (100 - 50)
+```
+
+If the extrapolated TTFT p99 at conc=200 exceeds a **user-acceptable threshold** (typically 5000ms = 5s for interactive applications, 10000ms = 10s for batch), conc=200 is **operationally irrelevant**.
+
+## Worked Example (from this session)
+
+### gpt-oss-120b
+| Conc | TTFT p99 |
+|------|----------|
+| 10 | 978 ms |
+| 25 | 1924 ms |
+| 50 | 3575 ms |
+| 100 | 7310 ms |
+
+Extrapolation to conc=200:
+- k = (7310 - 3575) / 50 = **74.7 ms/conc**
+- TTFT_p99(200) = 7310 + 74.7 × 100 = **14,780 ms**
+
+**Verdict: Unusable. Users won't wait 15 seconds for the first token.**
+
+### gemma-4-31b-it
+| Conc | TTFT p99 |
+|------|----------|
+| 10 | 381 ms |
+| 50 | 1042 ms |
+| 100 | 2060 ms |
+
+Extrapolation:
+- k = (2060 - 1042) / 50 = **20.4 ms/conc**
+- TTFT_p99(200) = 2060 + 20.4 × 100 = **4,100 ms**
+
+**Verdict: Borderline. 4.1s is acceptable for some batch use cases, unacceptable for chat.**
+
+### moonshotai-kimi-k2.6
+| Conc | TTFT p99 |
+|------|----------|
+| 10 | 695 ms |
+| 25 | 1178 ms |
+| 50 | 1989 ms |
+| 100 | 3926 ms |
+
+Extrapolation:
+- k = (3926 - 1989) / 50 = **38.7 ms/conc**
+- TTFT_p99(200) = 3926 + 38.7 × 100 = **7,796 ms**
+
+**Verdict: Unusable for interactive use.**
+
+## Alternative to Higher Concurrency: Longer Inputs
+
+When conc=100 is already the practical ceiling, the more valuable test is **long-context prefill stress**:
+
+| Test | What it measures | When to use |
+|------|-----------------|-------------|
+| conc=10→100 | Queueing / scheduling efficiency | Find max parallel users |
+| ISL=512 → 1024 → 2048 → 4096 | Memory bandwidth / KV-cache limits | RAG, document analysis, long-context agents |
+
+A model that degrades gracefully at conc=100 but collapses at ISL=2048 has a **memory-bandwidth bottleneck**, not a scheduling bottleneck. The fix is faster memory (HBM3e), not more GPU compute.
+
+## Breakpoint Detection Heuristic
+
+Formally define the "breakpoint" as the highest concurrency where **all three** conditions hold:
+
+1. TTFT p99 < 5000 ms (5 seconds)
+2. ITL p99 < 200 ms (readable streaming, ~5 tok/s per user)
+3. Output throughput still scales linearly with concurrency (no saturation)
+
+From observed data:
+
+| Model | Breakpoint (conc) | Limiting factor |
+|-------|-------------------|-----------------|
+| gemma-4-31b-it | ~100 | TTFT ≈ 2s still OK |
+| moonshotai-kimi-k2.6 | ~75 | TTFT ≈ 3.5s approaching limit |
+| gpt-oss-120b | ~40 | TTFT ≈ 3.5s at conc=50 |
+| qwen3.6-27b-nvfp4 | ~40 | TTFT ≈ 3s at conc=50 |
+
+## Operational Recommendation
+
+- Set **hard concurrency limits** at the breakpoint, not at the hardware maximum
+- Implement **queueing + load balancing** beyond the breakpoint
+- Test **LLM-as-a-Service tiers**:
+  - Tier 1 (Premium): conc ≤ 10, guaranteed < 1s TTFT
+  - Tier 2 (Standard): conc ≤ 50, guaranteed < 3s TTFT
+  - Tier 3 (Batch): conc ≤ 100, best-effort, queueing OK