Initial commit: Hermes Agent Skills collection

mlops/llm-inference-benchmarking/references/aiperf-long-input-benchmarking.md

@@ -0,0 +1,109 @@
+# AIPerf Long-Input and Extreme-Context Benchmarking
+
+Session: 2026-06-18 — Full suite benchmark of 4 models (gemma-4-31b-it, gpt-oss-120b, moonshotai/kimi-k2.6, qwen3.6-27b-nvfp4) on ai.noris.de
+
+## Key Finding: Long Inputs > High Concurrency for Bottleneck Isolation
+
+When evaluating maximum node capacity, testing conc=200+ is often operationally irrelevant because TTFT p99 exceeds acceptable thresholds (7s+ at conc=100). The real infrastructure bottleneck for RAG / long-context workloads is **memory-bandwidth-bound prefill** with long inputs.
+
+Recommended alternative: **Long-Input Prefill Test**
+- Concurrency: Fixed low (e.g., 10) to avoid queueing artifacts
+- Input length: 1024 / 2048 / 4096 tokens (double/triple the standard 512)
+- Output length: Short (128) to isolate prefill from generation
+- Duration: 300s to amortize cold-start
+
+## Extreme Context Test (10k Input / 10k Output)
+
+For stress-testing context-window limits and sustained generation capacity:
+
+- **Input:** 10,000 tokens
+- **Output:** 10,000 tokens
+- **Concurrency:** 1 (sequentially, because runtime is 200–1000 seconds per request)
+- **Requests:** 10 per model
+- **Purpose:** Validates the full KV-cache lifecycle and sustained decode performance
+
+**Expected runtime:** At 10–50 tok/s output, one request takes 200–1000 seconds. Schedule accordingly.
+
+**Template:**
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/MODEL_NAME"
+  endpoint:
+    urls: ["https://ai.noris.de/v1"]
+    type: "chat"
+    streaming: true
+  datasets:
+    - type: "synthetic"
+      name: "main"
+      entries: 10
+      isl:
+        type: "fixed"
+        value: 10000
+      osl:
+        type: "fixed"
+        value: 10000
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      requests: 10
+```
+
+## Correct AIPerf 0.10.0 Synthetic Dataset Config (Long Input)
+
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+  endpoint:
+    urls: ["https://ai.noris.de/v1"]
+    type: "chat"
+    streaming: true
+  datasets:
+    - name: "synthetic_long"
+      type: "synthetic"
+      prompt_template:
+        content: "Analyze the following topic in depth and provide detailed insights."
+      strategy:
+        type: "fixed"
+        max_total_tokens: 1152
+        input_tokens: 1024
+        output_tokens: 128
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      duration: 300
+```
+
+**Critical YAML structure rules learned:**
+- `datasets` MUST be a **list** (`- name: ...`), NOT a dict with `default:` key
+- `name` goes directly under the list item, NOT nested under a `synthetic:` key
+- `prompt_template` and `strategy` are siblings to `name` and `type`
+
+## Benchmark Results at conc=100 (Standard 512-in / 128-out)
+
+| Model | Input tok/s (total) | Output tok/s (total) | Total tok/s | TTFT p99 |
+|-------|--------------------|---------------------|-------------|----------|
+| moonshotai-kimi-k2.6 | 5,115 | 1,270 | 6,385 | 3,926ms |
+| gemma-4-31b-it | 4,385 | 1,083 | 5,468 | 2,060ms |
+| qwen3.6-27b-nvfp4 | 4,191 | 992 | 5,183 | 7,391ms |
+| gpt-oss-120b | 3,300 | 804 | 4,104 | 7,310ms |
+
+**Prefill degradation @ conc=100:** All models lose ~80% of per-user prefill speed vs conc=10.
+
+## MSP Service-Tier Perspective (User Preference)
+
+When presenting for infrastructure/commercial decisions, structure per **service tier** not model comparison:
+- Single-User Experience (conc=1): TTFT p99, Output/User, SLA class
+- Multi-Tenant Load (conc=10→100): Throughput scaling, degradation curve
+- Steady-State: Long-run stability, drift detection
+- Service Recommendation: Positioning (Entry/Premium), suitable use cases
+
+## Related Files
+- `templates/aiperf-long-input-config.yaml` — Working starter template
+- `references/aiperf-suite-orchestrator-pattern.md` — Full orchestration pattern
+- `references/aiperf-token-metrics-explained.md` — Token metric unit reference