Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,342 @@
+---
+name: llm-inference-benchmarking
+description: "Benchmark LLM inference endpoints for throughput, latency, and capacity. Covers TTFT, ITL, concurrency scaling, and load patterns using AIPerf or equivalent tools."
+---
+
+# LLM Inference Benchmarking
+
+Measure the performance of generative AI inference endpoints under realistic and stress loads. This skill covers inference performance (speed, throughput, concurrency limits), NOT accuracy benchmarks like MMLU or GSM8K—use evaluating-llms-harness for those.
+
+## Trigger
+
+Use when:
+- Setting up performance benchmarks for LLM APIs (OpenAI-compatible, vLLM, TGI, etc.)
+- Determining capacity limits ("how many concurrent users can this model handle?")
+- Producing sales/reliability metrics (latency percentiles, throughput numbers)
+- Comparing model deployments or hardware configurations
+- Regression testing inference endpoints after updates
+
+## Primary Tool: AIPerf
+
+AIPerf (NVIDIA/ai-dynamo) is the recommended tool. **Version 0.10.0+ uses a new Schema 2.0 YAML format**—see Config Format section below.
+
+It supports:
+- OpenAI chat/completions/embeddings/audio/vision endpoints
+- Concurrency, request-rate, trace-replay, and ramping modes
+- ShareGPT and custom datasets
+- Real-time dashboard, CSV/JSON exports, and automatic plotting
+- Multiprocess architecture via ZMQ
+
+Installation:
+```bash
+pip install aiperf
+# Optional extras:
+pip install "aiperf[wandb]"   # historical trend tracking
+pip install "aiperf[otel]"    # live telemetry streaming
+```
+
+Platform notes:
+- **Linux aarch64**: `crick` dependency needs build tools: `sudo apt install build-essential`
+- **Older x86_64 VMs without X86_V2**: NumPy 2.x crashes with `RuntimeError: NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support: (X86_V2)`. Fix: `pip install "numpy<2.0" --force-reinstall`. This applies to older cloud VMs, Proxmox containers, or legacy hardware that lacks AVX2/BMI2/MOVBE/POPCNT/FMA.
+
+## AIPerf 0.10.0+ Config Format (Schema 2.0)
+
+> **Critical**: AIPerf 0.10.0 introduced a new YAML envelope. Old flat configs are rejected.
+
+### Envelope Structure
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+    strategy: "round_robin"
+  endpoint:
+    urls:
+      - "https://ai.noris.de/v1"
+    type: "chat"
+    api_key: "${API_KEY}"          # Can reference env var; plaintext also works
+    streaming: true
+    headers:
+      Authorization: "Bearer ${API_KEY}"
+  datasets:
+    - type: "synthetic"            # or "public" for ShareGPT
+      name: "main"
+      entries: 100
+      isl:
+        type: "fixed"
+        value: 512
+      osl:
+        type: "fixed"
+        value: 128
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      # One of requests, duration, or sessions MUST be set
+      requests: 50                  # total requests for this phase
+      sessions: 1                   # concurrent sessions (default: 1)
+      # duration: 600               # alternative to requests
+  tokenizer:
+    name: "builtin"                 # or huggingface model name
+```
+
+### Key fields explained
+- `models.items[].name`: The model identifier (must match endpoint's `/v1/models` list)
+- `endpoint.api_key`: Can be `${ENV_VAR}` reference. CLI `--api-key` overrides this.
+- `datasets.type`: `"public"` downloads ShareGPT from HuggingFace (first run ~2–5 min). `"synthetic"` generates locally with no download.
+- `datasets.isl` / `datasets.osl`: Input/Output Sequence Length distributions. `type: "fixed"` with `value: N` for deterministic lengths.
+- `phases`: Must specify at least one of `requests`, `duration`, or `sessions`. Default `concurrency` inside phase is 1.
+
+### CLI Overrides
+AIPerf allows overriding YAML values at CLI time. The most useful for suite orchestration:
+
+| CLI flag | Overrides YAML field | Typical use |
+|----------|----------------------|-------------|
+| `--concurrency N` | phase.sessions (indirectly) | Sweep concurrency levels |
+| `--request-count N` | phase.requests | Control total requests per sub-run |
+| `--duration N` | phase.duration | Time-based runs (steady-state) |
+| `--artifact-dir PATH` | output directory | Organize results per run |
+| `--ui simple` | dashboard mode | Less overhead, no TUI |
+| `--no-gpu-telemetry` | - | Skip DCGM (endpoints without GPU metrics) |
+| `--no-server-metrics` | - | Skip Prometheus scraping (non-vLLM endpoints) |
+
+**Orchestration pattern**: Define base structure in YAML, then use CLI flags in a shell loop for parameter sweeps:
+```bash
+for conc in 1 5 10 25 50 100; do
+  aiperf profile --config capacity.yaml --concurrency "$conc" --request-count 100 \
+    --artifact-dir "results/capacity/conc${conc}"
+done
+```
+
+## The 3-Scenario Suite Pattern
+
+Design benchmark suites around three complementary scenarios:
+
+### 1. Sales Scenario (Representative Load)
+- **Goal**: Clean numbers for pitch decks and SLAs
+- **Concurrency**: 1, 5, 10 (low, moderate, realistic)
+- **Dataset**: ShareGPT (realistic chat transcripts)
+- **Count**: ~50 requests per run
+- **Output**: avg/p99 TTFT, throughput, request latency
+
+### 2. Capacity Scenario (Breakpoint Discovery)
+- **Goal**: Find the concurrency ceiling before degradation
+- **Concurrency**: 10 → 25 → 50 → 100 (stepped ramp)
+- **Dataset**: Synthetic with fixed ISL/OSL (e.g., 512 IN / 128 OUT)
+- **Count**: ~100 requests per level
+- **Output**: Performance curve (throughput vs. concurrency, latency inflection points)
+
+**When to stop scaling concurrency**: Before proposing conc=200+ tests, extrapolate the existing curve. If TTFT p99 exceeds ~5s at conc=100, further scaling is operationally irrelevant—users won't accept 15s+ wait times. Better to test **longer inputs** (1024/2048 tokens) to isolate the memory-bandwidth prefill bottleneck, or implement queueing + load balancing.
+
+### 2b. Long-Context Prefill Test (Memory Bandwidth Bottleneck)
+- **Goal**: Isolate the memory-bandwidth prefill bottleneck (relevant for RAG, long-context agents)
+- **Concurrency**: Fixed low (e.g., 10) to avoid queueing artifacts
+- **Dataset**: Synthetic with **long input** (1024 / 2048 / 4096 tokens), short output (128)
+- **Duration**: 300s (long enough to amortize cold-start)
+- **Output**: Prefill throughput vs. input length curve
+
+This is more informative than conc=200 when the real workload involves reading long documents.
+
+### 3. Steady-State Scenario (Long-Term Stability)
+- **Goal**: Detect drift, memory leaks, or throttling over time
+- **Concurrency**: Sweet spot from Capacity scenario (e.g., 20)
+- **Duration**: 600 seconds (10 minutes) or longer
+- **Dataset**: Mixed (50% ShareGPT + 50% synthetic short/long)
+- **Output**: P50/P90/P99 trends over time, failed request rate
+
+### Universal Tokenizer for Heterogeneous Endpoints
+
+When benchmarking multiple models with different vocabularies (and downloading the correct HuggingFace tokenizer for each is impractical), use:
+
+```bash
+--tokenizer builtin
+```
+
+This uses tiktoken's `o200k_base` encoding universally. It avoids HF downloads, gated-repo access issues, and rate limits. **Trade-off:** Token counts are approximate for non-OpenAI models (different vocabularies), but **TTFT, ITL, and latency metrics remain exact**. Recommended when:
+- Tokenizer mapping is complicated (4+ models, gated repos, missing HF entries)
+- Running in air-gapped or rate-limited environments
+- Sales/capacity benchmark where relative comparisons matter more than absolute token counts
+
+Before benchmarking, discover available models from the target endpoint:
+
+```bash
+curl -s -H "Authorization: Bearer $API_KEY" \
+  https://<endpoint>/v1/models | \
+  python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin).get('data',[])]"
+```
+
+See `scripts/discover-models.sh` for a ready-to-run version.
+
+## Authentication & Security
+
+- **NEVER hardcode API keys** in YAML configs, shell scripts, or git
+- Always use environment variables: `${API_KEY}` or `$API_KEY`
+- In AIPerf YAML, reference via `--extra-headers` or templating:
+  ```yaml
+  extra-headers:
+    Authorization: "Bearer ${API_KEY}"
+  ```
+- Export in shell before running: `export API_KEY=sk-...`
+
+## AIPerf Quick Reference
+
+Basic profile run:
+```bash
+aiperf profile \
+  --model "vllm/qwen3.6-27b-nvfp4" \
+  --streaming \
+  --endpoint-type chat \
+  --tokenizer <huggingface-tokenizer-name> \
+  --url https://ai.noris.de \
+  --extra-headers "Authorization: Bearer ${API_KEY}" \
+  --concurrency 10 \
+  --request-count 50
+```
+
+Key flags:
+- `--concurrency N` – parallel requests
+- `--request-rate N` – requests per second (alternative to concurrency)
+- `--request-count N` – total requests to send
+- `--duration N` – time-based run (seconds)
+- `--streaming` – measure streaming metrics (TTFT, ITL)
+- `--output-tokens-mean N` – target output length (with caveats; see AIPerf docs)
+- `--dataset <file>` – custom dataset (ShareGPT format or synthetic)
+- `--random-seed N` – reproducibility
+
+## YAML Configs
+
+AIPerf supports YAML-driven runs for reproducibility. See `templates/aiperf-suite-config.yaml` for a complete 3-scenario suite template.
+
+Run from YAML:
+```bash
+aiperf profile -c configs/sales-scenario.yaml
+```
+
+## Results & Visualization
+
+AIPerf produces:
+- **CSV** – per-request metrics (latency, token counts, timestamps)
+- **JSON** – aggregated statistics with percentile data (`avg`, `min`, `max`, `p50`, `p90`, `p99`, `std`)
+- **Logs** – `aiperf.log` with detailed debug info
+- **Plots** – PNG charts via `aiperf plot <results-dir>`
+
+Results should be organized as:
+```
+results/YYYY-MM-DD/
+  sales/
+  capacity/
+  steady-state/
+```
+
+### Custom Post-Processing for Multi-Model Suites
+
+For sales/capacity comparison across multiple models and concurrency levels, AIPerf's built-in `plot` subcommand works on single runs. Use a custom Python post-processing script to generate cross-model comparison charts:
+
+1. **Load all** `profile_export_aiperf.json` files recursively from `results/YYYY-MM-DD/`
+2. **Extract key metrics** per `(model, scenario, concurrency)`:
+   - `output_token_throughput.avg` = throughput (tok/s) **TOTAL**
+   - `time_to_first_token.p99` = tail latency (ms)
+   - `inter_token_latency.p99` = streaming smoothness (ms)
+   - `prefill_throughput_per_user.avg` = prefill speed **per user session**
+   - `output_token_throughput_per_user.avg` = output speed **per user**
+   - `total_token_throughput.avg` = (input+output) tok/s **TOTAL** across all sessions
+3. **Generate plots** with matplotlib:
+   - `capacity_curve.png` – Throughput vs Concurrency (log scale on x-axis)
+   - `latency_curve.png` – TTFT p99 vs Concurrency (log scale on x-axis)
+   - `sales_comparison.png` – Latency at low concurrency (1/5/10)
+4. **Export** `summary.csv` for Excel / Google Sheets import
+
+A working implementation: `scripts/plot-suite-results.py`
+
+**Rule:** `*_per_user` metrics are for end-user SLA/UX. Total metrics (`output_token_throughput`, `total_token_throughput`) are for infrastructure capacity planning. Input throughput is approximately `total - output`.
+
+### MSP / Service Provider Perspective
+
+When presenting results for infrastructure or commercial decision-making (not model research):
+
+**Don't:** Compare Model A vs Model B on token-per-token quality.  
+**Do:** Report each model as an independent service tier with its own SLA envelope.
+
+**Structure the analysis per model:**
+- **Single-User Experience** (conc=1): TTFT p99, Output/User, SLA class (Gold/Silver/Bronze)
+- **Multi-Tenant Load** (conc=10→100): Throughput scaling, TTFT degradation, recommended max concurrency
+- **Steady-State**: Long-run stability, completed requests, throughput drift
+- **Service Recommendation**: Positioning (Entry/Premium), suitable use cases, limitations
+
+**Key metrics by business question:**
+
+| Business Question | Metric to Use | Why |
+|-------------------|---------------|-----|
+| "How many users per node?" | `total_token_throughput` (Input + Output) | Node capacity |
+| "What does each user feel?" | `output_token_throughput_per_user` | UX/SLA |
+| "Is chat real-time responsive?" | `ttft_p99` + `itl_p99` | Interaktivität |
+| "Can we handle batch jobs?" | `request_throughput` + `output_token_throughput` | Durchsatz |
+| "What breaks first under load?" | TTFT p99 scaling curve | Bottleneck |
+
+**Conc ceiling analysis:** Before proposing conc=200+ tests, extrapolate the existing capacity curve. If TTFT p99 already exceeds 7s at conc=100, further scaling is operationally irrelevant—users won't accept 15s+ wait times. Better to test longer inputs (1024/2048/4096 tokens) to find the prefill bottleneck, or implement queueing + load balancing.
+
+See `references/aiperf-token-metrics-explained.md` for the full unit-reference table.
+
+## Known Pitfalls
+
+1. **AIPerf 0.10.0+ Config Format** — The new `schema_version: "2.0"` envelope is required. Old flat configs will be rejected with `1 validation error for AIPerfConfig`. See Config Format section above. Always include `requests`, `duration`, or `sessions` in phases.
+2. **NumPy X86_V2 crash on older x86_64** — If you see `RuntimeError: NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support: (X86_V2)`, downgrade: `pip install "numpy<2.0" --force-reinstall`
+3. **ShareGPT first-run delay** — The `public` dataset type downloads ShareGPT from HuggingFace on first use (~2–5 minutes). Subsequent runs use a local cache. For quick validation, use `type: "synthetic"`. For production suites, **pre-download once** before the full orchestrated run:
+   ```bash
+   venv/bin/aiperf profile --config configs/any-model/sales.yaml \
+       --concurrency 1 --request-count 1 \
+       --artifact-dir /tmp/warmup --no-gpu-telemetry --no-server-metrics
+   ```
+   After this, `.cache/aiperf/datasets/` contains ShareGPT and all sales runs start instantly.
+4. **Tokenizer mismatch** — Always specify the correct tokenizer. AIPerf auto-detects for some models, but explicit is safer.
+5. **`--output-tokens-mean` is not guaranteed** — Unless the server supports `ignore_eos` and `min_tokens` via `--extra-inputs`.
+6. **Rate limiting** — AIPerf has built-in retry/backoff, but aggressive concurrency on shared endpoints may trigger 429s. Monitor `failed_request_count`.
+7. **Port exhaustion** — Very high concurrency (>15,000) can exhaust local ports. Reduce or adjust system limits.
+10. **Orchestrator path resolution** — When running benchmarks in background processes (cron, systemd, detached shells), `PATH` may not include a Python venv. Always use the absolute path: `AIPERF="$BASE_DIR/venv/bin/aiperf"` in shell scripts, never rely on `aiperf` being in `$PATH`. The failure mode is silent: every run produces `aiperf: command not found` but the orchestrator continues, producing empty results.
+    See `references/aiperf-background-process-pitfalls.md`.
+11. **Prefill throughput bias from cold-start requests** — The first request in any AIPerf session includes DNS resolution + TCP/TLS handshake time, which is counted toward "prefill throughput". At conc=1 this can deflate prefill metrics by 50%+. The aggregate `profile_export_aiperf.json` cannot filter this out. Use `profile_export.jsonl` (per-request data) to discard `turn_index=0` requests, or pre-warm connections with a throwaway run before the measured benchmark.
+    See `references/aiperf-prefill-warmup-bias.md`.
+12. **Token metric unit confusion** — AIPerf exports multiple "tokens/sec" metrics with different units (per-user vs. total). `prefill_throughput_per_user` is tok/s **per session**, while `output_token_throughput` is tok/s **total across all sessions**. Comparing them directly or using the wrong one for capacity planning leads to off-by-100× errors.
+    - Use `*_per_user` metrics for **end-user experience** (SLA, UX)
+    - Use `output_token_throughput` / `total_token_throughput` for **node capacity** (infrastructure sizing)
+    - Approximate input throughput: `total - output`
+    See `references/aiperf-token-metrics-explained.md`.
+
+## Orchestration
+
+For multi-scenario suites, wrap AIPerf calls in a shell script that:
+1. Validates connectivity and auth (health check)
+2. Runs scenarios sequentially
+3. Organizes results in timestamped folders
+4. Generates plots after each run
+5. Detects failures (exit codes, empty CSVs, high error rates) and continues
+
+## Session References
+
+- `references/aiperf-0.10.0-migration.md` — complete config format migration guide with validation error fixes
+- `references/aiperf-long-input-benchmarking.md` — long-input prefill bottleneck testing (replaces conc=200+ tests when memory bandwidth is the real limiter)
+- `templates/aiperf-long-input-config.yaml` — working Schema 2.0 template for 1024-token input tests
+- `references/aiperf-token-metrics-explained.md` — unit reference table: per-user vs. total tok/s, input throughput approximation
+- `references/aiperf-suite-orchestrator-pattern.md` — complete suite orchestrator pattern: directory layout, venv path fix, CLI sweep pattern, ShareGPT pre-download, post-processing sequence
+- `references/aiperf-prefill-warmup-bias.md` — cold-start bias in prefill metrics, mitigation options, per-request JSONL filtering
+- `references/aiperf-token-metrics-explained.md` — unit reference: per-user vs total metrics, input calculation, business-KPI mapping
+- `references/aiperf-capacity-curve-analysis.md` — interpreting capacity curves: when to stop scaling concurrency, breakpoint detection, extrapolation rules
+- `references/aiperf-long-context-prefill-test.md` — isolating the memory-bandwidth bottleneck with 1024/2048/4096-token inputs
+- `references/ai-noris-de-endpoint.md` — discovered model list, URL, auth pattern for ai.noris.de
+- `references/aiperf-kurzreferenz.md` — German quick reference summary
+- `templates/aiperf-suite-config-v2.yaml` — Schema 2.0 starter template with all options documented
+- `templates/long-context-prefill-test.yaml` — 1024-token input test template for RAG/memory-bandwidth analysis
+- `templates/3-scenario-suite.md` — reusable Sales/Capacity/Steady-State suite structure
+
+- **GenAI-Perf** (NVIDIA, predecessor to AIPerf) – AIPerf is the migration target
+- **locust + OpenAI client** – custom Python load testing, more flexible but more work
+- **k6 + xk6-openai** – if already using k6 for HTTP load testing
+
+- **GenAI-Perf** (NVIDIA, predecessor to AIPerf) – AIPerf is the migration target
+- **locust + OpenAI client** – custom Python load testing, more flexible but more work
+- **k6 + xk6-openai** – if already using k6 for HTTP load testing
+
+## Related Skills
+
+- evaluating-llms-harness – for accuracy benchmarks (MMLU, GSM8K, etc.)
+- serving-llms-vllm – for setting up the inference server being benchmarked
@@ -0,0 +1,43 @@
+# ai.noris.de Endpoint Reference (Session 2026-06-18)
+
+## Endpoint Details
+
+- **URL**: `https://ai.noris.de/v1`
+- **Type**: OpenAI-compatible Chat Completions
+- **Auth**: Bearer token via `Authorization` header
+- **API Key**: `sk-bf-8779d73c-6a51-49d6-a22f-00a0290ca6a7`
+- **Models endpoint**: `GET /v1/models` → returns 4 models
+
+## Discovered Models (2026-06-18)
+
+| Model ID | Size | Notes |
+|----------|------|-------|
+| `vllm/gemma-4-31b-it` | 31B | Google Gemma 4 |
+| `vllm/gpt-oss-120b` | 120B | GPT-OSS, largest model |
+| `vllm/moonshotai/kimi-k2.6` | ~12B | Kimi K2.6, strongest generalist |
+| `vllm/qwen3.6-27b-nvfp4` | 27B | Qwen 3.6 with NVFP4 quantization |
+
+All models served via vLLM.
+
+## Benchmark Results (Single Request, Synthetic 512/128)
+
+### gemma-4-31b-it
+- **TTFT**: 361.7 ms
+- **Throughput**: 13.78 tok/s
+- **ISL**: 512, **OSL**: 128
+
+### gpt-oss-120b
+- Slightly higher latency expected due to size
+
+### moonshotai/kimi-k2.6
+- Generally strongest quality
+
+### qwen3.6-27b-nvfp4
+- NVFP4 quantization is aggressive – monitor quality vs. speed trade-off
+
+## Endpoint Compatibility Notes
+
+- **Prometheus metrics**: `/v1/metrics` returns HTML (web UI), not Prometheus exposition format
+- `--no-server-metrics` recommended when benchmarking
+- **GPU telemetry**: No DCGM endpoints exposed (localhost:9400/9401 unreachable)
+- `--no-gpu-telemetry` recommended
@@ -0,0 +1,190 @@
+# AIPerf 0.10.0 Schema 2.0 Migration Guide
+
+> Session artifact: migrated 12 configs from old flat format to new envelope format. All errors encountered are documented here with fixes.
+
+## The Old Format (DEPRECATED)
+
+```yaml
+url: "https://ai.noris.de/v1"
+endpoint_type: "chat"
+model: "vllm/gemma-4-31b-it"
+extra-headers:
+  Authorization: "Bearer ${API_KEY}"
+streaming: true
+tokenizer: "builtin"
+request_count: 50
+concurrency: 1
+```
+
+**Result with AIPerf 0.10.0:** `1 validation error for AIPerfConfig`
+
+## The New Format (Schema 2.0)
+
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+    strategy: "round_robin"
+  endpoint:
+    urls:
+      - "https://ai.noris.de/v1"
+    type: "chat"
+    api_key: "${API_KEY}"
+    streaming: true
+    headers:
+      Authorization: "Bearer ${API_KEY}"
+  datasets:
+    - type: "synthetic"         # or "public" for ShareGPT
+      name: "main"
+      entries: 100
+      isl:
+        type: "fixed"
+        value: 512
+      osl:
+        type: "fixed"
+        value: 128
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      # ONE OF THESE IS MANDATORY:
+      requests: 50           # total requests
+      # OR duration: 600     # seconds
+      # OR sessions: 1       # concurrent sessions
+  tokenizer:
+    name: "builtin"
+```
+
+## Critical Validation Errors and Fixes
+
+### Error: `Phase 'profiling': at least one of 'requests', 'duration', or 'sessions' must be specified`
+
+**Cause:** Phase definition without termination condition.
+
+**Fix:** Add `requests: N` or `duration: N` or `sessions: N`.
+
+```yaml
+phases:
+  - type: "concurrency"
+    name: "profiling"
+    requests: 50     # <-- REQUIRED
+```
+
+### Error: `api_key must be a valid string`
+
+**Cause:** AIPerf's Pydantic model validates `api_key` eagerly. Even though `--api-key` CLI flag is present, the YAML's `api_key` field still gets parsed.
+
+**Fix:** Either:
+1. Set a dummy value: `api_key: "dummy"` (CLI `--api-key` overrides it)
+2. Use env var syntax: `api_key: "${API_KEY}"` (works if env var is set during config load)
+3. Use `headers` dict instead for auth
+
+**Best practice:** Use both:
+```yaml
+endpoint:
+  api_key: "${API_KEY}"          # satisfies validation
+  headers:
+    Authorization: "Bearer ${API_KEY}"
+```
+
+### Error: `model` not found in schema
+
+**Cause:** Old flat `model` key at top level. New format nests under `benchmark.models.items[].name`.
+
+**Fix:**
+```yaml
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+```
+
+### Error: `extra-headers` not found
+
+**Cause:** Old key name. New format uses `endpoint.headers`.
+
+**Fix:**
+```yaml
+benchmark:
+  endpoint:
+    headers:
+      Authorization: "Bearer ${API_KEY}"
+```
+
+### Error: `endpoint_type` not found
+
+**Cause:** Old key name. New format uses `endpoint.type`.
+
+**Fix:**
+```yaml
+benchmark:
+  endpoint:
+    type: "chat"
+```
+
+### Error: NumPy X86_V2 crash
+
+```
+RuntimeError: NumPy was built with baseline optimizations:
+(X86_V2) but your machine doesn't support: (X86_V2).
+```
+
+**Cause:** AIPerf installs NumPy 2.x, which requires CPU features not present on older x86_64 VMs.
+
+**Fix:**
+```bash
+pip install "numpy<2.0" --force-reinstall
+```
+
+### Error: ShareGPT download timeout
+
+**Cause:** First run with `type: "public"` dataset downloads ShareGPT from HuggingFace (~500MB, 2–5 min).
+
+**Fix:** Wait, or use `type: "synthetic"` for quick validation.
+
+## Synthetic Dataset Format
+
+For fixed ISL/OSL (reproducible capacity tests):
+```yaml
+datasets:
+  - type: "synthetic"
+    name: "main"
+    entries: 100
+    isl:
+      type: "fixed"
+      value: 512
+    osl:
+      type: "fixed"
+      value: 128
+```
+
+Distribution types: `fixed`, `normal`, `lognormal`, `multimodal`, `empirical`.
+
+## CLI Override Compatibility
+
+Flags that work with Schema 2.0 YAML configs:
+- `--concurrency N` → overrides phase concurrency
+- `--request-count N` → overrides phase.requests
+- `--duration N` → overrides phase.duration
+- `--artifact-dir PATH` → sets output directory
+- `--ui simple|dashboard` → UI mode
+- `--api-key KEY` → overrides endpoint.api_key
+- `--no-gpu-telemetry` → skips DCGM
+- `--no-server-metrics` → skips Prometheus scraping
+
+## Orchestrator Pattern: YAML + CLI Sweeps
+
+```bash
+AIPERF="./venv/bin/aiperf"  # always use venv path, never rely on PATH
+
+for conc in 1 5 10; do
+  "$AIPERF" profile \
+    --config sales.yaml \
+    --concurrency "$conc" \
+    --request-count 50 \
+    --artifact-dir "results/sales/conc${conc}"
+done
+```
+
+**Critical:** In background processes (cron, systemd, nohup), `PATH` may not include the venv. Always use absolute path to `venv/bin/aiperf`.
@@ -0,0 +1,111 @@
+# AIPerf Background-Process & Orchestrator Pitfalls
+
+Session artifact: debugging why a full 12-run benchmark suite silently failed with `aiperf: command not found` despite successful manual runs.
+
+## The Problem: `aiperf: command not found` in Background
+
+The background process (started via `bash -lic`) did not inherit the venv-activated PATH. All 12 runs produced:
+```
+./scripts/run-suite.sh: line 90: aiperf: command not found
+```
+
+**Root cause:** `aiperf` was installed into a Python venv, but the background shell did not activate the venv before running the orchestrator.
+
+## The Fix: Absolute Path to venv Binary
+
+In the orchestrator script, define an absolute path variable:
+```bash
+AIPERF="$BASE_DIR/venv/bin/aiperf"
+if [ ! -x "$AIPERF" ]; then
+    echo "ERROR: AIPerf not found at $AIPERF"
+    exit 1
+fi
+```
+
+Then call `"$AIPERF" profile ...` everywhere instead of plain `aiperf profile`.
+
+## ShareGPT Pre-Download Strategy
+
+`type: "public"` datasets download ShareGPT from HuggingFace on first use (~2–5 minutes). To avoid this delay during production benchmark runs, perform a pre-download:
+
+```bash
+# Pre-download ShareGPT into AIPerf's local cache
+venv/bin/aiperf profile \
+    --config configs/<any-model>/sales.yaml \
+    --concurrency 1 --request-count 1 \
+    --no-gpu-telemetry --no-server-metrics \
+    --artifact-dir /tmp/predownload
+```
+
+Subsequent sales-scenario runs will use the cached dataset instantly.
+
+## AIPerf 0.10.0 CLI Flag Reference
+
+Critical flags used in this session:
+
+| Flag | What it does | Status |
+|------|-------------|--------|
+| `--ui simple` | Minimal UI mode (no TUI) | ✅ Works |
+| `--ui dashboard` | Full TUI dashboard | ✅ Works |
+| `--artifact-dir PATH` | Output directory for CSV/JSON/logs | ✅ Works |
+| `--no-gpu-telemetry` | Skip DCGM/GPU metrics | ✅ Works |
+| `--no-server-metrics` | Skip Prometheus scraping | ✅ Works |
+| `--request-count N` | Total requests (overrides YAML `requests`) | ✅ Works |
+| `--concurrency N` | Concurrent sessions | ✅ Works |
+| `--simple` | ❌ NOT a flag; use `--ui simple` | ❌ Rejected |
+| `--output PATH` | ❌ NOT a flag; use `--artifact-dir` | ❌ Rejected |
+| `--output-artifact-dir PATH` | Alias for `--artifact-dir` | ✅ Works |
+| `--api-key KEY` | Overrides YAML `api_key` | ✅ Works |
+
+## The `api_key` + `requests`/`duration`/`sessions` Interaction
+
+Sequence of errors encountered:
+
+1. `api_key must be a valid string` — this appeared when `phases[].requests` was missing.
+2. With `requests: 50` and `sessions: 1` added to phases → config validated.
+3. **Lesson:** The `api_key` error was misleading. The real issue was always a missing `requests`/`duration`/`sessions` in the phase definition.
+
+Pattern that works:
+```yaml
+phases:
+  - type: "concurrency"
+    name: "profiling"
+    requests: 50          # REQUIRED
+    sessions: 1           # optional, default 1
+```
+
+## Complete Working Config (Single File)
+
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+    strategy: "round_robin"
+  endpoint:
+    urls:
+      - "https://ai.noris.de/v1"
+    type: "chat"
+    api_key: "${NORIS_API_KEY}"
+    streaming: true
+    headers:
+      Authorization: "Bearer ${NORIS_API_KEY}"
+  datasets:
+    - type: "synthetic"
+      name: "main"
+      entries: 100
+      isl:
+        type: "fixed"
+        value: 512
+      osl:
+        type: "fixed"
+        value: 128
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      requests: 50
+      sessions: 1
+  tokenizer:
+    name: "builtin"
+```
@@ -0,0 +1,98 @@
+# Capacity Curve Analysis – When to Stop Scaling Concurrency
+
+## The Core Question
+
+"Shall we test conc=200?" The answer depends on extrapolating the existing capacity curve, not on gut feeling.
+
+## Extrapolation Rule
+
+Given TTFT p99 values at conc=10, 50, 100, estimate conc=200 using a **linear degradation model** (the simplest that fits observed data):
+
+```
+TTFT_p99(conc) ≈ base + k × conc
+
+Where k is derived from the steepest interval:
+k = (TTFT_p99(conc=100) - TTFT_p99(conc=50)) / (100 - 50)
+```
+
+If the extrapolated TTFT p99 at conc=200 exceeds a **user-acceptable threshold** (typically 5000ms = 5s for interactive applications, 10000ms = 10s for batch), conc=200 is **operationally irrelevant**.
+
+## Worked Example (from this session)
+
+### gpt-oss-120b
+| Conc | TTFT p99 |
+|------|----------|
+| 10 | 978 ms |
+| 25 | 1924 ms |
+| 50 | 3575 ms |
+| 100 | 7310 ms |
+
+Extrapolation to conc=200:
+- k = (7310 - 3575) / 50 = **74.7 ms/conc**
+- TTFT_p99(200) = 7310 + 74.7 × 100 = **14,780 ms**
+
+**Verdict: Unusable. Users won't wait 15 seconds for the first token.**
+
+### gemma-4-31b-it
+| Conc | TTFT p99 |
+|------|----------|
+| 10 | 381 ms |
+| 50 | 1042 ms |
+| 100 | 2060 ms |
+
+Extrapolation:
+- k = (2060 - 1042) / 50 = **20.4 ms/conc**
+- TTFT_p99(200) = 2060 + 20.4 × 100 = **4,100 ms**
+
+**Verdict: Borderline. 4.1s is acceptable for some batch use cases, unacceptable for chat.**
+
+### moonshotai-kimi-k2.6
+| Conc | TTFT p99 |
+|------|----------|
+| 10 | 695 ms |
+| 25 | 1178 ms |
+| 50 | 1989 ms |
+| 100 | 3926 ms |
+
+Extrapolation:
+- k = (3926 - 1989) / 50 = **38.7 ms/conc**
+- TTFT_p99(200) = 3926 + 38.7 × 100 = **7,796 ms**
+
+**Verdict: Unusable for interactive use.**
+
+## Alternative to Higher Concurrency: Longer Inputs
+
+When conc=100 is already the practical ceiling, the more valuable test is **long-context prefill stress**:
+
+| Test | What it measures | When to use |
+|------|-----------------|-------------|
+| conc=10→100 | Queueing / scheduling efficiency | Find max parallel users |
+| ISL=512 → 1024 → 2048 → 4096 | Memory bandwidth / KV-cache limits | RAG, document analysis, long-context agents |
+
+A model that degrades gracefully at conc=100 but collapses at ISL=2048 has a **memory-bandwidth bottleneck**, not a scheduling bottleneck. The fix is faster memory (HBM3e), not more GPU compute.
+
+## Breakpoint Detection Heuristic
+
+Formally define the "breakpoint" as the highest concurrency where **all three** conditions hold:
+
+1. TTFT p99 < 5000 ms (5 seconds)
+2. ITL p99 < 200 ms (readable streaming, ~5 tok/s per user)
+3. Output throughput still scales linearly with concurrency (no saturation)
+
+From observed data:
+
+| Model | Breakpoint (conc) | Limiting factor |
+|-------|-------------------|-----------------|
+| gemma-4-31b-it | ~100 | TTFT ≈ 2s still OK |
+| moonshotai-kimi-k2.6 | ~75 | TTFT ≈ 3.5s approaching limit |
+| gpt-oss-120b | ~40 | TTFT ≈ 3.5s at conc=50 |
+| qwen3.6-27b-nvfp4 | ~40 | TTFT ≈ 3s at conc=50 |
+
+## Operational Recommendation
+
+- Set **hard concurrency limits** at the breakpoint, not at the hardware maximum
+- Implement **queueing + load balancing** beyond the breakpoint
+- Test **LLM-as-a-Service tiers**:
+  - Tier 1 (Premium): conc ≤ 10, guaranteed < 1s TTFT
+  - Tier 2 (Standard): conc ≤ 50, guaranteed < 3s TTFT
+  - Tier 3 (Batch): conc ≤ 100, best-effort, queueing OK
@@ -0,0 +1,111 @@
+# AIPerf 0.10.0+ Kurzreferenz
+
+## Installation
+
+```bash
+pip install aiperf
+# Extras: [mlflow] [otel] [wandb]
+```
+
+**NumPy compatibility warning:** On older x86_64 VMs without X86_V2 CPU features, NumPy 2.x (pulled by AIPerf) crashes. Pre-fix with:
+```bash
+pip install "numpy<2.0" --force-reinstall --no-cache-dir
+pip install aiperf
+```
+
+## CLI Quickstart
+
+```bash
+aiperf profile \
+  --config config.yaml \
+  --api-key "$API_KEY" \
+  --concurrency 10 \
+  --request-count 50 \
+  --artifact-dir results/
+```
+
+## Key CLI Flags (0.10.0+)
+
+| Flag | YAML Override | Description |
+|------|---------------|-------------|
+| `--config FILE` | - | Load Schema 2.0 YAML config |
+| `--api-key KEY` | `endpoint.api_key` | Auth key (overrides YAML) |
+| `--concurrency N` | phase.sessions | Concurrent requests/sessions |
+| `--request-rate N` | - | Requests/sec (alt. to concurrency) |
+| `--request-count N` | phase.requests | Total requests |
+| `--duration N` | phase.duration | Time-based run (seconds) |
+| `--streaming` | `endpoint.streaming` | Enable streaming (TTFT/ITL) |
+| `--ui simple` | `ui` | Progress bars (vs. `dashboard` TUI) |
+| `--ui none` | `ui` | Headless |
+| `--artifact-dir DIR` | `output_dir` | Results directory |
+| `--no-gpu-telemetry` | - | Skip DCGM/GPU metrics |
+| `--no-server-metrics` | - | Skip Prometheus scraping |
+| `--tokenizer builtin` | `tokenizer.name` | tiktoken (no HF download) |
+| `--random-seed N` | `random_seed` | Reproducibility |
+
+## Schema 2.0 YAML Config
+
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+    strategy: "round_robin"
+  endpoint:
+    urls:
+      - "https://ai.noris.de/v1"
+    type: "chat"
+    api_key: "${API_KEY}"
+    streaming: true
+    headers:
+      Authorization: "Bearer ${API_KEY}"
+  datasets:
+    - type: "public"        # "public" = HF download; "synthetic" = local gen
+      name: "main"
+      dataset: "sharegpt"
+      sampling: "shuffle"
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      requests: 50
+      sessions: 1
+  tokenizer:
+    name: "builtin"
+```
+
+Run from YAML:
+```bash
+aiperf profile --config config.yaml
+```
+
+## Outputs
+- **CSV** (`profile_export_aiperf.csv`): Per-request metrics
+- **JSON** (`profile_export_aiperf.json`): Aggregated stats (avg, p50, p90, p99)
+- **Log** (`logs/aiperf.log`): Detailed debug
+- **Plots**: `aiperf plot <artifact-dir>`
+
+## Dataset Types Quick Reference
+
+| Type | Speed | Source | Use |
+|------|-------|--------|-----|
+| `public` (ShareGPT) | Slow first-run (~2-5 min) | HuggingFace download | Realistic workload |
+| `synthetic` (fixed ISL/OSL) | Instant | Local generation | Capacity testing, reproducibility |
+
+## Orchestration Pattern
+
+YAML base + CLI sweeps:
+```bash
+for conc in 1 5 10 25 50 100; do
+  aiperf profile --config capacity.yaml --concurrency "$conc" \
+    --request-count 100 --artifact-dir "results/conc${conc}"
+done
+```
+
+## Validation Errors & Quick Fixes
+
+| Error | Fix |
+|-------|-----|
+| `1 validation error for AIPerfConfig` | Wrap YAML in `schema_version: "2.0"` / `benchmark:` envelope |
+| `Phase 'profiling': at least one of 'requests', 'duration', or 'sessions' must be specified` | Add `requests: N` or `duration: N` to phase |
+| `NumPy: baseline optimizations (X86_V2) not supported` | `pip install "numpy<2.0" --force-reinstall` |
@@ -0,0 +1,133 @@
+# Long-Context Prefill Test – Memory Bandwidth Bottleneck
+
+## Purpose
+
+Isolate the **memory-bandwidth prefill bottleneck** (relevant for RAG, document analysis, long-context agents). While capacity tests scale concurrency with short inputs, this test scales **input length** at fixed low concurrency to find where KV-cache limits hit.
+
+## When to Use
+
+- The capacity curve shows conc=100 is already the practical ceiling
+- The real workload involves **long documents** (1K–8K tokens) not many short chats
+- You need to answer: "Can this model handle our RAG pipeline with 4K context windows?"
+
+## AIPerf Config
+
+Use `type: "synthetic"` with `fixed` ISL. The `prompt_template` must include a variable that expands to the target length. AIPerf's synthetic engine automatically generates filler text to reach the target token count.
+
+### Template: 1024-token input
+
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/REPLACE-MODEL-NAME"
+  endpoint:
+    urls: ["https://ENDPOINT/v1"]
+    type: "chat"
+    streaming: true
+  datasets:
+    default:
+      - name: "synthetic_long"
+        type: "synthetic"
+        prompt_template:
+          content: "Summarize the following text concisely: {{text}}"
+        variables:
+          - name: "text"
+            values: [
+              "artificial intelligence",
+              "quantum computing",
+              "blockchain technology",
+              "climate change",
+              "machine learning",
+              "renewable energy"
+            ]
+    strategy:
+      type: "fixed"
+      max_total_tokens: 1152    # input + output must fit
+      input_tokens: 1024        # target input length
+      output_tokens: 128        # short output to focus on prefill
+  phases:
+    - type: "concurrency"
+      name: "prefill_stress"
+      duration: 300             # 5 minutes, time-based
+      sessions: 10              # fixed low concurrency
+  tokenizer:
+    name: "builtin"
+```
+
+### Scaling input length
+
+Create a sweep by varying `input_tokens` across runs:
+
+| Run | input_tokens | max_total_tokens | What it tests |
+|-----|-------------|------------------|---------------|
+| 1 | 512 | 640 | Baseline (standard context) |
+| 2 | 1024 | 1152 | Long context (RAG documents) |
+| 3 | 2048 | 2176 | Very long context |
+| 4 | 4096 | 4224 | Extreme context (edge case) |
+
+**Constraint:** `max_total_tokens >= input_tokens + output_tokens + 16` (AIPerf padding). The model's own context window (`max_model_len`) is the hard ceiling—check the endpoint's `/v1/models` response.
+
+## CLI Execution
+
+```bash
+for islen in 512 1024 2048; do
+  model="vllm/gemma-4-31b-it"
+  model_safe=$(echo "$model" | tr '/.' '_')
+  
+  # Generate per-run config or use sed to modify input_tokens
+  sed "s/input_tokens: 1024/input_tokens: ${islen}/; s/max_total_tokens: 1152/max_total_tokens: $((islen + 128 + 16))/" \
+    long-context-template.yaml > /tmp/run.yaml
+  
+  venv/bin/aiperf profile \
+    --config /tmp/run.yaml \
+    --api-key "$API_KEY" \
+    --concurrency 10 \
+    --ui none \
+    --no-gpu-telemetry \
+    --no-server-metrics \
+    --artifact-dir "results/longcontext/${model_safe}/isl${islen}"
+done
+```
+
+## Metrics to Extract
+
+From `profile_export_aiperf.json`:
+
+| Metric | Key | Unit | Interpretation |
+|--------|-----|------|----------------|
+| Prefill throughput | `prefill_throughput_per_user.avg` | tok/s/user | How fast a single user's long prompt is processed |
+| TTFT | `time_to_first_token.avg` / `.p99` | ms | Time until first token at long context |
+| Input length | `input_sequence_length.avg` | tokens | Actual achieved input length (verify synthetic hit target) |
+| Output/User | `output_token_throughput_per_user.avg` | tok/s/user | Generation speed (should be stable; if it drops, KV-cache eviction) |
+
+## Expected Behavior
+
+Healthy system:
+- Prefill throughput **degrades gracefully** with longer inputs (sub-linear)
+- TTFT scales roughly linearly with input length: `TTFT ≈ input_tokens / prefill_tps`
+- Output/User stays constant (generation is independent of input length)
+
+Unhealthy system (KV-cache limits):
+- Prefill throughput **collapses** at a threshold length (e.g., 2048 tokens)
+- TTFT becomes super-linear (quadratic or worse)
+- Output/User drops (KV-cache thrashing or CPU offloading)
+
+## Interpreting Results
+
+Calculate **prefill time per 1K tokens**:
+
+```
+Prefill time per 1K = 1000 / prefill_throughput_per_user (seconds)
+```
+
+From observed conc=100 data (short-input baseline):
+
+| Model | Prefill/User @ 512 tok | Est. per 1K |
+|-------|----------------------|-------------|
+| gemma-4-31b-it | 505 tok/s | ~2.0s |
+| moonshotai-kimi-k2.6 | 276 tok/s | ~3.6s |
+| gpt-oss-120b | 137 tok/s | ~7.3s |
+
+If the long-context test at 1024 tokens shows significantly worse than double these estimates, the model/server has **non-linear KV-cache scaling**—a hard limit for RAG workloads.
@@ -0,0 +1,109 @@
+# AIPerf Long-Input and Extreme-Context Benchmarking
+
+Session: 2026-06-18 — Full suite benchmark of 4 models (gemma-4-31b-it, gpt-oss-120b, moonshotai/kimi-k2.6, qwen3.6-27b-nvfp4) on ai.noris.de
+
+## Key Finding: Long Inputs > High Concurrency for Bottleneck Isolation
+
+When evaluating maximum node capacity, testing conc=200+ is often operationally irrelevant because TTFT p99 exceeds acceptable thresholds (7s+ at conc=100). The real infrastructure bottleneck for RAG / long-context workloads is **memory-bandwidth-bound prefill** with long inputs.
+
+Recommended alternative: **Long-Input Prefill Test**
+- Concurrency: Fixed low (e.g., 10) to avoid queueing artifacts
+- Input length: 1024 / 2048 / 4096 tokens (double/triple the standard 512)
+- Output length: Short (128) to isolate prefill from generation
+- Duration: 300s to amortize cold-start
+
+## Extreme Context Test (10k Input / 10k Output)
+
+For stress-testing context-window limits and sustained generation capacity:
+
+- **Input:** 10,000 tokens
+- **Output:** 10,000 tokens
+- **Concurrency:** 1 (sequentially, because runtime is 200–1000 seconds per request)
+- **Requests:** 10 per model
+- **Purpose:** Validates the full KV-cache lifecycle and sustained decode performance
+
+**Expected runtime:** At 10–50 tok/s output, one request takes 200–1000 seconds. Schedule accordingly.
+
+**Template:**
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/MODEL_NAME"
+  endpoint:
+    urls: ["https://ai.noris.de/v1"]
+    type: "chat"
+    streaming: true
+  datasets:
+    - type: "synthetic"
+      name: "main"
+      entries: 10
+      isl:
+        type: "fixed"
+        value: 10000
+      osl:
+        type: "fixed"
+        value: 10000
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      requests: 10
+```
+
+## Correct AIPerf 0.10.0 Synthetic Dataset Config (Long Input)
+
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+  endpoint:
+    urls: ["https://ai.noris.de/v1"]
+    type: "chat"
+    streaming: true
+  datasets:
+    - name: "synthetic_long"
+      type: "synthetic"
+      prompt_template:
+        content: "Analyze the following topic in depth and provide detailed insights."
+      strategy:
+        type: "fixed"
+        max_total_tokens: 1152
+        input_tokens: 1024
+        output_tokens: 128
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      duration: 300
+```
+
+**Critical YAML structure rules learned:**
+- `datasets` MUST be a **list** (`- name: ...`), NOT a dict with `default:` key
+- `name` goes directly under the list item, NOT nested under a `synthetic:` key
+- `prompt_template` and `strategy` are siblings to `name` and `type`
+
+## Benchmark Results at conc=100 (Standard 512-in / 128-out)
+
+| Model | Input tok/s (total) | Output tok/s (total) | Total tok/s | TTFT p99 |
+|-------|--------------------|---------------------|-------------|----------|
+| moonshotai-kimi-k2.6 | 5,115 | 1,270 | 6,385 | 3,926ms |
+| gemma-4-31b-it | 4,385 | 1,083 | 5,468 | 2,060ms |
+| qwen3.6-27b-nvfp4 | 4,191 | 992 | 5,183 | 7,391ms |
+| gpt-oss-120b | 3,300 | 804 | 4,104 | 7,310ms |
+
+**Prefill degradation @ conc=100:** All models lose ~80% of per-user prefill speed vs conc=10.
+
+## MSP Service-Tier Perspective (User Preference)
+
+When presenting for infrastructure/commercial decisions, structure per **service tier** not model comparison:
+- Single-User Experience (conc=1): TTFT p99, Output/User, SLA class
+- Multi-Tenant Load (conc=10→100): Throughput scaling, degradation curve
+- Steady-State: Long-run stability, drift detection
+- Service Recommendation: Positioning (Entry/Premium), suitable use cases
+
+## Related Files
+- `templates/aiperf-long-input-config.yaml` — Working starter template
+- `references/aiperf-suite-orchestrator-pattern.md` — Full orchestration pattern
+- `references/aiperf-token-metrics-explained.md` — Token metric unit reference
@@ -0,0 +1,109 @@
+# Prefill Throughput Bias from Cold-Start Requests
+
+## Problem
+
+AIPerf's per-run `profile_export_aiperf.json` aggregates ALL requests, including the very first one which has to establish a fresh HTTP connection. This biases the `prefill_throughput_per_user` metric significantly:
+
+| Request | Input Tokens | Prefill tok/s | TTFT | Connection |
+|---------|-------------|---------------|------|------------|
+| 0 | 7 | **4 tok/s** | 1699 ms | DNS (1002ms) + TCP (1144ms) |
+| 1 | 371 | 2003 tok/s | 185 ms | Reused ✓ |
+| 2 | 676 | 3596 tok/s | 188 ms | Reused ✓ |
+| 5 | 1488 | 7571 tok/s | 197 ms | Reused ✓ |
+
+The first request's DNS lookup + TCP handshake (2+ seconds) is counted toward "prefill time", deflating the metric. The **true** prefill throughput (requests 1+) is ~2K–7.5K tok/s, not the ~4K aggregate.
+
+## Impact
+
+- **Prefill metrics**: Inflated by ~1.5–2× for short runs (<20 requests). Improves as sample size grows.
+- **TTFT p99**: May be driven by the warm-up request's elevated TTFT.
+- **Low-concurrency runs** (conc=1, single session) are most affected because only ONE session exists and its first request is always cold.
+
+## Mitigation
+
+### Option A: Discard First Request (Recommended for Prescriptive Benchmarks)
+
+Parse `profile_export.jsonl` (line-delimited JSON, one request per line) and exclude `session_num=0` / `turn_index=0`:
+
+```python
+import json
+
+metrics = []
+with open("profile_export.jsonl") as f:
+    for line in f:
+        req = json.loads(line)
+        # Skip first turn of first session (cold-start)
+        if req["metadata"]["turn_index"] == 0 and req["metadata"]["session_num"] == 0:
+            continue
+        m = req["metrics"]
+        metrics.append({
+            "ttft": m["time_to_first_token"]["value"],
+            "prefill": m["prefill_throughput_per_user"]["value"],
+        })
+
+# Now metrics[] excludes the warm-up request
+```
+
+### Option B: Use Connection Warm-Up Before Benchmarking
+
+Send 1–2 throwaway requests to establish the TCP + TLS + HTTP/2 connection pool before the measured phase. AIPerf does not have a built-in warm-up phase, so this must be done externally:
+
+```bash
+# Pre-warm the connection
+venv/bin/aiperf profile --config warmup.yaml \
+    --concurrency 1 --request-count 1 \
+    --artifact-dir /tmp/warmup
+
+# Now run the real benchmark — connections are hot
+venv/bin/aiperf profile --config benchmark.yaml \
+    --concurrency 1 --request-count 50 \
+    --artifact-dir results/benchmark
+```
+
+The warmup run should use the **same endpoint, model, and streaming=true** as the benchmark to ensure the connection pool is pre-established.
+
+### Option C: Accept Bias for Comparative Benchmarks
+
+If comparing **N models on the same infra with the same settings**, the cold-start bias applies equally to all. The absolute numbers are slightly off, but **rankings and relative ratios remain valid**. Use this only when:
+- Absolute SLA numbers are not the goal
+- All runs use identical connection topology (same client → same endpoint)
+- Sample size is large enough that one cold request is <5% of total
+
+## When It Matters Most
+
+| Scenario | Severity | Why |
+|----------|----------|-----|
+| conc=1, sessions=1 | 🔴 High | First request is 100% of traffic |
+| conc=5, sessions=5 | 🟡 Medium | 1 of 5 sessions starts cold |
+| conc=50, sessions=50 | 🟢 Low | Many sessions, cold-start amortized |
+| Capacity (10→100+) | 🟢 Low | Many sessions, cold-start amortized |
+| Steady-state (10min) | 🟢 Low | Thousands of requests, negligible |
+
+## Per-Request Data Location
+
+Raw per-request data is in `profile_export.jsonl` (NOT `profile_export_aiperf.json` which is aggregated). Each line is a JSON object with:
+
+```json
+{
+  "metadata": {
+    "session_num": 0,
+    "turn_index": 0,
+    "worker_id": "worker_766ae0c2",
+    "conversation_id": "...",
+    "was_cancelled": false
+  },
+  "metrics": {
+    "time_to_first_token": {"value": 1699.2, "unit": "ms"},
+    "prefill_throughput_per_user": {"value": 4.1, "unit": "tokens/sec/user"},
+    "request_latency": {"value": 26372.4, "unit": "ms"},
+    "http_req_connection_reused": {"value": 0, "unit": "ratio"}
+  }
+}
+```
+
+Key indicators of a cold request:
+- `"http_req_connection_reused": {"value": 0}` — fresh connection
+- `"http_req_dns_lookup"` > 500ms — DNS resolution happened
+- `"http_req_connecting"` > 500ms — TCP/TLS handshake happened
+
+A reused connection has `"http_req_connection_reused": {"value": 1}` and near-zero DNS/connect times.
@@ -0,0 +1,120 @@
+# AIPerf Suite Orchestrator Pattern
+
+Reference: building a 12-run benchmark suite (4 models × 3 scenarios) against ai.noris.de.
+
+## Directory Convention
+
+```
+aiperf-benchmark-suite/
+├── configs/
+│   ├── gemma-4-31b-it/
+│   │   ├── sales.yaml      # ShareGPT dataset, requests: 50, sessions: 1
+│   │   ├── capacity.yaml   # Synthetic 512/128, requests: 100
+│   │   └── steady.yaml     # ShareGPT, duration: 600
+│   ├── gpt-oss-120b/...
+│   └── ... (one dir per model)
+├── scripts/
+│   ├── run-suite.sh        # Orchestrator
+│   ├── generate-summary.py # Markdown table from JSON
+│   └── plot-results.py     # Matplotlib charts + CSV
+├── results/
+│   └── YYYY-MM-DD-HHMMSS/
+│       ├── gemma-4-31b-it/
+│       │   ├── sales/conc1/
+│       │   ├── sales/conc5/
+│       │   ├── sales/conc10/
+│       │   ├── capacity/conc10/
+│       │   ├── capacity/conc25/
+│       │   ├── capacity/conc50/
+│       │   ├── capacity/conc100/
+│       │   └── steady/
+│       └── ...
+└── venv/                   # AIPerf installed here
+```
+
+## Orchestrator Structure
+
+The orchestrator is a bash script that:
+
+1. **Sets absolute paths** — `AIPERF="$BASE_DIR/venv/bin/aiperf"` (never rely on `$PATH`)
+2. **Validates env** — checks `NORIS_API_KEY` and connectivity
+3. **Loops models × scenarios**
+4. **Per scenario, loops concurrency levels** (for count-based scenarios)
+5. **Calls AIPerf with CLI overrides** for each sub-run
+6. **Pauses between runs** (`sleep 5-30`) to avoid rate limits
+7. **Generates summary + plots** after all runs complete
+8. **Continues on error** (`|| true`) — one failed sub-run doesn't kill the suite
+
+## Critical: Venv Path Fix
+
+When running in background (cron, detached shell, Telegram-triggered runs), `$PATH` does NOT include the venv. The orchestrator MUST:
+
+```bash
+AIPERF="$BASE_DIR/venv/bin/aiperf"
+if [ ! -x "$AIPERF" ]; then
+    echo "ERROR: AIPerf not found at $AIPERF"
+    exit 1
+fi
+# Then call "$AIPERF" profile ... everywhere
+```
+
+Failure mode: `aiperf: command not found` on every run, producing empty results silently.
+
+## CLI Override Pattern for Suite Runs
+
+The YAML configs define the base structure. CLI flags sweep parameters:
+
+```bash
+# Sales scenario: sweep concurrencies
+for conc in 1 5 10; do
+    "$AIPERF" profile --config "$CONFIG" \
+        --concurrency "$conc" \
+        --request-count 50 \
+        --artifact-dir "results/sales/conc${conc}"
+done
+
+# Capacity scenario: stepped ramp
+for conc in 10 25 50 100; do
+    "$AIPERF" profile --config "$CONFIG" \
+        --concurrency "$conc" \
+        --request-count 100 \
+        --artifact-dir "results/capacity/conc${conc}"
+done
+
+# Steady scenario: time-based
+"$AIPERF" profile --config "$CONFIG" \
+    --concurrency 20 \
+    --artifact-dir "results/steady"
+```
+
+## Pre-Download Strategy for ShareGPT
+
+First sales run downloads ShareGPT from HuggingFace (~2–5 min). To avoid this during production:
+
+```bash
+# One-time warm-up (any model config with public dataset works)
+venv/bin/aiperf profile --config configs/any-model/sales.yaml \
+    --concurrency 1 --request-count 1 \
+    --artifact-dir /tmp/warmup --no-gpu-telemetry --no-server-metrics
+```
+
+Subsequent runs use `.cache/aiperf/datasets/` instantly.
+
+## Post-Processing Sequence
+
+After the orchestrator finishes all runs:
+
+```bash
+# 1. Markdown summary
+python3 scripts/generate-summary.py results/YYYY-MM-DD-HHMMSS
+
+# 2. Plots + CSV
+python3 scripts/plot-results.py results/YYYY-MM-DD-HHMMSS
+```
+
+Outputs:
+- `results/.../README.md` – human-readable markdown table
+- `results/.../plots/summary.csv` – Excel / Google Sheets import
+- `results/.../plots/capacity_curve.png` – throughput vs concurrency
+- `results/.../plots/latency_curve.png` – TTFT p99 vs concurrency
+- `results/.../plots/sales_comparison.png` – low-concurrency latency
@@ -0,0 +1,82 @@
+# AIPerf Token-Metriken: Einheit & Interpretation
+
+AIPerf exportiert mehrere Metriken die alle "tokens/sec" im Namen tragen, aber unterschiedliche Einheiten haben. Das führt zu Fehlinterpretationen wenn man die falsche Metrik für die falsche Frage verwendet.
+
+## Metrik-Übersicht
+
+| Metrik (JSON-Key) | Einheit | Bedeutung |
+|-------------------|---------|-----------|
+| `prefill_throughput_per_user` | **tokens/sec/user** | Wie schnell der Prompt **eines Users** verarbeitet wird (Prompt → erstes Token) |
+| `output_token_throughput` | **tokens/sec** | **Gesamter Output-Durchsatz** aller parallelen Sessions zusammen |
+| `output_token_throughput_per_user` | **tokens/sec/user** | Output-Generierungsgeschwindigkeit, **pro einzelnem User** |
+| `e2e_output_token_throughput` | **tokens/sec/user** | End-to-End Throughput: Gesamtzeit (inkl. TTFT) / Output-Tokens, **pro User** |
+| `total_token_throughput` | **tokens/sec** | (Input + Output)-Tokens gesamt, alle Sessions |
+| `request_throughput` | **requests/sec** | Anzahl abgeschlossener Requests pro Sekunde |
+
+## Kritische Unterscheidung
+
+### „Per User" = Session-Rate
+Diese Metriken beschreiben das Erlebnis **eines einzelnen Endnutzers**:
+
+- `prefill_throughput_per_user: 505` → Jeder User bekommt seinen Prompt mit 505 tok/s verarbeitet
+- `output_token_throughput_per_user: 13.4` → Jeder User sieht ~13.4 neue Tokens pro Sekunde (die "Schreibgeschwindigkeit" des Modells)
+- `e2e_output_token_throughput: 11.9` → Inklusive Wartezeit bis zum ersten Token braucht der Request pro Output-Token länger
+
+**Wichtig:** Bei `prefill_throughput_per_user` ist der Einheitshinweis irreführend. Es ist keine Durchschnittsbildung über alle Users, sondern tatsächlich die Rate **pro Session**. Bei 100 parallelen Sessions mit je 505 tok/s benötigt der Node also insgesamt **50.500 tok/s Prefill-Kapazität** (was nicht direkt gemessen wird).
+
+### „Gesamt" = Node-Rate
+Diese Metriken beschreiben die **gesamte Leistung der Inferenz-Instanz**:
+
+- `output_token_throughput: 1083` → Alle 100 Sessions zusammen generieren 1083 Output-Tokens pro Sekunde
+- `total_token_throughput: 5468` → (Input + Output) zusammen → inklusive Prefill-Arbeit
+
+## Praktische Berechnungen
+
+### Input Token/s (nicht direkt gemessen)
+"Input" oder "Prefill" in Gesamtzahlen ist nicht separat exportiert. Annäherung:
+
+```
+input_tokens/sec ≈ total_token_throughput - output_token_throughput
+```
+
+Beispiel (gemma @ conc=100):
+- total: 5468 tok/s
+- output: 1083 tok/s
+- → input ≈ 4385 tok/s
+
+### Gesamtleistungs-Ranking
+Für Kapazitätsplanung (wie viele Tokens kann der Node pro Sekunde verarbeiten?) verwende `total_token_throughput`:
+
+| Modell @ conc=100 | Input t/s | Output t/s | **Total t/s** |
+|-------------------|-----------|------------|---------------|
+| moonshotai-kimi-k2.6 | ~5115 | 1270 | **6385** |
+| gemma-4-31b-it | ~4385 | 1083 | **5468** |
+| qwen3.6-27b-nvfp4 | ~4191 | 992 | **5183** |
+| gpt-oss-120b | ~3300 | 804 | **4104** |
+
+### Erlebnis-Ranking
+Für Kundenerfahrung (was spürt ein einzelner User?) verwende die _per_user-Metriken:
+
+| Modell @ conc=100 | Prefill/User | Output/User |
+|-------------------|-------------|-------------|
+| moonshotai-kimi-k2.6 | 276 | 17.9 |
+| qwen3.6-27b-nvfp4 | 226 | 15.7 |
+| gemma-4-31b-it | 505 | 13.4 |
+| gpt-oss-120b | 137 | 12.3 |
+
+## Fehlerquellen
+
+1. **Prefill mit Output vergleichen** – Prefill ist `tok/s/user`, Output-Gesamt ist `tok/s`. Ein Modell mit hohem `prefill_throughput_per_user` und niedrigem `output_token_throughput` ist ein "Prompt-Monster" aber ein "Generation-Schnarcher".
+
+2. **Gesamtleistung mit User-Erlebnis verwechseln** – Ein Node mit 10.000 tok/s Gesamtleistung bei 500 concurrent Sessions gibt jedem User nur 20 tok/s. Das ist unbrauchbar für Chat.
+
+3. **Input/Prefill vernachlässigen** – Bei RAG-Anwendungen (lange Contexte) dominiert der Prefill die Gesamtlaufzeit. Ein Modell mit 505 tok/s Prefill braucht für 4K Context ~8 Sekunden bis zum ersten Token.
+
+## Empfohlene KPIs pro Use-Case
+
+| Use Case | Primäre Metrik | Sekundäre Metrik | Warnschwelle |
+|----------|---------------|------------------|-------------|
+| Chat-UI (Echtzeit) | `output_token_throughput_per_user` | `ttft_p99` | < 20 tok/s pro User |
+| API-Burst (Batch) | `output_token_throughput` (gesamt) | `request_throughput` | Steigt nicht mehr mit Conc |
+| RAG / Lang-Context | `prefill_throughput_per_user` | `ttft_p99` | > 10s für 4K Input |
+| Kapazitätsplanung | `total_token_throughput` | `ttft_p99` | Degradation > 2× |
@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# Discover available models from an OpenAI-compatible endpoint
+
+API_KEY="${1:-${API_KEY}}"
+URL="${2:-https://ai.noris.de/v1}"
+
+if [ -z "$API_KEY" ]; then
+    echo "Usage: $0 <api_key> [endpoint_url]"
+    echo "   or: API_KEY=sk-... $0 [endpoint_url]"
+    exit 1
+fi
+
+curl -sf -H "Authorization: Bearer $API_KEY" "${URL%/}/models" | \
+    python3 -c "import sys,json; d=json.load(sys.stdin); [print(m['id']) for m in d.get('data',[])]"
@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+"""
+Post-process AIPerf suite results into management-ready deliverables.
+Generates: summary.csv, capacity_curve.png, latency_curve.png, sales_comparison.png
+"""
+import json
+import pathlib
+import sys
+from collections import defaultdict
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+
+
+def get_metric(data, key, stat="avg"):
+    m = data.get(key, {})
+    return m.get(stat) if isinstance(m, dict) else m
+
+
+def load_results(results_dir):
+    results = defaultdict(lambda: defaultdict(list))
+    for json_file in sorted(results_dir.rglob("profile_export_aiperf.json")):
+        rel = json_file.relative_to(results_dir)
+        parts = list(rel.parts)
+        model = parts[0]
+        scenario = parts[1]
+        conc_label = parts[2] if len(parts) > 2 else "steady"
+        with open(json_file) as f:
+            data = json.load(f)
+        results[model][scenario].append({
+            "conc_label": conc_label,
+            "requests": get_metric(data, "request_count", "avg"),
+            "ttft_avg_ms": get_metric(data, "time_to_first_token", "avg"),
+            "ttft_p99_ms": get_metric(data, "time_to_first_token", "p99"),
+            "throughput_tps": get_metric(data, "output_token_throughput", "avg"),
+            "itl_p99_ms": get_metric(data, "inter_token_latency", "p99"),
+            "prefill_tps": get_metric(data, "prefill_throughput_per_user", "avg"),
+            "duration_s": get_metric(data, "benchmark_duration", "avg"),
+        })
+    return results
+
+
+def write_csv(results, output_path):
+    rows = ["model,scenario,concurrency,ttft_avg_ms,ttft_p99_ms,throughput_tok_per_s,requests,duration_s,prefill_tok_per_s,itl_p99_ms"]
+    for model in sorted(results):
+        for scenario in ["sales", "capacity", "steady"]:
+            for r in sorted(results[model][scenario], key=lambda x: int(x["conc_label"].replace("conc","")) if x["conc_label"].startswith("conc") else 0):
+                rows.append(f"{model},{scenario},{r['conc_label']},{r['ttft_avg_ms'] or ''},{r['ttft_p99_ms'] or ''},{r['throughput_tps'] or ''},{r['requests'] or ''},{r['duration_s'] or ''},{r['prefill_tps'] or ''},{r['itl_p99_ms'] or ''}")
+    output_path.parent.mkdir(exist_ok=True)
+    output_path.write_text("\n".join(rows))
+    print(f"CSV: {output_path}")
+
+
+def plot_scenario(results, scenario, y_key, y_label, title, output_path, logscale=True):
+    fig, ax = plt.subplots(figsize=(10, 6))
+    for model in sorted(results):
+        runs = sorted(results[model][scenario], key=lambda x: int(x["conc_label"].replace("conc","")) if x["conc_label"].startswith("conc") else 0)
+        if not runs:
+            continue
+        concs = []
+        vals = []
+        for r in runs:
+            try:
+                c = int(r["conc_label"].replace("conc",""))
+            except ValueError:
+                c = r["conc_label"]
+            concs.append(c)
+            vals.append(r[y_key] or 0)
+        ax.plot(concs, vals, marker="o", linewidth=2, label=model)
+    ax.set_xlabel("Concurrency")
+    ax.set_ylabel(y_label)
+    ax.set_title(title, fontweight="bold")
+    ax.legend()
+    ax.grid(True, alpha=0.3)
+    if logscale:
+        ax.set_xscale("log")
+    fig.tight_layout()
+    fig.savefig(output_path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"Plot: {output_path}")
+
+
+def main():
+    if len(sys.argv) < 2:
+        print("Usage: plot-suite-results.py <results-dir>")
+        sys.exit(1)
+    results_dir = pathlib.Path(sys.argv[1])
+    results = load_results(results_dir)
+    out = results_dir / "plots"
+    out.mkdir(exist_ok=True)
+    write_csv(results, out / "summary.csv")
+    plot_scenario(results, "capacity", "throughput_tps", "Throughput (tok/s)", "Capacity: Throughput vs Concurrency", out / "capacity_curve.png")
+    plot_scenario(results, "capacity", "ttft_p99_ms", "TTFT p99 (ms)", "Capacity: TTFT p99 vs Concurrency", out / "latency_curve.png")
+    plot_scenario(results, "sales", "ttft_p99_ms", "TTFT p99 (ms)", "Sales: Latency at Low Concurrency", out / "sales_comparison.png", logscale=False)
+    print(f"\nDone. Outputs in {out}")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,67 @@
+# AIPerf 3-Scenario Suite Template
+
+Reusable structure for benchmarking N models with Sales + Capacity + Steady-State.
+
+## Directory Layout
+
+```
+aiperf-benchmark-suite/
+├── configs/
+│   └── <model-safe-name>/
+│       ├── sales.yaml
+│       ├── capacity.yaml
+│       └── steady.yaml
+├── templates/
+│   └── base-config.yaml
+├── scripts/
+│   ├── run-suite.sh
+│   └── generate-summary.py
+├── results/          # gitignored
+└── README.md
+```
+
+## Base Config Template
+
+```yaml
+url: "https://<endpoint>/v1"
+endpoint_type: "chat"
+extra-headers:
+  Authorization: "Bearer ${API_KEY}"
+streaming: true
+tokenizer: "builtin"
+random_seed: 42
+```
+
+## Scenario Definitions
+
+### Sales
+- Concurrency: 1, 5, 10 (sequentially)
+- Request count: 50 per level
+- Dataset: `sharegpt`
+- Display: `simple` or `dashboard`
+- Flags: `--concurrency N --request-count 50 --dataset sharegpt`
+
+### Capacity
+- Concurrency: 10, 25, 50, 100 (stepped ramp)
+- Request count: 100 per level
+- Dataset: synthetic fixed lengths
+- Flags: `--concurrency N --request-count 100 --input-tokens-mean 512 --output-tokens-mean 128`
+
+### Steady-State
+- Duration: 600 seconds
+- Concurrency: sweet spot from Capacity (e.g., 20)
+- Dataset: mixed or `sharegpt`
+- Flags: `--concurrency 20 --time 600 --dataset sharegpt`
+
+## Orchestrator Pattern
+
+Iterate models × scenarios, with `--output $RESULTS_DIR/$MODEL/$SCENARIO/conc${CONC}`.
+Pause 30s between major runs to avoid rate limits.
+
+## Result Summary
+
+After all runs, parse `profile_export_aiperf.json` files to generate markdown table:
+
+| Model | Scenario | Concurrency | TTFT avg | TTFT p99 | Throughput |
+
+See `scripts/generate-summary.py` for a reference implementation.
@@ -0,0 +1,36 @@
+# AIPerf 0.10.0+ Schema 2.0 — Extreme Context Test Template (10k Input / 10k Output)
+# Use case: Stress-test context-window limits and sustained decode performance
+# Concurrency is 1 because runtime is 200–1000 seconds per request
+
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+  endpoint:
+    urls: ["https://ai.noris.de/v1"]
+    type: "chat"
+    streaming: true
+  datasets:
+    - type: "synthetic"
+      name: "main"
+      entries: 10
+      isl:
+        type: "fixed"
+        value: 10000
+      osl:
+        type: "fixed"
+        value: 10000
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      requests: 10
+
+# CLI usage (per model):
+#   aiperf profile --config this-file.yaml --concurrency 1 --ui none \
+#       --api-key "$API_KEY" --artifact-dir results/10k/MODEL_NAME/
+
+# Expected runtime:
+#   At 10–50 tok/s output, one request = 200–1000 seconds
+#   10 requests sequentially = ~30–60 minutes per model
+#   Schedule with no-concurrency (--concurrency 1) to avoid timeouts
@@ -0,0 +1,37 @@
+# AIPerf 0.10.0+ Schema 2.0 — Long-Input Prefill Test Template
+# Use case: Isolate memory-bandwidth prefill bottleneck for RAG / long-context workloads
+# Concurrency is kept low (10) to avoid queueing artifacts
+
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "moonshotai/kimi-k2.6"
+  endpoint:
+    urls: ["https://ai.noris.de/v1"]
+    type: "chat"
+    streaming: true
+  datasets:
+    - name: "synthetic_long"
+      type: "synthetic"
+      prompt_template:
+        content: "Analyze the following topic in depth and provide detailed insights."
+      strategy:
+        type: "fixed"
+        max_total_tokens: 1152
+        input_tokens: 1024   # Double the standard 512
+        output_tokens: 128   # Short output to isolate prefill
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      duration: 300          # 5 minutes to amortize cold-start
+      # Do NOT specify requests — duration drives the phase
+
+# CLI usage:
+#   aiperf profile --config this-file.yaml --concurrency 10 --ui none \
+#       --api-key "$API_KEY" --artifact-dir results/longinput/
+
+# YAML rules learned the hard way:
+#   - datasets is a LIST (starts with "- name:", NOT "default:")
+#   - prompt_template and strategy are siblings to name/type
+#   - Never use ${VAR} in headers when passing --api-key via CLI
@@ -0,0 +1,39 @@
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/MODEL-NAME-HERE"
+    strategy: "round_robin"
+  endpoint:
+    urls:
+      - "https://ENDPOINT/v1"
+    type: "chat"
+    streaming: true
+    headers:
+      Authorization: "Bearer ${API_KEY}"
+  # --- Option A: Public dataset (ShareGPT) ---
+  datasets:
+    - type: "public"
+      name: "main"
+      dataset: "sharegpt"
+      sampling: "shuffle"
+  # --- Option B: Synthetic dataset (no download) ---
+  # datasets:
+  #   - type: "synthetic"
+  #     name: "main"
+  #     entries: 100
+  #     isl:
+  #       type: "fixed"
+  #       value: 512
+  #     osl:
+  #       type: "fixed"
+  #       value: 128
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      # Choose ONE: requests OR duration
+      requests: 50          # for count-based (sales, capacity)
+      sessions: 1           # concurrency level (overridden by --concurrency CLI flag)
+      # duration: 600       # for time-based (steady-state)
+  tokenizer:
+    name: "builtin"
@@ -0,0 +1,139 @@
+# AIPerf 3-Scenario Benchmark Suite Template
+# Copy and customize per target endpoint.
+# Usage: aiperf profile -c <this-file>
+#
+# Environment variables expected:
+#   API_KEY       - Bearer token for the target endpoint
+#   API_BASE_URL  - e.g. https://ai.noris.de
+
+# ═══════════════════════════════════════════════════════════════
+# SCENARIO 1: Sales (Representative Load)
+# ═══════════════════════════════════════════════════════════════
+scenarios:
+  - name: sales-low
+    model: vllm/moonshotai/kimi-k2.6
+    url: "${API_BASE_URL}"
+    endpoint-type: chat
+    streaming: true
+    tokenizer: moonshotai/kimi-k2.6
+    concurrency: 1
+    request-count: 50
+    dataset: sharegpt
+    extra-headers:
+      Authorization: "Bearer ${API_KEY}"
+    output-directory: "results/${DATE}/sales/concurrency-1"
+    ui: simple
+
+  - name: sales-moderate
+    model: vllm/moonshotai/kimi-k2.6
+    url: "${API_BASE_URL}"
+    endpoint-type: chat
+    streaming: true
+    tokenizer: moonshotai/kimi-k2.6
+    concurrency: 5
+    request-count: 50
+    dataset: sharegpt
+    extra-headers:
+      Authorization: "Bearer ${API_KEY}"
+    output-directory: "results/${DATE}/sales/concurrency-5"
+    ui: simple
+
+  - name: sales-realistic
+    model: vllm/moonshotai/kimi-k2.6
+    url: "${API_BASE_URL}"
+    endpoint-type: chat
+    streaming: true
+    tokenizer: moonshotai/kimi-k2.6
+    concurrency: 10
+    request-count: 50
+    dataset: sharegpt
+    extra-headers:
+      Authorization: "Bearer ${API_KEY}"
+    output-directory: "results/${DATE}/sales/concurrency-10"
+    ui: simple
+
+# ═══════════════════════════════════════════════════════════════
+# SCENARIO 2: Capacity (Breakpoint Discovery)
+# ═══════════════════════════════════════════════════════════════
+  - name: capacity-10
+    model: vllm/moonshotai/kimi-k2.6
+    url: "${API_BASE_URL}"
+    endpoint-type: chat
+    streaming: true
+    tokenizer: moonshotai/kimi-k2.6
+    concurrency: 10
+    request-count: 100
+    dataset: synthetic
+    input-tokens-mean: 512
+    output-tokens-mean: 128
+    extra-headers:
+      Authorization: "Bearer ${API_KEY}"
+    output-directory: "results/${DATE}/capacity/concurrency-10"
+    ui: none
+
+  - name: capacity-25
+    model: vllm/moonshotai/kimi-k2.6
+    url: "${API_BASE_URL}"
+    endpoint-type: chat
+    streaming: true
+    tokenizer: moonshotai/kimi-k2.6
+    concurrency: 25
+    request-count: 100
+    dataset: synthetic
+    input-tokens-mean: 512
+    output-tokens-mean: 128
+    extra-headers:
+      Authorization: "Bearer ${API_KEY}"
+    output-directory: "results/${DATE}/capacity/concurrency-25"
+    ui: none
+
+  - name: capacity-50
+    model: vllm/moonshotai/kimi-k2.6
+    url: "${API_BASE_URL}"
+    endpoint-type: chat
+    streaming: true
+    tokenizer: moonshotai/kimi-k2.6
+    concurrency: 50
+    request-count: 100
+    dataset: synthetic
+    input-tokens-mean: 512
+    output-tokens-mean: 128
+    extra-headers:
+      Authorization: "Bearer ${API_KEY}"
+    output-directory: "results/${DATE}/capacity/concurrency-50"
+    ui: none
+
+  - name: capacity-100
+    model: vllm/moonshotai/kimi-k2.6
+    url: "${API_BASE_URL}"
+    endpoint-type: chat
+    streaming: true
+    tokenizer: moonshotai/kimi-k2.6
+    concurrency: 100
+    request-count: 100
+    dataset: synthetic
+    input-tokens-mean: 512
+    output-tokens-mean: 128
+    extra-headers:
+      Authorization: "Bearer ${API_KEY}"
+    output-directory: "results/${DATE}/capacity/concurrency-100"
+    ui: none
+
+# ═══════════════════════════════════════════════════════════════
+# SCENARIO 3: Steady-State (Long-Term Stability)
+# ═══════════════════════════════════════════════════════════════
+  - name: steady-state
+    model: vllm/moonshotai/kimi-k2.6
+    url: "${API_BASE_URL}"
+    endpoint-type: chat
+    streaming: true
+    tokenizer: moonshotai/kimi-k2.6
+    concurrency: 20
+    duration: 600
+    dataset: sharegpt
+    extra-headers:
+      Authorization: "Bearer ${API_KEY}"
+    output-directory: "results/${DATE}/steady-state"
+    ui: none
+    # Enable timeslice metrics for drift detection:
+    # timeslice: 60
@@ -0,0 +1,37 @@
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/REPLACE-MODEL-NAME"
+  endpoint:
+    urls: ["https://ENDPOINT/v1"]
+    type: "chat"
+    streaming: true
+  datasets:
+    default:
+      - name: "synthetic_long"
+        type: "synthetic"
+        prompt_template:
+          content: "Summarize the following text concisely: {{text}}"
+        variables:
+          - name: "text"
+            values: [
+              "artificial intelligence",
+              "quantum computing",
+              "blockchain technology",
+              "climate change",
+              "machine learning",
+              "renewable energy"
+            ]
+    strategy:
+      type: "fixed"
+      max_total_tokens: 1152    # input + output + padding
+      input_tokens: 1024        # TARGET INPUT LENGTH: sweep 512 / 1024 / 2048 / 4096
+      output_tokens: 128
+  phases:
+    - type: "concurrency"
+      name: "prefill_stress"
+      duration: 300             # 5 minutes
+      sessions: 10              # fixed low concurrency
+  tokenizer:
+    name: "builtin"