hermes-skills/mlops/llm-inference-benchmarking/SKILL.md

---
name: llm-inference-benchmarking
description: "Benchmark LLM inference endpoints for throughput, latency, and capacity. Covers TTFT, ITL, concurrency scaling, and load patterns using AIPerf or equivalent tools."
---

# LLM Inference Benchmarking

Measure the performance of generative AI inference endpoints under realistic and stress loads. This skill covers inference performance (speed, throughput, concurrency limits), NOT accuracy benchmarks like MMLU or GSM8K—use evaluating-llms-harness for those.

## Trigger

Use when:
- Setting up performance benchmarks for LLM APIs (OpenAI-compatible, vLLM, TGI, etc.)
- Determining capacity limits ("how many concurrent users can this model handle?")
- Producing sales/reliability metrics (latency percentiles, throughput numbers)
- Comparing model deployments or hardware configurations
- Regression testing inference endpoints after updates

## Primary Tool: AIPerf

AIPerf (NVIDIA/ai-dynamo) is the recommended tool. **Version 0.10.0+ uses a new Schema 2.0 YAML format**—see Config Format section below.

It supports:
- OpenAI chat/completions/embeddings/audio/vision endpoints
- Concurrency, request-rate, trace-replay, and ramping modes
- ShareGPT and custom datasets
- Real-time dashboard, CSV/JSON exports, and automatic plotting
- Multiprocess architecture via ZMQ

Installation:
```bash
pip install aiperf
# Optional extras:
pip install "aiperf[wandb]"   # historical trend tracking
pip install "aiperf[otel]"    # live telemetry streaming
```

Platform notes:
- **Linux aarch64**: `crick` dependency needs build tools: `sudo apt install build-essential`
- **Older x86_64 VMs without X86_V2**: NumPy 2.x crashes with `RuntimeError: NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support: (X86_V2)`. Fix: `pip install "numpy<2.0" --force-reinstall`. This applies to older cloud VMs, Proxmox containers, or legacy hardware that lacks AVX2/BMI2/MOVBE/POPCNT/FMA.

## AIPerf 0.10.0+ Config Format (Schema 2.0)

> **Critical**: AIPerf 0.10.0 introduced a new YAML envelope. Old flat configs are rejected.

### Envelope Structure
```yaml
schema_version: "2.0"
benchmark:
  models:
    items:
      - name: "vllm/gemma-4-31b-it"
    strategy: "round_robin"
  endpoint:
    urls:
      - "https://ai.noris.de/v1"
    type: "chat"
    api_key: "${API_KEY}"          # Can reference env var; plaintext also works
    streaming: true
    headers:
      Authorization: "Bearer ${API_KEY}"
  datasets:
    - type: "synthetic"            # or "public" for ShareGPT
      name: "main"
      entries: 100
      isl:
        type: "fixed"
        value: 512
      osl:
        type: "fixed"
        value: 128
  phases:
    - type: "concurrency"
      name: "profiling"
      # One of requests, duration, or sessions MUST be set
      requests: 50                  # total requests for this phase
      sessions: 1                   # concurrent sessions (default: 1)
      # duration: 600               # alternative to requests
  tokenizer:
    name: "builtin"                 # or huggingface model name
```

### Key fields explained
- `models.items[].name`: The model identifier (must match endpoint's `/v1/models` list)
- `endpoint.api_key`: Can be `${ENV_VAR}` reference. CLI `--api-key` overrides this.
- `datasets.type`: `"public"` downloads ShareGPT from HuggingFace (first run ~2–5 min). `"synthetic"` generates locally with no download.
- `datasets.isl` / `datasets.osl`: Input/Output Sequence Length distributions. `type: "fixed"` with `value: N` for deterministic lengths.
- `phases`: Must specify at least one of `requests`, `duration`, or `sessions`. Default `concurrency` inside phase is 1.

### CLI Overrides
AIPerf allows overriding YAML values at CLI time. The most useful for suite orchestration:

| CLI flag | Overrides YAML field | Typical use |
|----------|----------------------|-------------|
| `--concurrency N` | phase.sessions (indirectly) | Sweep concurrency levels |
| `--request-count N` | phase.requests | Control total requests per sub-run |
| `--duration N` | phase.duration | Time-based runs (steady-state) |
| `--artifact-dir PATH` | output directory | Organize results per run |
| `--ui simple` | dashboard mode | Less overhead, no TUI |
| `--no-gpu-telemetry` | - | Skip DCGM (endpoints without GPU metrics) |
| `--no-server-metrics` | - | Skip Prometheus scraping (non-vLLM endpoints) |

**Orchestration pattern**: Define base structure in YAML, then use CLI flags in a shell loop for parameter sweeps:
```bash
for conc in 1 5 10 25 50 100; do
  aiperf profile --config capacity.yaml --concurrency "$conc" --request-count 100 \
    --artifact-dir "results/capacity/conc${conc}"
done
```

## The 3-Scenario Suite Pattern

Design benchmark suites around three complementary scenarios:

### 1. Sales Scenario (Representative Load)
- **Goal**: Clean numbers for pitch decks and SLAs
- **Concurrency**: 1, 5, 10 (low, moderate, realistic)
- **Dataset**: ShareGPT (realistic chat transcripts)
- **Count**: ~50 requests per run
- **Output**: avg/p99 TTFT, throughput, request latency

### 2. Capacity Scenario (Breakpoint Discovery)
- **Goal**: Find the concurrency ceiling before degradation
- **Concurrency**: 10 → 25 → 50 → 100 (stepped ramp)
- **Dataset**: Synthetic with fixed ISL/OSL (e.g., 512 IN / 128 OUT)
- **Count**: ~100 requests per level
- **Output**: Performance curve (throughput vs. concurrency, latency inflection points)

**When to stop scaling concurrency**: Before proposing conc=200+ tests, extrapolate the existing curve. If TTFT p99 exceeds ~5s at conc=100, further scaling is operationally irrelevant—users won't accept 15s+ wait times. Better to test **longer inputs** (1024/2048 tokens) to isolate the memory-bandwidth prefill bottleneck, or implement queueing + load balancing.

### 2b. Long-Context Prefill Test (Memory Bandwidth Bottleneck)
- **Goal**: Isolate the memory-bandwidth prefill bottleneck (relevant for RAG, long-context agents)
- **Concurrency**: Fixed low (e.g., 10) to avoid queueing artifacts
- **Dataset**: Synthetic with **long input** (1024 / 2048 / 4096 tokens), short output (128)
- **Duration**: 300s (long enough to amortize cold-start)
- **Output**: Prefill throughput vs. input length curve

This is more informative than conc=200 when the real workload involves reading long documents.

### 3. Steady-State Scenario (Long-Term Stability)
- **Goal**: Detect drift, memory leaks, or throttling over time
- **Concurrency**: Sweet spot from Capacity scenario (e.g., 20)
- **Duration**: 600 seconds (10 minutes) or longer
- **Dataset**: Mixed (50% ShareGPT + 50% synthetic short/long)
- **Output**: P50/P90/P99 trends over time, failed request rate

### Universal Tokenizer for Heterogeneous Endpoints

When benchmarking multiple models with different vocabularies (and downloading the correct HuggingFace tokenizer for each is impractical), use:

```bash
--tokenizer builtin
```

This uses tiktoken's `o200k_base` encoding universally. It avoids HF downloads, gated-repo access issues, and rate limits. **Trade-off:** Token counts are approximate for non-OpenAI models (different vocabularies), but **TTFT, ITL, and latency metrics remain exact**. Recommended when:
- Tokenizer mapping is complicated (4+ models, gated repos, missing HF entries)
- Running in air-gapped or rate-limited environments
- Sales/capacity benchmark where relative comparisons matter more than absolute token counts

Before benchmarking, discover available models from the target endpoint:

```bash
curl -s -H "Authorization: Bearer $API_KEY" \
  https://<endpoint>/v1/models | \
  python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin).get('data',[])]"
```

See `scripts/discover-models.sh` for a ready-to-run version.

## Authentication & Security

- **NEVER hardcode API keys** in YAML configs, shell scripts, or git
- Always use environment variables: `${API_KEY}` or `$API_KEY`
- In AIPerf YAML, reference via `--extra-headers` or templating:
  ```yaml
  extra-headers:
    Authorization: "Bearer ${API_KEY}"
  ```
- Export in shell before running: `export API_KEY=sk-...`

## AIPerf Quick Reference

Basic profile run:
```bash
aiperf profile \
  --model "vllm/qwen3.6-27b-nvfp4" \
  --streaming \
  --endpoint-type chat \
  --tokenizer <huggingface-tokenizer-name> \
  --url https://ai.noris.de \
  --extra-headers "Authorization: Bearer ${API_KEY}" \
  --concurrency 10 \
  --request-count 50
```

Key flags:
- `--concurrency N` – parallel requests
- `--request-rate N` – requests per second (alternative to concurrency)
- `--request-count N` – total requests to send
- `--duration N` – time-based run (seconds)
- `--streaming` – measure streaming metrics (TTFT, ITL)
- `--output-tokens-mean N` – target output length (with caveats; see AIPerf docs)
- `--dataset <file>` – custom dataset (ShareGPT format or synthetic)
- `--random-seed N` – reproducibility

## YAML Configs

AIPerf supports YAML-driven runs for reproducibility. See `templates/aiperf-suite-config.yaml` for a complete 3-scenario suite template.

Run from YAML:
```bash
aiperf profile -c configs/sales-scenario.yaml
```

## Results & Visualization

AIPerf produces:
- **CSV** – per-request metrics (latency, token counts, timestamps)
- **JSON** – aggregated statistics with percentile data (`avg`, `min`, `max`, `p50`, `p90`, `p99`, `std`)
- **Logs** – `aiperf.log` with detailed debug info
- **Plots** – PNG charts via `aiperf plot <results-dir>`

Results should be organized as:
```
results/YYYY-MM-DD/
  sales/
  capacity/
  steady-state/
```

### Custom Post-Processing for Multi-Model Suites

For sales/capacity comparison across multiple models and concurrency levels, AIPerf's built-in `plot` subcommand works on single runs. Use a custom Python post-processing script to generate cross-model comparison charts:

1. **Load all** `profile_export_aiperf.json` files recursively from `results/YYYY-MM-DD/`
2. **Extract key metrics** per `(model, scenario, concurrency)`:
   - `output_token_throughput.avg` = throughput (tok/s) **TOTAL**
   - `time_to_first_token.p99` = tail latency (ms)
   - `inter_token_latency.p99` = streaming smoothness (ms)
   - `prefill_throughput_per_user.avg` = prefill speed **per user session**
   - `output_token_throughput_per_user.avg` = output speed **per user**
   - `total_token_throughput.avg` = (input+output) tok/s **TOTAL** across all sessions
3. **Generate plots** with matplotlib:
   - `capacity_curve.png` – Throughput vs Concurrency (log scale on x-axis)
   - `latency_curve.png` – TTFT p99 vs Concurrency (log scale on x-axis)
   - `sales_comparison.png` – Latency at low concurrency (1/5/10)
4. **Export** `summary.csv` for Excel / Google Sheets import

A working implementation: `scripts/plot-suite-results.py`

**Rule:** `*_per_user` metrics are for end-user SLA/UX. Total metrics (`output_token_throughput`, `total_token_throughput`) are for infrastructure capacity planning. Input throughput is approximately `total - output`.

### MSP / Service Provider Perspective

When presenting results for infrastructure or commercial decision-making (not model research):

**Don't:** Compare Model A vs Model B on token-per-token quality.  
**Do:** Report each model as an independent service tier with its own SLA envelope.

**Structure the analysis per model:**
- **Single-User Experience** (conc=1): TTFT p99, Output/User, SLA class (Gold/Silver/Bronze)
- **Multi-Tenant Load** (conc=10→100): Throughput scaling, TTFT degradation, recommended max concurrency
- **Steady-State**: Long-run stability, completed requests, throughput drift
- **Service Recommendation**: Positioning (Entry/Premium), suitable use cases, limitations

**Key metrics by business question:**

| Business Question | Metric to Use | Why |
|-------------------|---------------|-----|
| "How many users per node?" | `total_token_throughput` (Input + Output) | Node capacity |
| "What does each user feel?" | `output_token_throughput_per_user` | UX/SLA |
| "Is chat real-time responsive?" | `ttft_p99` + `itl_p99` | Interaktivität |
| "Can we handle batch jobs?" | `request_throughput` + `output_token_throughput` | Durchsatz |
| "What breaks first under load?" | TTFT p99 scaling curve | Bottleneck |

**Conc ceiling analysis:** Before proposing conc=200+ tests, extrapolate the existing capacity curve. If TTFT p99 already exceeds 7s at conc=100, further scaling is operationally irrelevant—users won't accept 15s+ wait times. Better to test longer inputs (1024/2048/4096 tokens) to find the prefill bottleneck, or implement queueing + load balancing.

See `references/aiperf-token-metrics-explained.md` for the full unit-reference table.

## Known Pitfalls

1. **AIPerf 0.10.0+ Config Format** — The new `schema_version: "2.0"` envelope is required. Old flat configs will be rejected with `1 validation error for AIPerfConfig`. See Config Format section above. Always include `requests`, `duration`, or `sessions` in phases.
2. **NumPy X86_V2 crash on older x86_64** — If you see `RuntimeError: NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support: (X86_V2)`, downgrade: `pip install "numpy<2.0" --force-reinstall`
3. **ShareGPT first-run delay** — The `public` dataset type downloads ShareGPT from HuggingFace on first use (~2–5 minutes). Subsequent runs use a local cache. For quick validation, use `type: "synthetic"`. For production suites, **pre-download once** before the full orchestrated run:
   ```bash
   venv/bin/aiperf profile --config configs/any-model/sales.yaml \
       --concurrency 1 --request-count 1 \
       --artifact-dir /tmp/warmup --no-gpu-telemetry --no-server-metrics
   ```
   After this, `.cache/aiperf/datasets/` contains ShareGPT and all sales runs start instantly.
4. **Tokenizer mismatch** — Always specify the correct tokenizer. AIPerf auto-detects for some models, but explicit is safer.
5. **`--output-tokens-mean` is not guaranteed** — Unless the server supports `ignore_eos` and `min_tokens` via `--extra-inputs`.
6. **Rate limiting** — AIPerf has built-in retry/backoff, but aggressive concurrency on shared endpoints may trigger 429s. Monitor `failed_request_count`.
7. **Port exhaustion** — Very high concurrency (>15,000) can exhaust local ports. Reduce or adjust system limits.
10. **Orchestrator path resolution** — When running benchmarks in background processes (cron, systemd, detached shells), `PATH` may not include a Python venv. Always use the absolute path: `AIPERF="$BASE_DIR/venv/bin/aiperf"` in shell scripts, never rely on `aiperf` being in `$PATH`. The failure mode is silent: every run produces `aiperf: command not found` but the orchestrator continues, producing empty results.
    See `references/aiperf-background-process-pitfalls.md`.
11. **Prefill throughput bias from cold-start requests** — The first request in any AIPerf session includes DNS resolution + TCP/TLS handshake time, which is counted toward "prefill throughput". At conc=1 this can deflate prefill metrics by 50%+. The aggregate `profile_export_aiperf.json` cannot filter this out. Use `profile_export.jsonl` (per-request data) to discard `turn_index=0` requests, or pre-warm connections with a throwaway run before the measured benchmark.
    See `references/aiperf-prefill-warmup-bias.md`.
12. **Token metric unit confusion** — AIPerf exports multiple "tokens/sec" metrics with different units (per-user vs. total). `prefill_throughput_per_user` is tok/s **per session**, while `output_token_throughput` is tok/s **total across all sessions**. Comparing them directly or using the wrong one for capacity planning leads to off-by-100× errors.
    - Use `*_per_user` metrics for **end-user experience** (SLA, UX)
    - Use `output_token_throughput` / `total_token_throughput` for **node capacity** (infrastructure sizing)
    - Approximate input throughput: `total - output`
    See `references/aiperf-token-metrics-explained.md`.

## Orchestration

For multi-scenario suites, wrap AIPerf calls in a shell script that:
1. Validates connectivity and auth (health check)
2. Runs scenarios sequentially
3. Organizes results in timestamped folders
4. Generates plots after each run
5. Detects failures (exit codes, empty CSVs, high error rates) and continues

## Session References

- `references/aiperf-0.10.0-migration.md` — complete config format migration guide with validation error fixes
- `references/aiperf-long-input-benchmarking.md` — long-input prefill bottleneck testing (replaces conc=200+ tests when memory bandwidth is the real limiter)
- `templates/aiperf-long-input-config.yaml` — working Schema 2.0 template for 1024-token input tests
- `references/aiperf-token-metrics-explained.md` — unit reference table: per-user vs. total tok/s, input throughput approximation
- `references/aiperf-suite-orchestrator-pattern.md` — complete suite orchestrator pattern: directory layout, venv path fix, CLI sweep pattern, ShareGPT pre-download, post-processing sequence
- `references/aiperf-prefill-warmup-bias.md` — cold-start bias in prefill metrics, mitigation options, per-request JSONL filtering
- `references/aiperf-token-metrics-explained.md` — unit reference: per-user vs total metrics, input calculation, business-KPI mapping
- `references/aiperf-capacity-curve-analysis.md` — interpreting capacity curves: when to stop scaling concurrency, breakpoint detection, extrapolation rules
- `references/aiperf-long-context-prefill-test.md` — isolating the memory-bandwidth bottleneck with 1024/2048/4096-token inputs
- `references/ai-noris-de-endpoint.md` — discovered model list, URL, auth pattern for ai.noris.de
- `references/aiperf-kurzreferenz.md` — German quick reference summary
- `templates/aiperf-suite-config-v2.yaml` — Schema 2.0 starter template with all options documented
- `templates/long-context-prefill-test.yaml` — 1024-token input test template for RAG/memory-bandwidth analysis
- `templates/3-scenario-suite.md` — reusable Sales/Capacity/Steady-State suite structure

- **GenAI-Perf** (NVIDIA, predecessor to AIPerf) – AIPerf is the migration target
- **locust + OpenAI client** – custom Python load testing, more flexible but more work
- **k6 + xk6-openai** – if already using k6 for HTTP load testing

- **GenAI-Perf** (NVIDIA, predecessor to AIPerf) – AIPerf is the migration target
- **locust + OpenAI client** – custom Python load testing, more flexible but more work
- **k6 + xk6-openai** – if already using k6 for HTTP load testing

## Related Skills

- evaluating-llms-harness – for accuracy benchmarks (MMLU, GSM8K, etc.)
- serving-llms-vllm – for setting up the inference server being benchmarked