hermes-skills/mlops/llm-inference-benchmarking/SKILL.md

name, description

name	description
llm-inference-benchmarking	Benchmark LLM inference endpoints for throughput, latency, and capacity. Covers TTFT, ITL, concurrency scaling, and load patterns using AIPerf or equivalent tools.

LLM Inference Benchmarking

Measure the performance of generative AI inference endpoints under realistic and stress loads. This skill covers inference performance (speed, throughput, concurrency limits), NOT accuracy benchmarks like MMLU or GSM8K—use evaluating-llms-harness for those.

Trigger

Use when:

• Setting up performance benchmarks for LLM APIs (OpenAI-compatible, vLLM, TGI, etc.)
• Determining capacity limits ("how many concurrent users can this model handle?")
• Producing sales/reliability metrics (latency percentiles, throughput numbers)
• Comparing model deployments or hardware configurations
• Regression testing inference endpoints after updates

Primary Tool: AIPerf

AIPerf (NVIDIA/ai-dynamo) is the recommended tool. Version 0.10.0+ uses a new Schema 2.0 YAML format—see Config Format section below.

It supports:

• OpenAI chat/completions/embeddings/audio/vision endpoints
• Concurrency, request-rate, trace-replay, and ramping modes
• ShareGPT and custom datasets
• Real-time dashboard, CSV/JSON exports, and automatic plotting
• Multiprocess architecture via ZMQ

Installation:

pip install aiperf
# Optional extras:
pip install "aiperf[wandb]"   # historical trend tracking
pip install "aiperf[otel]"    # live telemetry streaming

Platform notes:

• Linux aarch64: crick dependency needs build tools: sudo apt install build-essential
• Older x86_64 VMs without X86_V2: NumPy 2.x crashes with RuntimeError: NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support: (X86_V2). Fix: pip install "numpy<2.0" --force-reinstall. This applies to older cloud VMs, Proxmox containers, or legacy hardware that lacks AVX2/BMI2/MOVBE/POPCNT/FMA.

AIPerf 0.10.0+ Config Format (Schema 2.0)

Critical: AIPerf 0.10.0 introduced a new YAML envelope. Old flat configs are rejected.

Envelope Structure

schema_version: "2.0"
benchmark:
  models:
    items:
      - name: "vllm/gemma-4-31b-it"
    strategy: "round_robin"
  endpoint:
    urls:
      - "https://ai.noris.de/v1"
    type: "chat"
    api_key: "${API_KEY}"          # Can reference env var; plaintext also works
    streaming: true
    headers:
      Authorization: "Bearer ${API_KEY}"
  datasets:
    - type: "synthetic"            # or "public" for ShareGPT
      name: "main"
      entries: 100
      isl:
        type: "fixed"
        value: 512
      osl:
        type: "fixed"
        value: 128
  phases:
    - type: "concurrency"
      name: "profiling"
      # One of requests, duration, or sessions MUST be set
      requests: 50                  # total requests for this phase
      sessions: 1                   # concurrent sessions (default: 1)
      # duration: 600               # alternative to requests
  tokenizer:
    name: "builtin"                 # or huggingface model name

Key fields explained

• models.items[].name: The model identifier (must match endpoint's /v1/models list)
• endpoint.api_key: Can be ${ENV_VAR} reference. CLI --api-key overrides this.
• datasets.type: "public" downloads ShareGPT from HuggingFace (first run ~2–5 min). "synthetic" generates locally with no download.
• datasets.isl / datasets.osl: Input/Output Sequence Length distributions. type: "fixed" with value: N for deterministic lengths.
• phases: Must specify at least one of requests, duration, or sessions. Default concurrency inside phase is 1.

CLI Overrides

AIPerf allows overriding YAML values at CLI time. The most useful for suite orchestration:

CLI flag	Overrides YAML field	Typical use
--concurrency N	phase.sessions (indirectly)	Sweep concurrency levels
--request-count N	phase.requests	Control total requests per sub-run
--duration N	phase.duration	Time-based runs (steady-state)
--artifact-dir PATH	output directory	Organize results per run
--ui simple	dashboard mode	Less overhead, no TUI
--no-gpu-telemetry	-	Skip DCGM (endpoints without GPU metrics)
--no-server-metrics	-	Skip Prometheus scraping (non-vLLM endpoints)

Orchestration pattern: Define base structure in YAML, then use CLI flags in a shell loop for parameter sweeps:

for conc in 1 5 10 25 50 100; do
  aiperf profile --config capacity.yaml --concurrency "$conc" --request-count 100 \
    --artifact-dir "results/capacity/conc${conc}"
done

The 3-Scenario Suite Pattern

Design benchmark suites around three complementary scenarios:

1. Sales Scenario (Representative Load)

• Goal: Clean numbers for pitch decks and SLAs
• Concurrency: 1, 5, 10 (low, moderate, realistic)
• Dataset: ShareGPT (realistic chat transcripts)
• Count: ~50 requests per run
• Output: avg/p99 TTFT, throughput, request latency

2. Capacity Scenario (Breakpoint Discovery)

• Goal: Find the concurrency ceiling before degradation
• Concurrency: 10 → 25 → 50 → 100 (stepped ramp)
• Dataset: Synthetic with fixed ISL/OSL (e.g., 512 IN / 128 OUT)
• Count: ~100 requests per level
• Output: Performance curve (throughput vs. concurrency, latency inflection points)

When to stop scaling concurrency: Before proposing conc=200+ tests, extrapolate the existing curve. If TTFT p99 exceeds ~5s at conc=100, further scaling is operationally irrelevant—users won't accept 15s+ wait times. Better to test longer inputs (1024/2048 tokens) to isolate the memory-bandwidth prefill bottleneck, or implement queueing + load balancing.

2b. Long-Context Prefill Test (Memory Bandwidth Bottleneck)

• Goal: Isolate the memory-bandwidth prefill bottleneck (relevant for RAG, long-context agents)
• Concurrency: Fixed low (e.g., 10) to avoid queueing artifacts
• Dataset: Synthetic with long input (1024 / 2048 / 4096 tokens), short output (128)
• Duration: 300s (long enough to amortize cold-start)
• Output: Prefill throughput vs. input length curve

This is more informative than conc=200 when the real workload involves reading long documents.

3. Steady-State Scenario (Long-Term Stability)

• Goal: Detect drift, memory leaks, or throttling over time
• Concurrency: Sweet spot from Capacity scenario (e.g., 20)
• Duration: 600 seconds (10 minutes) or longer
• Dataset: Mixed (50% ShareGPT + 50% synthetic short/long)
• Output: P50/P90/P99 trends over time, failed request rate

Universal Tokenizer for Heterogeneous Endpoints

When benchmarking multiple models with different vocabularies (and downloading the correct HuggingFace tokenizer for each is impractical), use:

--tokenizer builtin

This uses tiktoken's o200k_base encoding universally. It avoids HF downloads, gated-repo access issues, and rate limits. Trade-off: Token counts are approximate for non-OpenAI models (different vocabularies), but TTFT, ITL, and latency metrics remain exact. Recommended when:

• Tokenizer mapping is complicated (4+ models, gated repos, missing HF entries)
• Running in air-gapped or rate-limited environments
• Sales/capacity benchmark where relative comparisons matter more than absolute token counts

Before benchmarking, discover available models from the target endpoint:

curl -s -H "Authorization: Bearer $API_KEY" \
  https://<endpoint>/v1/models | \
  python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin).get('data',[])]"

See scripts/discover-models.sh for a ready-to-run version.

Authentication & Security

• NEVER hardcode API keys in YAML configs, shell scripts, or git
• Always use environment variables: ${API_KEY} or $API_KEY
• In AIPerf YAML, reference via --extra-headers or templating:

  extra-headers:
    Authorization: "Bearer ${API_KEY}"
  

• Export in shell before running: export API_KEY=sk-...

AIPerf Quick Reference

Basic profile run:

aiperf profile \
  --model "vllm/qwen3.6-27b-nvfp4" \
  --streaming \
  --endpoint-type chat \
  --tokenizer <huggingface-tokenizer-name> \
  --url https://ai.noris.de \
  --extra-headers "Authorization: Bearer ${API_KEY}" \
  --concurrency 10 \
  --request-count 50

Key flags:

• --concurrency N – parallel requests
• --request-rate N – requests per second (alternative to concurrency)
• --request-count N – total requests to send
• --duration N – time-based run (seconds)
• --streaming – measure streaming metrics (TTFT, ITL)
• --output-tokens-mean N – target output length (with caveats; see AIPerf docs)
• --dataset <file> – custom dataset (ShareGPT format or synthetic)
• --random-seed N – reproducibility

YAML Configs

AIPerf supports YAML-driven runs for reproducibility. See templates/aiperf-suite-config.yaml for a complete 3-scenario suite template.

Run from YAML:

aiperf profile -c configs/sales-scenario.yaml

Results & Visualization

AIPerf produces:

• CSV – per-request metrics (latency, token counts, timestamps)
• JSON – aggregated statistics with percentile data (avg, min, max, p50, p90, p99, std)
• Logs – aiperf.log with detailed debug info
• Plots – PNG charts via aiperf plot <results-dir>

Results should be organized as:

results/YYYY-MM-DD/
  sales/
  capacity/
  steady-state/

Custom Post-Processing for Multi-Model Suites

For sales/capacity comparison across multiple models and concurrency levels, AIPerf's built-in plot subcommand works on single runs. Use a custom Python post-processing script to generate cross-model comparison charts:

1. Load all profile_export_aiperf.json files recursively from results/YYYY-MM-DD/
2. Extract key metrics per (model, scenario, concurrency):
   • output_token_throughput.avg = throughput (tok/s) TOTAL
   • time_to_first_token.p99 = tail latency (ms)
   • inter_token_latency.p99 = streaming smoothness (ms)
   • prefill_throughput_per_user.avg = prefill speed per user session
   • output_token_throughput_per_user.avg = output speed per user
   • total_token_throughput.avg = (input+output) tok/s TOTAL across all sessions
3. Generate plots with matplotlib:
   • capacity_curve.png – Throughput vs Concurrency (log scale on x-axis)
   • latency_curve.png – TTFT p99 vs Concurrency (log scale on x-axis)
   • sales_comparison.png – Latency at low concurrency (1/5/10)
4. Export summary.csv for Excel / Google Sheets import

A working implementation: scripts/plot-suite-results.py

Rule: *_per_user metrics are for end-user SLA/UX. Total metrics (output_token_throughput, total_token_throughput) are for infrastructure capacity planning. Input throughput is approximately total - output.

MSP / Service Provider Perspective

When presenting results for infrastructure or commercial decision-making (not model research):

Don't: Compare Model A vs Model B on token-per-token quality.
Do: Report each model as an independent service tier with its own SLA envelope.

Structure the analysis per model:

• Single-User Experience (conc=1): TTFT p99, Output/User, SLA class (Gold/Silver/Bronze)
• Multi-Tenant Load (conc=10→100): Throughput scaling, TTFT degradation, recommended max concurrency
• Steady-State: Long-run stability, completed requests, throughput drift
• Service Recommendation: Positioning (Entry/Premium), suitable use cases, limitations

Key metrics by business question:

Business Question	Metric to Use	Why
"How many users per node?"	total_token_throughput (Input + Output)	Node capacity
"What does each user feel?"	output_token_throughput_per_user	UX/SLA
"Is chat real-time responsive?"	ttft_p99 + itl_p99	Interaktivität
"Can we handle batch jobs?"	request_throughput + output_token_throughput	Durchsatz
"What breaks first under load?"	TTFT p99 scaling curve	Bottleneck

Conc ceiling analysis: Before proposing conc=200+ tests, extrapolate the existing capacity curve. If TTFT p99 already exceeds 7s at conc=100, further scaling is operationally irrelevant—users won't accept 15s+ wait times. Better to test longer inputs (1024/2048/4096 tokens) to find the prefill bottleneck, or implement queueing + load balancing.

See references/aiperf-token-metrics-explained.md for the full unit-reference table.

Known Pitfalls

1. AIPerf 0.10.0+ Config Format — The new schema_version: "2.0" envelope is required. Old flat configs will be rejected with 1 validation error for AIPerfConfig. See Config Format section above. Always include requests, duration, or sessions in phases.
2. NumPy X86_V2 crash on older x86_64 — If you see RuntimeError: NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support: (X86_V2), downgrade: pip install "numpy<2.0" --force-reinstall
3. ShareGPT first-run delay — The public dataset type downloads ShareGPT from HuggingFace on first use (~2–5 minutes). Subsequent runs use a local cache. For quick validation, use type: "synthetic". For production suites, pre-download once before the full orchestrated run:

   venv/bin/aiperf profile --config configs/any-model/sales.yaml \
       --concurrency 1 --request-count 1 \
       --artifact-dir /tmp/warmup --no-gpu-telemetry --no-server-metrics
   

   After this, .cache/aiperf/datasets/ contains ShareGPT and all sales runs start instantly.
4. Tokenizer mismatch — Always specify the correct tokenizer. AIPerf auto-detects for some models, but explicit is safer.
5. --output-tokens-mean is not guaranteed — Unless the server supports ignore_eos and min_tokens via --extra-inputs.
6. Rate limiting — AIPerf has built-in retry/backoff, but aggressive concurrency on shared endpoints may trigger 429s. Monitor failed_request_count.
7. Port exhaustion — Very high concurrency (>15,000) can exhaust local ports. Reduce or adjust system limits.
8. Orchestrator path resolution — When running benchmarks in background processes (cron, systemd, detached shells), PATH may not include a Python venv. Always use the absolute path: AIPERF="$BASE_DIR/venv/bin/aiperf" in shell scripts, never rely on aiperf being in $PATH. The failure mode is silent: every run produces aiperf: command not found but the orchestrator continues, producing empty results. See references/aiperf-background-process-pitfalls.md.
9. Prefill throughput bias from cold-start requests — The first request in any AIPerf session includes DNS resolution + TCP/TLS handshake time, which is counted toward "prefill throughput". At conc=1 this can deflate prefill metrics by 50%+. The aggregate profile_export_aiperf.json cannot filter this out. Use profile_export.jsonl (per-request data) to discard turn_index=0 requests, or pre-warm connections with a throwaway run before the measured benchmark. See references/aiperf-prefill-warmup-bias.md.
10. Token metric unit confusion — AIPerf exports multiple "tokens/sec" metrics with different units (per-user vs. total). prefill_throughput_per_user is tok/s per session, while output_token_throughput is tok/s total across all sessions. Comparing them directly or using the wrong one for capacity planning leads to off-by-100× errors.
    • Use *_per_user metrics for end-user experience (SLA, UX)
    • Use output_token_throughput / total_token_throughput for node capacity (infrastructure sizing)
    • Approximate input throughput: total - output See references/aiperf-token-metrics-explained.md.

Orchestration

For multi-scenario suites, wrap AIPerf calls in a shell script that:

1. Validates connectivity and auth (health check)
2. Runs scenarios sequentially
3. Organizes results in timestamped folders
4. Generates plots after each run
5. Detects failures (exit codes, empty CSVs, high error rates) and continues

Session References

• references/aiperf-0.10.0-migration.md — complete config format migration guide with validation error fixes

• references/aiperf-long-input-benchmarking.md — long-input prefill bottleneck testing (replaces conc=200+ tests when memory bandwidth is the real limiter)

• templates/aiperf-long-input-config.yaml — working Schema 2.0 template for 1024-token input tests

• references/aiperf-token-metrics-explained.md — unit reference table: per-user vs. total tok/s, input throughput approximation

• references/aiperf-suite-orchestrator-pattern.md — complete suite orchestrator pattern: directory layout, venv path fix, CLI sweep pattern, ShareGPT pre-download, post-processing sequence

• references/aiperf-prefill-warmup-bias.md — cold-start bias in prefill metrics, mitigation options, per-request JSONL filtering

• references/aiperf-token-metrics-explained.md — unit reference: per-user vs total metrics, input calculation, business-KPI mapping

• references/aiperf-capacity-curve-analysis.md — interpreting capacity curves: when to stop scaling concurrency, breakpoint detection, extrapolation rules

• references/aiperf-long-context-prefill-test.md — isolating the memory-bandwidth bottleneck with 1024/2048/4096-token inputs

• references/ai-noris-de-endpoint.md — discovered model list, URL, auth pattern for ai.noris.de

• references/aiperf-kurzreferenz.md — German quick reference summary

• templates/aiperf-suite-config-v2.yaml — Schema 2.0 starter template with all options documented

• templates/long-context-prefill-test.yaml — 1024-token input test template for RAG/memory-bandwidth analysis

• templates/3-scenario-suite.md — reusable Sales/Capacity/Steady-State suite structure

• GenAI-Perf (NVIDIA, predecessor to AIPerf) – AIPerf is the migration target

• locust + OpenAI client – custom Python load testing, more flexible but more work

• k6 + xk6-openai – if already using k6 for HTTP load testing

• GenAI-Perf (NVIDIA, predecessor to AIPerf) – AIPerf is the migration target

• locust + OpenAI client – custom Python load testing, more flexible but more work

• k6 + xk6-openai – if already using k6 for HTTP load testing

Related Skills

• evaluating-llms-harness – for accuracy benchmarks (MMLU, GSM8K, etc.)
• serving-llms-vllm – for setting up the inference server being benchmarked