hermes-skills/mlops/llm-inference-benchmarking/references/aiperf-0.10.0-migration.md

AIPerf 0.10.0 Schema 2.0 Migration Guide

Session artifact: migrated 12 configs from old flat format to new envelope format. All errors encountered are documented here with fixes.

The Old Format (DEPRECATED)

url: "https://ai.noris.de/v1"
endpoint_type: "chat"
model: "vllm/gemma-4-31b-it"
extra-headers:
  Authorization: "Bearer ${API_KEY}"
streaming: true
tokenizer: "builtin"
request_count: 50
concurrency: 1

Result with AIPerf 0.10.0: 1 validation error for AIPerfConfig

The New Format (Schema 2.0)

schema_version: "2.0"
benchmark:
  models:
    items:
      - name: "vllm/gemma-4-31b-it"
    strategy: "round_robin"
  endpoint:
    urls:
      - "https://ai.noris.de/v1"
    type: "chat"
    api_key: "${API_KEY}"
    streaming: true
    headers:
      Authorization: "Bearer ${API_KEY}"
  datasets:
    - type: "synthetic"         # or "public" for ShareGPT
      name: "main"
      entries: 100
      isl:
        type: "fixed"
        value: 512
      osl:
        type: "fixed"
        value: 128
  phases:
    - type: "concurrency"
      name: "profiling"
      # ONE OF THESE IS MANDATORY:
      requests: 50           # total requests
      # OR duration: 600     # seconds
      # OR sessions: 1       # concurrent sessions
  tokenizer:
    name: "builtin"

Critical Validation Errors and Fixes

Error: Phase 'profiling': at least one of 'requests', 'duration', or 'sessions' must be specified

Cause: Phase definition without termination condition.

Fix: Add requests: N or duration: N or sessions: N.

phases:
  - type: "concurrency"
    name: "profiling"
    requests: 50     # <-- REQUIRED

Error: api_key must be a valid string

Cause: AIPerf's Pydantic model validates api_key eagerly. Even though --api-key CLI flag is present, the YAML's api_key field still gets parsed.

Fix: Either:

1. Set a dummy value: api_key: "dummy" (CLI --api-key overrides it)
2. Use env var syntax: api_key: "${API_KEY}" (works if env var is set during config load)
3. Use headers dict instead for auth

Best practice: Use both:

endpoint:
  api_key: "${API_KEY}"          # satisfies validation
  headers:
    Authorization: "Bearer ${API_KEY}"

Error: model not found in schema

Cause: Old flat model key at top level. New format nests under benchmark.models.items[].name.

Fix:

benchmark:
  models:
    items:
      - name: "vllm/gemma-4-31b-it"

Error: extra-headers not found

Cause: Old key name. New format uses endpoint.headers.

Fix:

benchmark:
  endpoint:
    headers:
      Authorization: "Bearer ${API_KEY}"

Error: endpoint_type not found

Cause: Old key name. New format uses endpoint.type.

Fix:

benchmark:
  endpoint:
    type: "chat"

Error: NumPy X86_V2 crash

RuntimeError: NumPy was built with baseline optimizations:
(X86_V2) but your machine doesn't support: (X86_V2).

Cause: AIPerf installs NumPy 2.x, which requires CPU features not present on older x86_64 VMs.

Fix:

pip install "numpy<2.0" --force-reinstall

Error: ShareGPT download timeout

Cause: First run with type: "public" dataset downloads ShareGPT from HuggingFace (~500MB, 2–5 min).

Fix: Wait, or use type: "synthetic" for quick validation.

Synthetic Dataset Format

For fixed ISL/OSL (reproducible capacity tests):

datasets:
  - type: "synthetic"
    name: "main"
    entries: 100
    isl:
      type: "fixed"
      value: 512
    osl:
      type: "fixed"
      value: 128

Distribution types: fixed, normal, lognormal, multimodal, empirical.

CLI Override Compatibility

Flags that work with Schema 2.0 YAML configs:

• --concurrency N → overrides phase concurrency
• --request-count N → overrides phase.requests
• --duration N → overrides phase.duration
• --artifact-dir PATH → sets output directory
• --ui simple|dashboard → UI mode
• --api-key KEY → overrides endpoint.api_key
• --no-gpu-telemetry → skips DCGM
• --no-server-metrics → skips Prometheus scraping

Orchestrator Pattern: YAML + CLI Sweeps

AIPERF="./venv/bin/aiperf"  # always use venv path, never rely on PATH

for conc in 1 5 10; do
  "$AIPERF" profile \
    --config sales.yaml \
    --concurrency "$conc" \
    --request-count 50 \
    --artifact-dir "results/sales/conc${conc}"
done

Critical: In background processes (cron, systemd, nohup), PATH may not include the venv. Always use absolute path to venv/bin/aiperf.