hermes-skills/mlops/llm-inference-benchmarking/references/aiperf-0.10.0-migration.md

# AIPerf 0.10.0 Schema 2.0 Migration Guide

> Session artifact: migrated 12 configs from old flat format to new envelope format. All errors encountered are documented here with fixes.

## The Old Format (DEPRECATED)

```yaml
url: "https://ai.noris.de/v1"
endpoint_type: "chat"
model: "vllm/gemma-4-31b-it"
extra-headers:
  Authorization: "Bearer ${API_KEY}"
streaming: true
tokenizer: "builtin"
request_count: 50
concurrency: 1
```

**Result with AIPerf 0.10.0:** `1 validation error for AIPerfConfig`

## The New Format (Schema 2.0)

```yaml
schema_version: "2.0"
benchmark:
  models:
    items:
      - name: "vllm/gemma-4-31b-it"
    strategy: "round_robin"
  endpoint:
    urls:
      - "https://ai.noris.de/v1"
    type: "chat"
    api_key: "${API_KEY}"
    streaming: true
    headers:
      Authorization: "Bearer ${API_KEY}"
  datasets:
    - type: "synthetic"         # or "public" for ShareGPT
      name: "main"
      entries: 100
      isl:
        type: "fixed"
        value: 512
      osl:
        type: "fixed"
        value: 128
  phases:
    - type: "concurrency"
      name: "profiling"
      # ONE OF THESE IS MANDATORY:
      requests: 50           # total requests
      # OR duration: 600     # seconds
      # OR sessions: 1       # concurrent sessions
  tokenizer:
    name: "builtin"
```

## Critical Validation Errors and Fixes

### Error: `Phase 'profiling': at least one of 'requests', 'duration', or 'sessions' must be specified`

**Cause:** Phase definition without termination condition.

**Fix:** Add `requests: N` or `duration: N` or `sessions: N`.

```yaml
phases:
  - type: "concurrency"
    name: "profiling"
    requests: 50     # <-- REQUIRED
```

### Error: `api_key must be a valid string`

**Cause:** AIPerf's Pydantic model validates `api_key` eagerly. Even though `--api-key` CLI flag is present, the YAML's `api_key` field still gets parsed.

**Fix:** Either:
1. Set a dummy value: `api_key: "dummy"` (CLI `--api-key` overrides it)
2. Use env var syntax: `api_key: "${API_KEY}"` (works if env var is set during config load)
3. Use `headers` dict instead for auth

**Best practice:** Use both:
```yaml
endpoint:
  api_key: "${API_KEY}"          # satisfies validation
  headers:
    Authorization: "Bearer ${API_KEY}"
```

### Error: `model` not found in schema

**Cause:** Old flat `model` key at top level. New format nests under `benchmark.models.items[].name`.

**Fix:**
```yaml
benchmark:
  models:
    items:
      - name: "vllm/gemma-4-31b-it"
```

### Error: `extra-headers` not found

**Cause:** Old key name. New format uses `endpoint.headers`.

**Fix:**
```yaml
benchmark:
  endpoint:
    headers:
      Authorization: "Bearer ${API_KEY}"
```

### Error: `endpoint_type` not found

**Cause:** Old key name. New format uses `endpoint.type`.

**Fix:**
```yaml
benchmark:
  endpoint:
    type: "chat"
```

### Error: NumPy X86_V2 crash

```
RuntimeError: NumPy was built with baseline optimizations:
(X86_V2) but your machine doesn't support: (X86_V2).
```

**Cause:** AIPerf installs NumPy 2.x, which requires CPU features not present on older x86_64 VMs.

**Fix:**
```bash
pip install "numpy<2.0" --force-reinstall
```

### Error: ShareGPT download timeout

**Cause:** First run with `type: "public"` dataset downloads ShareGPT from HuggingFace (~500MB, 2–5 min).

**Fix:** Wait, or use `type: "synthetic"` for quick validation.

## Synthetic Dataset Format

For fixed ISL/OSL (reproducible capacity tests):
```yaml
datasets:
  - type: "synthetic"
    name: "main"
    entries: 100
    isl:
      type: "fixed"
      value: 512
    osl:
      type: "fixed"
      value: 128
```

Distribution types: `fixed`, `normal`, `lognormal`, `multimodal`, `empirical`.

## CLI Override Compatibility

Flags that work with Schema 2.0 YAML configs:
- `--concurrency N` → overrides phase concurrency
- `--request-count N` → overrides phase.requests
- `--duration N` → overrides phase.duration
- `--artifact-dir PATH` → sets output directory
- `--ui simple|dashboard` → UI mode
- `--api-key KEY` → overrides endpoint.api_key
- `--no-gpu-telemetry` → skips DCGM
- `--no-server-metrics` → skips Prometheus scraping

## Orchestrator Pattern: YAML + CLI Sweeps

```bash
AIPERF="./venv/bin/aiperf"  # always use venv path, never rely on PATH

for conc in 1 5 10; do
  "$AIPERF" profile \
    --config sales.yaml \
    --concurrency "$conc" \
    --request-count 50 \
    --artifact-dir "results/sales/conc${conc}"
done
```

**Critical:** In background processes (cron, systemd, nohup), `PATH` may not include the venv. Always use absolute path to `venv/bin/aiperf`.