AIPerf Background-Process & Orchestrator Pitfalls

Session artifact: debugging why a full 12-run benchmark suite silently failed with aiperf: command not found despite successful manual runs.

The Problem: `aiperf: command not found` in Background

The background process (started via bash -lic) did not inherit the venv-activated PATH. All 12 runs produced:

./scripts/run-suite.sh: line 90: aiperf: command not found

Root cause: aiperf was installed into a Python venv, but the background shell did not activate the venv before running the orchestrator.

The Fix: Absolute Path to venv Binary

In the orchestrator script, define an absolute path variable:

AIPERF="$BASE_DIR/venv/bin/aiperf"
if [ ! -x "$AIPERF" ]; then
    echo "ERROR: AIPerf not found at $AIPERF"
    exit 1
fi

Then call "$AIPERF" profile ... everywhere instead of plain aiperf profile.

ShareGPT Pre-Download Strategy

type: "public" datasets download ShareGPT from HuggingFace on first use (~2–5 minutes). To avoid this delay during production benchmark runs, perform a pre-download:

# Pre-download ShareGPT into AIPerf's local cache
venv/bin/aiperf profile \
    --config configs/<any-model>/sales.yaml \
    --concurrency 1 --request-count 1 \
    --no-gpu-telemetry --no-server-metrics \
    --artifact-dir /tmp/predownload

Subsequent sales-scenario runs will use the cached dataset instantly.

AIPerf 0.10.0 CLI Flag Reference

Critical flags used in this session:

Flag	What it does	Status
`--ui simple`	Minimal UI mode (no TUI)	✅ Works
`--ui dashboard`	Full TUI dashboard	✅ Works
`--artifact-dir PATH`	Output directory for CSV/JSON/logs	✅ Works
`--no-gpu-telemetry`	Skip DCGM/GPU metrics	✅ Works
`--no-server-metrics`	Skip Prometheus scraping	✅ Works
`--request-count N`	Total requests (overrides YAML `requests`)	✅ Works
`--concurrency N`	Concurrent sessions	✅ Works
`--simple`	❌ NOT a flag; use `--ui simple`	❌ Rejected
`--output PATH`	❌ NOT a flag; use `--artifact-dir`	❌ Rejected
`--output-artifact-dir PATH`	Alias for `--artifact-dir`	✅ Works
`--api-key KEY`	Overrides YAML `api_key`	✅ Works

The `api_key` + `requests`/`duration`/`sessions` Interaction

Sequence of errors encountered:

api_key must be a valid string — this appeared when phases[].requests was missing.
With requests: 50 and sessions: 1 added to phases → config validated.
Lesson: The api_key error was misleading. The real issue was always a missing requests/duration/sessions in the phase definition.

Pattern that works:

phases:
  - type: "concurrency"
    name: "profiling"
    requests: 50          # REQUIRED
    sessions: 1           # optional, default 1

Complete Working Config (Single File)

schema_version: "2.0"
benchmark:
  models:
    items:
      - name: "vllm/gemma-4-31b-it"
    strategy: "round_robin"
  endpoint:
    urls:
      - "https://ai.noris.de/v1"
    type: "chat"
    api_key: "${NORIS_API_KEY}"
    streaming: true
    headers:
      Authorization: "Bearer ${NORIS_API_KEY}"
  datasets:
    - type: "synthetic"
      name: "main"
      entries: 100
      isl:
        type: "fixed"
        value: 512
      osl:
        type: "fixed"
        value: 128
  phases:
    - type: "concurrency"
      name: "profiling"
      requests: 50
      sessions: 1
  tokenizer:
    name: "builtin"

3.5 KiB Raw Blame History Unescape Escape