AIPerf Suite Orchestrator Pattern

Reference: building a 12-run benchmark suite (4 models × 3 scenarios) against ai.noris.de.

Directory Convention

aiperf-benchmark-suite/
├── configs/
│   ├── gemma-4-31b-it/
│   │   ├── sales.yaml      # ShareGPT dataset, requests: 50, sessions: 1
│   │   ├── capacity.yaml   # Synthetic 512/128, requests: 100
│   │   └── steady.yaml     # ShareGPT, duration: 600
│   ├── gpt-oss-120b/...
│   └── ... (one dir per model)
├── scripts/
│   ├── run-suite.sh        # Orchestrator
│   ├── generate-summary.py # Markdown table from JSON
│   └── plot-results.py     # Matplotlib charts + CSV
├── results/
│   └── YYYY-MM-DD-HHMMSS/
│       ├── gemma-4-31b-it/
│       │   ├── sales/conc1/
│       │   ├── sales/conc5/
│       │   ├── sales/conc10/
│       │   ├── capacity/conc10/
│       │   ├── capacity/conc25/
│       │   ├── capacity/conc50/
│       │   ├── capacity/conc100/
│       │   └── steady/
│       └── ...
└── venv/                   # AIPerf installed here

Orchestrator Structure

The orchestrator is a bash script that:

Sets absolute paths — AIPERF="$BASE_DIR/venv/bin/aiperf" (never rely on $PATH)
Validates env — checks NORIS_API_KEY and connectivity
Loops models × scenarios
Per scenario, loops concurrency levels (for count-based scenarios)
Calls AIPerf with CLI overrides for each sub-run
Pauses between runs (sleep 5-30) to avoid rate limits
Generates summary + plots after all runs complete
Continues on error (|| true) — one failed sub-run doesn't kill the suite

Critical: Venv Path Fix

When running in background (cron, detached shell, Telegram-triggered runs), $PATH does NOT include the venv. The orchestrator MUST:

AIPERF="$BASE_DIR/venv/bin/aiperf"
if [ ! -x "$AIPERF" ]; then
    echo "ERROR: AIPerf not found at $AIPERF"
    exit 1
fi
# Then call "$AIPERF" profile ... everywhere

Failure mode: aiperf: command not found on every run, producing empty results silently.

CLI Override Pattern for Suite Runs

The YAML configs define the base structure. CLI flags sweep parameters:

# Sales scenario: sweep concurrencies
for conc in 1 5 10; do
    "$AIPERF" profile --config "$CONFIG" \
        --concurrency "$conc" \
        --request-count 50 \
        --artifact-dir "results/sales/conc${conc}"
done

# Capacity scenario: stepped ramp
for conc in 10 25 50 100; do
    "$AIPERF" profile --config "$CONFIG" \
        --concurrency "$conc" \
        --request-count 100 \
        --artifact-dir "results/capacity/conc${conc}"
done

# Steady scenario: time-based
"$AIPERF" profile --config "$CONFIG" \
    --concurrency 20 \
    --artifact-dir "results/steady"

Pre-Download Strategy for ShareGPT

First sales run downloads ShareGPT from HuggingFace (~2–5 min). To avoid this during production:

# One-time warm-up (any model config with public dataset works)
venv/bin/aiperf profile --config configs/any-model/sales.yaml \
    --concurrency 1 --request-count 1 \
    --artifact-dir /tmp/warmup --no-gpu-telemetry --no-server-metrics

Subsequent runs use .cache/aiperf/datasets/ instantly.

Post-Processing Sequence

After the orchestrator finishes all runs:

# 1. Markdown summary
python3 scripts/generate-summary.py results/YYYY-MM-DD-HHMMSS

# 2. Plots + CSV
python3 scripts/plot-results.py results/YYYY-MM-DD-HHMMSS

Outputs:

results/.../README.md – human-readable markdown table
results/.../plots/summary.csv – Excel / Google Sheets import
results/.../plots/capacity_curve.png – throughput vs concurrency
results/.../plots/latency_curve.png – TTFT p99 vs concurrency
results/.../plots/sales_comparison.png – low-concurrency latency

3.9 KiB Raw Permalink Blame History Unescape Escape