AIPerf 3-Scenario Suite Template

Reusable structure for benchmarking N models with Sales + Capacity + Steady-State.

Directory Layout

aiperf-benchmark-suite/
├── configs/
│   └── <model-safe-name>/
│       ├── sales.yaml
│       ├── capacity.yaml
│       └── steady.yaml
├── templates/
│   └── base-config.yaml
├── scripts/
│   ├── run-suite.sh
│   └── generate-summary.py
├── results/          # gitignored
└── README.md

Base Config Template

url: "https://<endpoint>/v1"
endpoint_type: "chat"
extra-headers:
  Authorization: "Bearer ${API_KEY}"
streaming: true
tokenizer: "builtin"
random_seed: 42

Scenario Definitions

Sales

Concurrency: 1, 5, 10 (sequentially)
Request count: 50 per level
Dataset: sharegpt
Display: simple or dashboard
Flags: --concurrency N --request-count 50 --dataset sharegpt

Capacity

Concurrency: 10, 25, 50, 100 (stepped ramp)
Request count: 100 per level
Dataset: synthetic fixed lengths
Flags: --concurrency N --request-count 100 --input-tokens-mean 512 --output-tokens-mean 128

Steady-State

Duration: 600 seconds
Concurrency: sweet spot from Capacity (e.g., 20)
Dataset: mixed or sharegpt
Flags: --concurrency 20 --time 600 --dataset sharegpt

Orchestrator Pattern

Iterate models × scenarios, with --output $RESULTS_DIR/$MODEL/$SCENARIO/conc${CONC}. Pause 30s between major runs to avoid rate limits.

Result Summary

After all runs, parse profile_export_aiperf.json files to generate markdown table:

See scripts/generate-summary.py for a reference implementation.

1.7 KiB Raw Blame History Unescape Escape