68 lines
1.7 KiB
Markdown
68 lines
1.7 KiB
Markdown
# AIPerf 3-Scenario Suite Template
|
||
|
||
Reusable structure for benchmarking N models with Sales + Capacity + Steady-State.
|
||
|
||
## Directory Layout
|
||
|
||
```
|
||
aiperf-benchmark-suite/
|
||
├── configs/
|
||
│ └── <model-safe-name>/
|
||
│ ├── sales.yaml
|
||
│ ├── capacity.yaml
|
||
│ └── steady.yaml
|
||
├── templates/
|
||
│ └── base-config.yaml
|
||
├── scripts/
|
||
│ ├── run-suite.sh
|
||
│ └── generate-summary.py
|
||
├── results/ # gitignored
|
||
└── README.md
|
||
```
|
||
|
||
## Base Config Template
|
||
|
||
```yaml
|
||
url: "https://<endpoint>/v1"
|
||
endpoint_type: "chat"
|
||
extra-headers:
|
||
Authorization: "Bearer ${API_KEY}"
|
||
streaming: true
|
||
tokenizer: "builtin"
|
||
random_seed: 42
|
||
```
|
||
|
||
## Scenario Definitions
|
||
|
||
### Sales
|
||
- Concurrency: 1, 5, 10 (sequentially)
|
||
- Request count: 50 per level
|
||
- Dataset: `sharegpt`
|
||
- Display: `simple` or `dashboard`
|
||
- Flags: `--concurrency N --request-count 50 --dataset sharegpt`
|
||
|
||
### Capacity
|
||
- Concurrency: 10, 25, 50, 100 (stepped ramp)
|
||
- Request count: 100 per level
|
||
- Dataset: synthetic fixed lengths
|
||
- Flags: `--concurrency N --request-count 100 --input-tokens-mean 512 --output-tokens-mean 128`
|
||
|
||
### Steady-State
|
||
- Duration: 600 seconds
|
||
- Concurrency: sweet spot from Capacity (e.g., 20)
|
||
- Dataset: mixed or `sharegpt`
|
||
- Flags: `--concurrency 20 --time 600 --dataset sharegpt`
|
||
|
||
## Orchestrator Pattern
|
||
|
||
Iterate models × scenarios, with `--output $RESULTS_DIR/$MODEL/$SCENARIO/conc${CONC}`.
|
||
Pause 30s between major runs to avoid rate limits.
|
||
|
||
## Result Summary
|
||
|
||
After all runs, parse `profile_export_aiperf.json` files to generate markdown table:
|
||
|
||
| Model | Scenario | Concurrency | TTFT avg | TTFT p99 | Throughput |
|
||
|
||
See `scripts/generate-summary.py` for a reference implementation.
|