# AIPerf 3-Scenario Suite Template

Reusable structure for benchmarking N models with Sales + Capacity + Steady-State.

## Directory Layout

```
aiperf-benchmark-suite/
├── configs/
│   └── <model-safe-name>/
│       ├── sales.yaml
│       ├── capacity.yaml
│       └── steady.yaml
├── templates/
│   └── base-config.yaml
├── scripts/
│   ├── run-suite.sh
│   └── generate-summary.py
├── results/          # gitignored
└── README.md
```

## Base Config Template

```yaml
url: "https://<endpoint>/v1"
endpoint_type: "chat"
extra-headers:
  Authorization: "Bearer ${API_KEY}"
streaming: true
tokenizer: "builtin"
random_seed: 42
```

## Scenario Definitions

### Sales
- Concurrency: 1, 5, 10 (sequentially)
- Request count: 50 per level
- Dataset: `sharegpt`
- Display: `simple` or `dashboard`
- Flags: `--concurrency N --request-count 50 --dataset sharegpt`

### Capacity
- Concurrency: 10, 25, 50, 100 (stepped ramp)
- Request count: 100 per level
- Dataset: synthetic fixed lengths
- Flags: `--concurrency N --request-count 100 --input-tokens-mean 512 --output-tokens-mean 128`

### Steady-State
- Duration: 600 seconds
- Concurrency: sweet spot from Capacity (e.g., 20)
- Dataset: mixed or `sharegpt`
- Flags: `--concurrency 20 --time 600 --dataset sharegpt`

## Orchestrator Pattern

Iterate models × scenarios, with `--output $RESULTS_DIR/$MODEL/$SCENARIO/conc${CONC}`.
Pause 30s between major runs to avoid rate limits.

## Result Summary

After all runs, parse `profile_export_aiperf.json` files to generate markdown table:

| Model | Scenario | Concurrency | TTFT avg | TTFT p99 | Throughput |

See `scripts/generate-summary.py` for a reference implementation.