112 lines
3.2 KiB
Markdown
112 lines
3.2 KiB
Markdown
# AIPerf 0.10.0+ Kurzreferenz
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
pip install aiperf
|
|
# Extras: [mlflow] [otel] [wandb]
|
|
```
|
|
|
|
**NumPy compatibility warning:** On older x86_64 VMs without X86_V2 CPU features, NumPy 2.x (pulled by AIPerf) crashes. Pre-fix with:
|
|
```bash
|
|
pip install "numpy<2.0" --force-reinstall --no-cache-dir
|
|
pip install aiperf
|
|
```
|
|
|
|
## CLI Quickstart
|
|
|
|
```bash
|
|
aiperf profile \
|
|
--config config.yaml \
|
|
--api-key "$API_KEY" \
|
|
--concurrency 10 \
|
|
--request-count 50 \
|
|
--artifact-dir results/
|
|
```
|
|
|
|
## Key CLI Flags (0.10.0+)
|
|
|
|
| Flag | YAML Override | Description |
|
|
|------|---------------|-------------|
|
|
| `--config FILE` | - | Load Schema 2.0 YAML config |
|
|
| `--api-key KEY` | `endpoint.api_key` | Auth key (overrides YAML) |
|
|
| `--concurrency N` | phase.sessions | Concurrent requests/sessions |
|
|
| `--request-rate N` | - | Requests/sec (alt. to concurrency) |
|
|
| `--request-count N` | phase.requests | Total requests |
|
|
| `--duration N` | phase.duration | Time-based run (seconds) |
|
|
| `--streaming` | `endpoint.streaming` | Enable streaming (TTFT/ITL) |
|
|
| `--ui simple` | `ui` | Progress bars (vs. `dashboard` TUI) |
|
|
| `--ui none` | `ui` | Headless |
|
|
| `--artifact-dir DIR` | `output_dir` | Results directory |
|
|
| `--no-gpu-telemetry` | - | Skip DCGM/GPU metrics |
|
|
| `--no-server-metrics` | - | Skip Prometheus scraping |
|
|
| `--tokenizer builtin` | `tokenizer.name` | tiktoken (no HF download) |
|
|
| `--random-seed N` | `random_seed` | Reproducibility |
|
|
|
|
## Schema 2.0 YAML Config
|
|
|
|
```yaml
|
|
schema_version: "2.0"
|
|
benchmark:
|
|
models:
|
|
items:
|
|
- name: "vllm/gemma-4-31b-it"
|
|
strategy: "round_robin"
|
|
endpoint:
|
|
urls:
|
|
- "https://ai.noris.de/v1"
|
|
type: "chat"
|
|
api_key: "${API_KEY}"
|
|
streaming: true
|
|
headers:
|
|
Authorization: "Bearer ${API_KEY}"
|
|
datasets:
|
|
- type: "public" # "public" = HF download; "synthetic" = local gen
|
|
name: "main"
|
|
dataset: "sharegpt"
|
|
sampling: "shuffle"
|
|
phases:
|
|
- type: "concurrency"
|
|
name: "profiling"
|
|
requests: 50
|
|
sessions: 1
|
|
tokenizer:
|
|
name: "builtin"
|
|
```
|
|
|
|
Run from YAML:
|
|
```bash
|
|
aiperf profile --config config.yaml
|
|
```
|
|
|
|
## Outputs
|
|
- **CSV** (`profile_export_aiperf.csv`): Per-request metrics
|
|
- **JSON** (`profile_export_aiperf.json`): Aggregated stats (avg, p50, p90, p99)
|
|
- **Log** (`logs/aiperf.log`): Detailed debug
|
|
- **Plots**: `aiperf plot <artifact-dir>`
|
|
|
|
## Dataset Types Quick Reference
|
|
|
|
| Type | Speed | Source | Use |
|
|
|------|-------|--------|-----|
|
|
| `public` (ShareGPT) | Slow first-run (~2-5 min) | HuggingFace download | Realistic workload |
|
|
| `synthetic` (fixed ISL/OSL) | Instant | Local generation | Capacity testing, reproducibility |
|
|
|
|
## Orchestration Pattern
|
|
|
|
YAML base + CLI sweeps:
|
|
```bash
|
|
for conc in 1 5 10 25 50 100; do
|
|
aiperf profile --config capacity.yaml --concurrency "$conc" \
|
|
--request-count 100 --artifact-dir "results/conc${conc}"
|
|
done
|
|
```
|
|
|
|
## Validation Errors & Quick Fixes
|
|
|
|
| Error | Fix |
|
|
|-------|-----|
|
|
| `1 validation error for AIPerfConfig` | Wrap YAML in `schema_version: "2.0"` / `benchmark:` envelope |
|
|
| `Phase 'profiling': at least one of 'requests', 'duration', or 'sessions' must be specified` | Add `requests: N` or `duration: N` to phase |
|
|
| `NumPy: baseline optimizations (X86_V2) not supported` | `pip install "numpy<2.0" --force-reinstall` |
|