112 lines
3.5 KiB
Markdown
112 lines
3.5 KiB
Markdown
# AIPerf Background-Process & Orchestrator Pitfalls
|
||
|
||
Session artifact: debugging why a full 12-run benchmark suite silently failed with `aiperf: command not found` despite successful manual runs.
|
||
|
||
## The Problem: `aiperf: command not found` in Background
|
||
|
||
The background process (started via `bash -lic`) did not inherit the venv-activated PATH. All 12 runs produced:
|
||
```
|
||
./scripts/run-suite.sh: line 90: aiperf: command not found
|
||
```
|
||
|
||
**Root cause:** `aiperf` was installed into a Python venv, but the background shell did not activate the venv before running the orchestrator.
|
||
|
||
## The Fix: Absolute Path to venv Binary
|
||
|
||
In the orchestrator script, define an absolute path variable:
|
||
```bash
|
||
AIPERF="$BASE_DIR/venv/bin/aiperf"
|
||
if [ ! -x "$AIPERF" ]; then
|
||
echo "ERROR: AIPerf not found at $AIPERF"
|
||
exit 1
|
||
fi
|
||
```
|
||
|
||
Then call `"$AIPERF" profile ...` everywhere instead of plain `aiperf profile`.
|
||
|
||
## ShareGPT Pre-Download Strategy
|
||
|
||
`type: "public"` datasets download ShareGPT from HuggingFace on first use (~2–5 minutes). To avoid this delay during production benchmark runs, perform a pre-download:
|
||
|
||
```bash
|
||
# Pre-download ShareGPT into AIPerf's local cache
|
||
venv/bin/aiperf profile \
|
||
--config configs/<any-model>/sales.yaml \
|
||
--concurrency 1 --request-count 1 \
|
||
--no-gpu-telemetry --no-server-metrics \
|
||
--artifact-dir /tmp/predownload
|
||
```
|
||
|
||
Subsequent sales-scenario runs will use the cached dataset instantly.
|
||
|
||
## AIPerf 0.10.0 CLI Flag Reference
|
||
|
||
Critical flags used in this session:
|
||
|
||
| Flag | What it does | Status |
|
||
|------|-------------|--------|
|
||
| `--ui simple` | Minimal UI mode (no TUI) | ✅ Works |
|
||
| `--ui dashboard` | Full TUI dashboard | ✅ Works |
|
||
| `--artifact-dir PATH` | Output directory for CSV/JSON/logs | ✅ Works |
|
||
| `--no-gpu-telemetry` | Skip DCGM/GPU metrics | ✅ Works |
|
||
| `--no-server-metrics` | Skip Prometheus scraping | ✅ Works |
|
||
| `--request-count N` | Total requests (overrides YAML `requests`) | ✅ Works |
|
||
| `--concurrency N` | Concurrent sessions | ✅ Works |
|
||
| `--simple` | ❌ NOT a flag; use `--ui simple` | ❌ Rejected |
|
||
| `--output PATH` | ❌ NOT a flag; use `--artifact-dir` | ❌ Rejected |
|
||
| `--output-artifact-dir PATH` | Alias for `--artifact-dir` | ✅ Works |
|
||
| `--api-key KEY` | Overrides YAML `api_key` | ✅ Works |
|
||
|
||
## The `api_key` + `requests`/`duration`/`sessions` Interaction
|
||
|
||
Sequence of errors encountered:
|
||
|
||
1. `api_key must be a valid string` — this appeared when `phases[].requests` was missing.
|
||
2. With `requests: 50` and `sessions: 1` added to phases → config validated.
|
||
3. **Lesson:** The `api_key` error was misleading. The real issue was always a missing `requests`/`duration`/`sessions` in the phase definition.
|
||
|
||
Pattern that works:
|
||
```yaml
|
||
phases:
|
||
- type: "concurrency"
|
||
name: "profiling"
|
||
requests: 50 # REQUIRED
|
||
sessions: 1 # optional, default 1
|
||
```
|
||
|
||
## Complete Working Config (Single File)
|
||
|
||
```yaml
|
||
schema_version: "2.0"
|
||
benchmark:
|
||
models:
|
||
items:
|
||
- name: "vllm/gemma-4-31b-it"
|
||
strategy: "round_robin"
|
||
endpoint:
|
||
urls:
|
||
- "https://ai.noris.de/v1"
|
||
type: "chat"
|
||
api_key: "${NORIS_API_KEY}"
|
||
streaming: true
|
||
headers:
|
||
Authorization: "Bearer ${NORIS_API_KEY}"
|
||
datasets:
|
||
- type: "synthetic"
|
||
name: "main"
|
||
entries: 100
|
||
isl:
|
||
type: "fixed"
|
||
value: 512
|
||
osl:
|
||
type: "fixed"
|
||
value: 128
|
||
phases:
|
||
- type: "concurrency"
|
||
name: "profiling"
|
||
requests: 50
|
||
sessions: 1
|
||
tokenizer:
|
||
name: "builtin"
|
||
```
|