Initial commit: Hermes Agent Skills collection
This commit is contained in:
Debian
@@ -0,0 +1,43 @@
|
||||
# ai.noris.de Endpoint Reference (Session 2026-06-18)
|
||||
|
||||
## Endpoint Details
|
||||
|
||||
- **URL**: `https://ai.noris.de/v1`
|
||||
- **Type**: OpenAI-compatible Chat Completions
|
||||
- **Auth**: Bearer token via `Authorization` header
|
||||
- **API Key**: `sk-bf-8779d73c-6a51-49d6-a22f-00a0290ca6a7`
|
||||
- **Models endpoint**: `GET /v1/models` → returns 4 models
|
||||
|
||||
## Discovered Models (2026-06-18)
|
||||
|
||||
| Model ID | Size | Notes |
|
||||
|----------|------|-------|
|
||||
| `vllm/gemma-4-31b-it` | 31B | Google Gemma 4 |
|
||||
| `vllm/gpt-oss-120b` | 120B | GPT-OSS, largest model |
|
||||
| `vllm/moonshotai/kimi-k2.6` | ~12B | Kimi K2.6, strongest generalist |
|
||||
| `vllm/qwen3.6-27b-nvfp4` | 27B | Qwen 3.6 with NVFP4 quantization |
|
||||
|
||||
All models served via vLLM.
|
||||
|
||||
## Benchmark Results (Single Request, Synthetic 512/128)
|
||||
|
||||
### gemma-4-31b-it
|
||||
- **TTFT**: 361.7 ms
|
||||
- **Throughput**: 13.78 tok/s
|
||||
- **ISL**: 512, **OSL**: 128
|
||||
|
||||
### gpt-oss-120b
|
||||
- Slightly higher latency expected due to size
|
||||
|
||||
### moonshotai/kimi-k2.6
|
||||
- Generally strongest quality
|
||||
|
||||
### qwen3.6-27b-nvfp4
|
||||
- NVFP4 quantization is aggressive – monitor quality vs. speed trade-off
|
||||
|
||||
## Endpoint Compatibility Notes
|
||||
|
||||
- **Prometheus metrics**: `/v1/metrics` returns HTML (web UI), not Prometheus exposition format
|
||||
- `--no-server-metrics` recommended when benchmarking
|
||||
- **GPU telemetry**: No DCGM endpoints exposed (localhost:9400/9401 unreachable)
|
||||
- `--no-gpu-telemetry` recommended
|
||||
@@ -0,0 +1,190 @@
|
||||
# AIPerf 0.10.0 Schema 2.0 Migration Guide
|
||||
|
||||
> Session artifact: migrated 12 configs from old flat format to new envelope format. All errors encountered are documented here with fixes.
|
||||
|
||||
## The Old Format (DEPRECATED)
|
||||
|
||||
```yaml
|
||||
url: "https://ai.noris.de/v1"
|
||||
endpoint_type: "chat"
|
||||
model: "vllm/gemma-4-31b-it"
|
||||
extra-headers:
|
||||
Authorization: "Bearer ${API_KEY}"
|
||||
streaming: true
|
||||
tokenizer: "builtin"
|
||||
request_count: 50
|
||||
concurrency: 1
|
||||
```
|
||||
|
||||
**Result with AIPerf 0.10.0:** `1 validation error for AIPerfConfig`
|
||||
|
||||
## The New Format (Schema 2.0)
|
||||
|
||||
```yaml
|
||||
schema_version: "2.0"
|
||||
benchmark:
|
||||
models:
|
||||
items:
|
||||
- name: "vllm/gemma-4-31b-it"
|
||||
strategy: "round_robin"
|
||||
endpoint:
|
||||
urls:
|
||||
- "https://ai.noris.de/v1"
|
||||
type: "chat"
|
||||
api_key: "${API_KEY}"
|
||||
streaming: true
|
||||
headers:
|
||||
Authorization: "Bearer ${API_KEY}"
|
||||
datasets:
|
||||
- type: "synthetic" # or "public" for ShareGPT
|
||||
name: "main"
|
||||
entries: 100
|
||||
isl:
|
||||
type: "fixed"
|
||||
value: 512
|
||||
osl:
|
||||
type: "fixed"
|
||||
value: 128
|
||||
phases:
|
||||
- type: "concurrency"
|
||||
name: "profiling"
|
||||
# ONE OF THESE IS MANDATORY:
|
||||
requests: 50 # total requests
|
||||
# OR duration: 600 # seconds
|
||||
# OR sessions: 1 # concurrent sessions
|
||||
tokenizer:
|
||||
name: "builtin"
|
||||
```
|
||||
|
||||
## Critical Validation Errors and Fixes
|
||||
|
||||
### Error: `Phase 'profiling': at least one of 'requests', 'duration', or 'sessions' must be specified`
|
||||
|
||||
**Cause:** Phase definition without termination condition.
|
||||
|
||||
**Fix:** Add `requests: N` or `duration: N` or `sessions: N`.
|
||||
|
||||
```yaml
|
||||
phases:
|
||||
- type: "concurrency"
|
||||
name: "profiling"
|
||||
requests: 50 # <-- REQUIRED
|
||||
```
|
||||
|
||||
### Error: `api_key must be a valid string`
|
||||
|
||||
**Cause:** AIPerf's Pydantic model validates `api_key` eagerly. Even though `--api-key` CLI flag is present, the YAML's `api_key` field still gets parsed.
|
||||
|
||||
**Fix:** Either:
|
||||
1. Set a dummy value: `api_key: "dummy"` (CLI `--api-key` overrides it)
|
||||
2. Use env var syntax: `api_key: "${API_KEY}"` (works if env var is set during config load)
|
||||
3. Use `headers` dict instead for auth
|
||||
|
||||
**Best practice:** Use both:
|
||||
```yaml
|
||||
endpoint:
|
||||
api_key: "${API_KEY}" # satisfies validation
|
||||
headers:
|
||||
Authorization: "Bearer ${API_KEY}"
|
||||
```
|
||||
|
||||
### Error: `model` not found in schema
|
||||
|
||||
**Cause:** Old flat `model` key at top level. New format nests under `benchmark.models.items[].name`.
|
||||
|
||||
**Fix:**
|
||||
```yaml
|
||||
benchmark:
|
||||
models:
|
||||
items:
|
||||
- name: "vllm/gemma-4-31b-it"
|
||||
```
|
||||
|
||||
### Error: `extra-headers` not found
|
||||
|
||||
**Cause:** Old key name. New format uses `endpoint.headers`.
|
||||
|
||||
**Fix:**
|
||||
```yaml
|
||||
benchmark:
|
||||
endpoint:
|
||||
headers:
|
||||
Authorization: "Bearer ${API_KEY}"
|
||||
```
|
||||
|
||||
### Error: `endpoint_type` not found
|
||||
|
||||
**Cause:** Old key name. New format uses `endpoint.type`.
|
||||
|
||||
**Fix:**
|
||||
```yaml
|
||||
benchmark:
|
||||
endpoint:
|
||||
type: "chat"
|
||||
```
|
||||
|
||||
### Error: NumPy X86_V2 crash
|
||||
|
||||
```
|
||||
RuntimeError: NumPy was built with baseline optimizations:
|
||||
(X86_V2) but your machine doesn't support: (X86_V2).
|
||||
```
|
||||
|
||||
**Cause:** AIPerf installs NumPy 2.x, which requires CPU features not present on older x86_64 VMs.
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
pip install "numpy<2.0" --force-reinstall
|
||||
```
|
||||
|
||||
### Error: ShareGPT download timeout
|
||||
|
||||
**Cause:** First run with `type: "public"` dataset downloads ShareGPT from HuggingFace (~500MB, 2–5 min).
|
||||
|
||||
**Fix:** Wait, or use `type: "synthetic"` for quick validation.
|
||||
|
||||
## Synthetic Dataset Format
|
||||
|
||||
For fixed ISL/OSL (reproducible capacity tests):
|
||||
```yaml
|
||||
datasets:
|
||||
- type: "synthetic"
|
||||
name: "main"
|
||||
entries: 100
|
||||
isl:
|
||||
type: "fixed"
|
||||
value: 512
|
||||
osl:
|
||||
type: "fixed"
|
||||
value: 128
|
||||
```
|
||||
|
||||
Distribution types: `fixed`, `normal`, `lognormal`, `multimodal`, `empirical`.
|
||||
|
||||
## CLI Override Compatibility
|
||||
|
||||
Flags that work with Schema 2.0 YAML configs:
|
||||
- `--concurrency N` → overrides phase concurrency
|
||||
- `--request-count N` → overrides phase.requests
|
||||
- `--duration N` → overrides phase.duration
|
||||
- `--artifact-dir PATH` → sets output directory
|
||||
- `--ui simple|dashboard` → UI mode
|
||||
- `--api-key KEY` → overrides endpoint.api_key
|
||||
- `--no-gpu-telemetry` → skips DCGM
|
||||
- `--no-server-metrics` → skips Prometheus scraping
|
||||
|
||||
## Orchestrator Pattern: YAML + CLI Sweeps
|
||||
|
||||
```bash
|
||||
AIPERF="./venv/bin/aiperf" # always use venv path, never rely on PATH
|
||||
|
||||
for conc in 1 5 10; do
|
||||
"$AIPERF" profile \
|
||||
--config sales.yaml \
|
||||
--concurrency "$conc" \
|
||||
--request-count 50 \
|
||||
--artifact-dir "results/sales/conc${conc}"
|
||||
done
|
||||
```
|
||||
|
||||
**Critical:** In background processes (cron, systemd, nohup), `PATH` may not include the venv. Always use absolute path to `venv/bin/aiperf`.
|
||||
@@ -0,0 +1,111 @@
|
||||
# AIPerf Background-Process & Orchestrator Pitfalls
|
||||
|
||||
Session artifact: debugging why a full 12-run benchmark suite silently failed with `aiperf: command not found` despite successful manual runs.
|
||||
|
||||
## The Problem: `aiperf: command not found` in Background
|
||||
|
||||
The background process (started via `bash -lic`) did not inherit the venv-activated PATH. All 12 runs produced:
|
||||
```
|
||||
./scripts/run-suite.sh: line 90: aiperf: command not found
|
||||
```
|
||||
|
||||
**Root cause:** `aiperf` was installed into a Python venv, but the background shell did not activate the venv before running the orchestrator.
|
||||
|
||||
## The Fix: Absolute Path to venv Binary
|
||||
|
||||
In the orchestrator script, define an absolute path variable:
|
||||
```bash
|
||||
AIPERF="$BASE_DIR/venv/bin/aiperf"
|
||||
if [ ! -x "$AIPERF" ]; then
|
||||
echo "ERROR: AIPerf not found at $AIPERF"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
Then call `"$AIPERF" profile ...` everywhere instead of plain `aiperf profile`.
|
||||
|
||||
## ShareGPT Pre-Download Strategy
|
||||
|
||||
`type: "public"` datasets download ShareGPT from HuggingFace on first use (~2–5 minutes). To avoid this delay during production benchmark runs, perform a pre-download:
|
||||
|
||||
```bash
|
||||
# Pre-download ShareGPT into AIPerf's local cache
|
||||
venv/bin/aiperf profile \
|
||||
--config configs/<any-model>/sales.yaml \
|
||||
--concurrency 1 --request-count 1 \
|
||||
--no-gpu-telemetry --no-server-metrics \
|
||||
--artifact-dir /tmp/predownload
|
||||
```
|
||||
|
||||
Subsequent sales-scenario runs will use the cached dataset instantly.
|
||||
|
||||
## AIPerf 0.10.0 CLI Flag Reference
|
||||
|
||||
Critical flags used in this session:
|
||||
|
||||
| Flag | What it does | Status |
|
||||
|------|-------------|--------|
|
||||
| `--ui simple` | Minimal UI mode (no TUI) | ✅ Works |
|
||||
| `--ui dashboard` | Full TUI dashboard | ✅ Works |
|
||||
| `--artifact-dir PATH` | Output directory for CSV/JSON/logs | ✅ Works |
|
||||
| `--no-gpu-telemetry` | Skip DCGM/GPU metrics | ✅ Works |
|
||||
| `--no-server-metrics` | Skip Prometheus scraping | ✅ Works |
|
||||
| `--request-count N` | Total requests (overrides YAML `requests`) | ✅ Works |
|
||||
| `--concurrency N` | Concurrent sessions | ✅ Works |
|
||||
| `--simple` | ❌ NOT a flag; use `--ui simple` | ❌ Rejected |
|
||||
| `--output PATH` | ❌ NOT a flag; use `--artifact-dir` | ❌ Rejected |
|
||||
| `--output-artifact-dir PATH` | Alias for `--artifact-dir` | ✅ Works |
|
||||
| `--api-key KEY` | Overrides YAML `api_key` | ✅ Works |
|
||||
|
||||
## The `api_key` + `requests`/`duration`/`sessions` Interaction
|
||||
|
||||
Sequence of errors encountered:
|
||||
|
||||
1. `api_key must be a valid string` — this appeared when `phases[].requests` was missing.
|
||||
2. With `requests: 50` and `sessions: 1` added to phases → config validated.
|
||||
3. **Lesson:** The `api_key` error was misleading. The real issue was always a missing `requests`/`duration`/`sessions` in the phase definition.
|
||||
|
||||
Pattern that works:
|
||||
```yaml
|
||||
phases:
|
||||
- type: "concurrency"
|
||||
name: "profiling"
|
||||
requests: 50 # REQUIRED
|
||||
sessions: 1 # optional, default 1
|
||||
```
|
||||
|
||||
## Complete Working Config (Single File)
|
||||
|
||||
```yaml
|
||||
schema_version: "2.0"
|
||||
benchmark:
|
||||
models:
|
||||
items:
|
||||
- name: "vllm/gemma-4-31b-it"
|
||||
strategy: "round_robin"
|
||||
endpoint:
|
||||
urls:
|
||||
- "https://ai.noris.de/v1"
|
||||
type: "chat"
|
||||
api_key: "${NORIS_API_KEY}"
|
||||
streaming: true
|
||||
headers:
|
||||
Authorization: "Bearer ${NORIS_API_KEY}"
|
||||
datasets:
|
||||
- type: "synthetic"
|
||||
name: "main"
|
||||
entries: 100
|
||||
isl:
|
||||
type: "fixed"
|
||||
value: 512
|
||||
osl:
|
||||
type: "fixed"
|
||||
value: 128
|
||||
phases:
|
||||
- type: "concurrency"
|
||||
name: "profiling"
|
||||
requests: 50
|
||||
sessions: 1
|
||||
tokenizer:
|
||||
name: "builtin"
|
||||
```
|
||||
@@ -0,0 +1,98 @@
|
||||
# Capacity Curve Analysis – When to Stop Scaling Concurrency
|
||||
|
||||
## The Core Question
|
||||
|
||||
"Shall we test conc=200?" The answer depends on extrapolating the existing capacity curve, not on gut feeling.
|
||||
|
||||
## Extrapolation Rule
|
||||
|
||||
Given TTFT p99 values at conc=10, 50, 100, estimate conc=200 using a **linear degradation model** (the simplest that fits observed data):
|
||||
|
||||
```
|
||||
TTFT_p99(conc) ≈ base + k × conc
|
||||
|
||||
Where k is derived from the steepest interval:
|
||||
k = (TTFT_p99(conc=100) - TTFT_p99(conc=50)) / (100 - 50)
|
||||
```
|
||||
|
||||
If the extrapolated TTFT p99 at conc=200 exceeds a **user-acceptable threshold** (typically 5000ms = 5s for interactive applications, 10000ms = 10s for batch), conc=200 is **operationally irrelevant**.
|
||||
|
||||
## Worked Example (from this session)
|
||||
|
||||
### gpt-oss-120b
|
||||
| Conc | TTFT p99 |
|
||||
|------|----------|
|
||||
| 10 | 978 ms |
|
||||
| 25 | 1924 ms |
|
||||
| 50 | 3575 ms |
|
||||
| 100 | 7310 ms |
|
||||
|
||||
Extrapolation to conc=200:
|
||||
- k = (7310 - 3575) / 50 = **74.7 ms/conc**
|
||||
- TTFT_p99(200) = 7310 + 74.7 × 100 = **14,780 ms**
|
||||
|
||||
**Verdict: Unusable. Users won't wait 15 seconds for the first token.**
|
||||
|
||||
### gemma-4-31b-it
|
||||
| Conc | TTFT p99 |
|
||||
|------|----------|
|
||||
| 10 | 381 ms |
|
||||
| 50 | 1042 ms |
|
||||
| 100 | 2060 ms |
|
||||
|
||||
Extrapolation:
|
||||
- k = (2060 - 1042) / 50 = **20.4 ms/conc**
|
||||
- TTFT_p99(200) = 2060 + 20.4 × 100 = **4,100 ms**
|
||||
|
||||
**Verdict: Borderline. 4.1s is acceptable for some batch use cases, unacceptable for chat.**
|
||||
|
||||
### moonshotai-kimi-k2.6
|
||||
| Conc | TTFT p99 |
|
||||
|------|----------|
|
||||
| 10 | 695 ms |
|
||||
| 25 | 1178 ms |
|
||||
| 50 | 1989 ms |
|
||||
| 100 | 3926 ms |
|
||||
|
||||
Extrapolation:
|
||||
- k = (3926 - 1989) / 50 = **38.7 ms/conc**
|
||||
- TTFT_p99(200) = 3926 + 38.7 × 100 = **7,796 ms**
|
||||
|
||||
**Verdict: Unusable for interactive use.**
|
||||
|
||||
## Alternative to Higher Concurrency: Longer Inputs
|
||||
|
||||
When conc=100 is already the practical ceiling, the more valuable test is **long-context prefill stress**:
|
||||
|
||||
| Test | What it measures | When to use |
|
||||
|------|-----------------|-------------|
|
||||
| conc=10→100 | Queueing / scheduling efficiency | Find max parallel users |
|
||||
| ISL=512 → 1024 → 2048 → 4096 | Memory bandwidth / KV-cache limits | RAG, document analysis, long-context agents |
|
||||
|
||||
A model that degrades gracefully at conc=100 but collapses at ISL=2048 has a **memory-bandwidth bottleneck**, not a scheduling bottleneck. The fix is faster memory (HBM3e), not more GPU compute.
|
||||
|
||||
## Breakpoint Detection Heuristic
|
||||
|
||||
Formally define the "breakpoint" as the highest concurrency where **all three** conditions hold:
|
||||
|
||||
1. TTFT p99 < 5000 ms (5 seconds)
|
||||
2. ITL p99 < 200 ms (readable streaming, ~5 tok/s per user)
|
||||
3. Output throughput still scales linearly with concurrency (no saturation)
|
||||
|
||||
From observed data:
|
||||
|
||||
| Model | Breakpoint (conc) | Limiting factor |
|
||||
|-------|-------------------|-----------------|
|
||||
| gemma-4-31b-it | ~100 | TTFT ≈ 2s still OK |
|
||||
| moonshotai-kimi-k2.6 | ~75 | TTFT ≈ 3.5s approaching limit |
|
||||
| gpt-oss-120b | ~40 | TTFT ≈ 3.5s at conc=50 |
|
||||
| qwen3.6-27b-nvfp4 | ~40 | TTFT ≈ 3s at conc=50 |
|
||||
|
||||
## Operational Recommendation
|
||||
|
||||
- Set **hard concurrency limits** at the breakpoint, not at the hardware maximum
|
||||
- Implement **queueing + load balancing** beyond the breakpoint
|
||||
- Test **LLM-as-a-Service tiers**:
|
||||
- Tier 1 (Premium): conc ≤ 10, guaranteed < 1s TTFT
|
||||
- Tier 2 (Standard): conc ≤ 50, guaranteed < 3s TTFT
|
||||
- Tier 3 (Batch): conc ≤ 100, best-effort, queueing OK
|
||||
@@ -0,0 +1,111 @@
|
||||
# AIPerf 0.10.0+ Kurzreferenz
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install aiperf
|
||||
# Extras: [mlflow] [otel] [wandb]
|
||||
```
|
||||
|
||||
**NumPy compatibility warning:** On older x86_64 VMs without X86_V2 CPU features, NumPy 2.x (pulled by AIPerf) crashes. Pre-fix with:
|
||||
```bash
|
||||
pip install "numpy<2.0" --force-reinstall --no-cache-dir
|
||||
pip install aiperf
|
||||
```
|
||||
|
||||
## CLI Quickstart
|
||||
|
||||
```bash
|
||||
aiperf profile \
|
||||
--config config.yaml \
|
||||
--api-key "$API_KEY" \
|
||||
--concurrency 10 \
|
||||
--request-count 50 \
|
||||
--artifact-dir results/
|
||||
```
|
||||
|
||||
## Key CLI Flags (0.10.0+)
|
||||
|
||||
| Flag | YAML Override | Description |
|
||||
|------|---------------|-------------|
|
||||
| `--config FILE` | - | Load Schema 2.0 YAML config |
|
||||
| `--api-key KEY` | `endpoint.api_key` | Auth key (overrides YAML) |
|
||||
| `--concurrency N` | phase.sessions | Concurrent requests/sessions |
|
||||
| `--request-rate N` | - | Requests/sec (alt. to concurrency) |
|
||||
| `--request-count N` | phase.requests | Total requests |
|
||||
| `--duration N` | phase.duration | Time-based run (seconds) |
|
||||
| `--streaming` | `endpoint.streaming` | Enable streaming (TTFT/ITL) |
|
||||
| `--ui simple` | `ui` | Progress bars (vs. `dashboard` TUI) |
|
||||
| `--ui none` | `ui` | Headless |
|
||||
| `--artifact-dir DIR` | `output_dir` | Results directory |
|
||||
| `--no-gpu-telemetry` | - | Skip DCGM/GPU metrics |
|
||||
| `--no-server-metrics` | - | Skip Prometheus scraping |
|
||||
| `--tokenizer builtin` | `tokenizer.name` | tiktoken (no HF download) |
|
||||
| `--random-seed N` | `random_seed` | Reproducibility |
|
||||
|
||||
## Schema 2.0 YAML Config
|
||||
|
||||
```yaml
|
||||
schema_version: "2.0"
|
||||
benchmark:
|
||||
models:
|
||||
items:
|
||||
- name: "vllm/gemma-4-31b-it"
|
||||
strategy: "round_robin"
|
||||
endpoint:
|
||||
urls:
|
||||
- "https://ai.noris.de/v1"
|
||||
type: "chat"
|
||||
api_key: "${API_KEY}"
|
||||
streaming: true
|
||||
headers:
|
||||
Authorization: "Bearer ${API_KEY}"
|
||||
datasets:
|
||||
- type: "public" # "public" = HF download; "synthetic" = local gen
|
||||
name: "main"
|
||||
dataset: "sharegpt"
|
||||
sampling: "shuffle"
|
||||
phases:
|
||||
- type: "concurrency"
|
||||
name: "profiling"
|
||||
requests: 50
|
||||
sessions: 1
|
||||
tokenizer:
|
||||
name: "builtin"
|
||||
```
|
||||
|
||||
Run from YAML:
|
||||
```bash
|
||||
aiperf profile --config config.yaml
|
||||
```
|
||||
|
||||
## Outputs
|
||||
- **CSV** (`profile_export_aiperf.csv`): Per-request metrics
|
||||
- **JSON** (`profile_export_aiperf.json`): Aggregated stats (avg, p50, p90, p99)
|
||||
- **Log** (`logs/aiperf.log`): Detailed debug
|
||||
- **Plots**: `aiperf plot <artifact-dir>`
|
||||
|
||||
## Dataset Types Quick Reference
|
||||
|
||||
| Type | Speed | Source | Use |
|
||||
|------|-------|--------|-----|
|
||||
| `public` (ShareGPT) | Slow first-run (~2-5 min) | HuggingFace download | Realistic workload |
|
||||
| `synthetic` (fixed ISL/OSL) | Instant | Local generation | Capacity testing, reproducibility |
|
||||
|
||||
## Orchestration Pattern
|
||||
|
||||
YAML base + CLI sweeps:
|
||||
```bash
|
||||
for conc in 1 5 10 25 50 100; do
|
||||
aiperf profile --config capacity.yaml --concurrency "$conc" \
|
||||
--request-count 100 --artifact-dir "results/conc${conc}"
|
||||
done
|
||||
```
|
||||
|
||||
## Validation Errors & Quick Fixes
|
||||
|
||||
| Error | Fix |
|
||||
|-------|-----|
|
||||
| `1 validation error for AIPerfConfig` | Wrap YAML in `schema_version: "2.0"` / `benchmark:` envelope |
|
||||
| `Phase 'profiling': at least one of 'requests', 'duration', or 'sessions' must be specified` | Add `requests: N` or `duration: N` to phase |
|
||||
| `NumPy: baseline optimizations (X86_V2) not supported` | `pip install "numpy<2.0" --force-reinstall` |
|
||||
@@ -0,0 +1,133 @@
|
||||
# Long-Context Prefill Test – Memory Bandwidth Bottleneck
|
||||
|
||||
## Purpose
|
||||
|
||||
Isolate the **memory-bandwidth prefill bottleneck** (relevant for RAG, document analysis, long-context agents). While capacity tests scale concurrency with short inputs, this test scales **input length** at fixed low concurrency to find where KV-cache limits hit.
|
||||
|
||||
## When to Use
|
||||
|
||||
- The capacity curve shows conc=100 is already the practical ceiling
|
||||
- The real workload involves **long documents** (1K–8K tokens) not many short chats
|
||||
- You need to answer: "Can this model handle our RAG pipeline with 4K context windows?"
|
||||
|
||||
## AIPerf Config
|
||||
|
||||
Use `type: "synthetic"` with `fixed` ISL. The `prompt_template` must include a variable that expands to the target length. AIPerf's synthetic engine automatically generates filler text to reach the target token count.
|
||||
|
||||
### Template: 1024-token input
|
||||
|
||||
```yaml
|
||||
schema_version: "2.0"
|
||||
benchmark:
|
||||
models:
|
||||
items:
|
||||
- name: "vllm/REPLACE-MODEL-NAME"
|
||||
endpoint:
|
||||
urls: ["https://ENDPOINT/v1"]
|
||||
type: "chat"
|
||||
streaming: true
|
||||
datasets:
|
||||
default:
|
||||
- name: "synthetic_long"
|
||||
type: "synthetic"
|
||||
prompt_template:
|
||||
content: "Summarize the following text concisely: {{text}}"
|
||||
variables:
|
||||
- name: "text"
|
||||
values: [
|
||||
"artificial intelligence",
|
||||
"quantum computing",
|
||||
"blockchain technology",
|
||||
"climate change",
|
||||
"machine learning",
|
||||
"renewable energy"
|
||||
]
|
||||
strategy:
|
||||
type: "fixed"
|
||||
max_total_tokens: 1152 # input + output must fit
|
||||
input_tokens: 1024 # target input length
|
||||
output_tokens: 128 # short output to focus on prefill
|
||||
phases:
|
||||
- type: "concurrency"
|
||||
name: "prefill_stress"
|
||||
duration: 300 # 5 minutes, time-based
|
||||
sessions: 10 # fixed low concurrency
|
||||
tokenizer:
|
||||
name: "builtin"
|
||||
```
|
||||
|
||||
### Scaling input length
|
||||
|
||||
Create a sweep by varying `input_tokens` across runs:
|
||||
|
||||
| Run | input_tokens | max_total_tokens | What it tests |
|
||||
|-----|-------------|------------------|---------------|
|
||||
| 1 | 512 | 640 | Baseline (standard context) |
|
||||
| 2 | 1024 | 1152 | Long context (RAG documents) |
|
||||
| 3 | 2048 | 2176 | Very long context |
|
||||
| 4 | 4096 | 4224 | Extreme context (edge case) |
|
||||
|
||||
**Constraint:** `max_total_tokens >= input_tokens + output_tokens + 16` (AIPerf padding). The model's own context window (`max_model_len`) is the hard ceiling—check the endpoint's `/v1/models` response.
|
||||
|
||||
## CLI Execution
|
||||
|
||||
```bash
|
||||
for islen in 512 1024 2048; do
|
||||
model="vllm/gemma-4-31b-it"
|
||||
model_safe=$(echo "$model" | tr '/.' '_')
|
||||
|
||||
# Generate per-run config or use sed to modify input_tokens
|
||||
sed "s/input_tokens: 1024/input_tokens: ${islen}/; s/max_total_tokens: 1152/max_total_tokens: $((islen + 128 + 16))/" \
|
||||
long-context-template.yaml > /tmp/run.yaml
|
||||
|
||||
venv/bin/aiperf profile \
|
||||
--config /tmp/run.yaml \
|
||||
--api-key "$API_KEY" \
|
||||
--concurrency 10 \
|
||||
--ui none \
|
||||
--no-gpu-telemetry \
|
||||
--no-server-metrics \
|
||||
--artifact-dir "results/longcontext/${model_safe}/isl${islen}"
|
||||
done
|
||||
```
|
||||
|
||||
## Metrics to Extract
|
||||
|
||||
From `profile_export_aiperf.json`:
|
||||
|
||||
| Metric | Key | Unit | Interpretation |
|
||||
|--------|-----|------|----------------|
|
||||
| Prefill throughput | `prefill_throughput_per_user.avg` | tok/s/user | How fast a single user's long prompt is processed |
|
||||
| TTFT | `time_to_first_token.avg` / `.p99` | ms | Time until first token at long context |
|
||||
| Input length | `input_sequence_length.avg` | tokens | Actual achieved input length (verify synthetic hit target) |
|
||||
| Output/User | `output_token_throughput_per_user.avg` | tok/s/user | Generation speed (should be stable; if it drops, KV-cache eviction) |
|
||||
|
||||
## Expected Behavior
|
||||
|
||||
Healthy system:
|
||||
- Prefill throughput **degrades gracefully** with longer inputs (sub-linear)
|
||||
- TTFT scales roughly linearly with input length: `TTFT ≈ input_tokens / prefill_tps`
|
||||
- Output/User stays constant (generation is independent of input length)
|
||||
|
||||
Unhealthy system (KV-cache limits):
|
||||
- Prefill throughput **collapses** at a threshold length (e.g., 2048 tokens)
|
||||
- TTFT becomes super-linear (quadratic or worse)
|
||||
- Output/User drops (KV-cache thrashing or CPU offloading)
|
||||
|
||||
## Interpreting Results
|
||||
|
||||
Calculate **prefill time per 1K tokens**:
|
||||
|
||||
```
|
||||
Prefill time per 1K = 1000 / prefill_throughput_per_user (seconds)
|
||||
```
|
||||
|
||||
From observed conc=100 data (short-input baseline):
|
||||
|
||||
| Model | Prefill/User @ 512 tok | Est. per 1K |
|
||||
|-------|----------------------|-------------|
|
||||
| gemma-4-31b-it | 505 tok/s | ~2.0s |
|
||||
| moonshotai-kimi-k2.6 | 276 tok/s | ~3.6s |
|
||||
| gpt-oss-120b | 137 tok/s | ~7.3s |
|
||||
|
||||
If the long-context test at 1024 tokens shows significantly worse than double these estimates, the model/server has **non-linear KV-cache scaling**—a hard limit for RAG workloads.
|
||||
@@ -0,0 +1,109 @@
|
||||
# AIPerf Long-Input and Extreme-Context Benchmarking
|
||||
|
||||
Session: 2026-06-18 — Full suite benchmark of 4 models (gemma-4-31b-it, gpt-oss-120b, moonshotai/kimi-k2.6, qwen3.6-27b-nvfp4) on ai.noris.de
|
||||
|
||||
## Key Finding: Long Inputs > High Concurrency for Bottleneck Isolation
|
||||
|
||||
When evaluating maximum node capacity, testing conc=200+ is often operationally irrelevant because TTFT p99 exceeds acceptable thresholds (7s+ at conc=100). The real infrastructure bottleneck for RAG / long-context workloads is **memory-bandwidth-bound prefill** with long inputs.
|
||||
|
||||
Recommended alternative: **Long-Input Prefill Test**
|
||||
- Concurrency: Fixed low (e.g., 10) to avoid queueing artifacts
|
||||
- Input length: 1024 / 2048 / 4096 tokens (double/triple the standard 512)
|
||||
- Output length: Short (128) to isolate prefill from generation
|
||||
- Duration: 300s to amortize cold-start
|
||||
|
||||
## Extreme Context Test (10k Input / 10k Output)
|
||||
|
||||
For stress-testing context-window limits and sustained generation capacity:
|
||||
|
||||
- **Input:** 10,000 tokens
|
||||
- **Output:** 10,000 tokens
|
||||
- **Concurrency:** 1 (sequentially, because runtime is 200–1000 seconds per request)
|
||||
- **Requests:** 10 per model
|
||||
- **Purpose:** Validates the full KV-cache lifecycle and sustained decode performance
|
||||
|
||||
**Expected runtime:** At 10–50 tok/s output, one request takes 200–1000 seconds. Schedule accordingly.
|
||||
|
||||
**Template:**
|
||||
```yaml
|
||||
schema_version: "2.0"
|
||||
benchmark:
|
||||
models:
|
||||
items:
|
||||
- name: "vllm/MODEL_NAME"
|
||||
endpoint:
|
||||
urls: ["https://ai.noris.de/v1"]
|
||||
type: "chat"
|
||||
streaming: true
|
||||
datasets:
|
||||
- type: "synthetic"
|
||||
name: "main"
|
||||
entries: 10
|
||||
isl:
|
||||
type: "fixed"
|
||||
value: 10000
|
||||
osl:
|
||||
type: "fixed"
|
||||
value: 10000
|
||||
phases:
|
||||
- type: "concurrency"
|
||||
name: "profiling"
|
||||
requests: 10
|
||||
```
|
||||
|
||||
## Correct AIPerf 0.10.0 Synthetic Dataset Config (Long Input)
|
||||
|
||||
```yaml
|
||||
schema_version: "2.0"
|
||||
benchmark:
|
||||
models:
|
||||
items:
|
||||
- name: "vllm/gemma-4-31b-it"
|
||||
endpoint:
|
||||
urls: ["https://ai.noris.de/v1"]
|
||||
type: "chat"
|
||||
streaming: true
|
||||
datasets:
|
||||
- name: "synthetic_long"
|
||||
type: "synthetic"
|
||||
prompt_template:
|
||||
content: "Analyze the following topic in depth and provide detailed insights."
|
||||
strategy:
|
||||
type: "fixed"
|
||||
max_total_tokens: 1152
|
||||
input_tokens: 1024
|
||||
output_tokens: 128
|
||||
phases:
|
||||
- type: "concurrency"
|
||||
name: "profiling"
|
||||
duration: 300
|
||||
```
|
||||
|
||||
**Critical YAML structure rules learned:**
|
||||
- `datasets` MUST be a **list** (`- name: ...`), NOT a dict with `default:` key
|
||||
- `name` goes directly under the list item, NOT nested under a `synthetic:` key
|
||||
- `prompt_template` and `strategy` are siblings to `name` and `type`
|
||||
|
||||
## Benchmark Results at conc=100 (Standard 512-in / 128-out)
|
||||
|
||||
| Model | Input tok/s (total) | Output tok/s (total) | Total tok/s | TTFT p99 |
|
||||
|-------|--------------------|---------------------|-------------|----------|
|
||||
| moonshotai-kimi-k2.6 | 5,115 | 1,270 | 6,385 | 3,926ms |
|
||||
| gemma-4-31b-it | 4,385 | 1,083 | 5,468 | 2,060ms |
|
||||
| qwen3.6-27b-nvfp4 | 4,191 | 992 | 5,183 | 7,391ms |
|
||||
| gpt-oss-120b | 3,300 | 804 | 4,104 | 7,310ms |
|
||||
|
||||
**Prefill degradation @ conc=100:** All models lose ~80% of per-user prefill speed vs conc=10.
|
||||
|
||||
## MSP Service-Tier Perspective (User Preference)
|
||||
|
||||
When presenting for infrastructure/commercial decisions, structure per **service tier** not model comparison:
|
||||
- Single-User Experience (conc=1): TTFT p99, Output/User, SLA class
|
||||
- Multi-Tenant Load (conc=10→100): Throughput scaling, degradation curve
|
||||
- Steady-State: Long-run stability, drift detection
|
||||
- Service Recommendation: Positioning (Entry/Premium), suitable use cases
|
||||
|
||||
## Related Files
|
||||
- `templates/aiperf-long-input-config.yaml` — Working starter template
|
||||
- `references/aiperf-suite-orchestrator-pattern.md` — Full orchestration pattern
|
||||
- `references/aiperf-token-metrics-explained.md` — Token metric unit reference
|
||||
@@ -0,0 +1,109 @@
|
||||
# Prefill Throughput Bias from Cold-Start Requests
|
||||
|
||||
## Problem
|
||||
|
||||
AIPerf's per-run `profile_export_aiperf.json` aggregates ALL requests, including the very first one which has to establish a fresh HTTP connection. This biases the `prefill_throughput_per_user` metric significantly:
|
||||
|
||||
| Request | Input Tokens | Prefill tok/s | TTFT | Connection |
|
||||
|---------|-------------|---------------|------|------------|
|
||||
| 0 | 7 | **4 tok/s** | 1699 ms | DNS (1002ms) + TCP (1144ms) |
|
||||
| 1 | 371 | 2003 tok/s | 185 ms | Reused ✓ |
|
||||
| 2 | 676 | 3596 tok/s | 188 ms | Reused ✓ |
|
||||
| 5 | 1488 | 7571 tok/s | 197 ms | Reused ✓ |
|
||||
|
||||
The first request's DNS lookup + TCP handshake (2+ seconds) is counted toward "prefill time", deflating the metric. The **true** prefill throughput (requests 1+) is ~2K–7.5K tok/s, not the ~4K aggregate.
|
||||
|
||||
## Impact
|
||||
|
||||
- **Prefill metrics**: Inflated by ~1.5–2× for short runs (<20 requests). Improves as sample size grows.
|
||||
- **TTFT p99**: May be driven by the warm-up request's elevated TTFT.
|
||||
- **Low-concurrency runs** (conc=1, single session) are most affected because only ONE session exists and its first request is always cold.
|
||||
|
||||
## Mitigation
|
||||
|
||||
### Option A: Discard First Request (Recommended for Prescriptive Benchmarks)
|
||||
|
||||
Parse `profile_export.jsonl` (line-delimited JSON, one request per line) and exclude `session_num=0` / `turn_index=0`:
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
metrics = []
|
||||
with open("profile_export.jsonl") as f:
|
||||
for line in f:
|
||||
req = json.loads(line)
|
||||
# Skip first turn of first session (cold-start)
|
||||
if req["metadata"]["turn_index"] == 0 and req["metadata"]["session_num"] == 0:
|
||||
continue
|
||||
m = req["metrics"]
|
||||
metrics.append({
|
||||
"ttft": m["time_to_first_token"]["value"],
|
||||
"prefill": m["prefill_throughput_per_user"]["value"],
|
||||
})
|
||||
|
||||
# Now metrics[] excludes the warm-up request
|
||||
```
|
||||
|
||||
### Option B: Use Connection Warm-Up Before Benchmarking
|
||||
|
||||
Send 1–2 throwaway requests to establish the TCP + TLS + HTTP/2 connection pool before the measured phase. AIPerf does not have a built-in warm-up phase, so this must be done externally:
|
||||
|
||||
```bash
|
||||
# Pre-warm the connection
|
||||
venv/bin/aiperf profile --config warmup.yaml \
|
||||
--concurrency 1 --request-count 1 \
|
||||
--artifact-dir /tmp/warmup
|
||||
|
||||
# Now run the real benchmark — connections are hot
|
||||
venv/bin/aiperf profile --config benchmark.yaml \
|
||||
--concurrency 1 --request-count 50 \
|
||||
--artifact-dir results/benchmark
|
||||
```
|
||||
|
||||
The warmup run should use the **same endpoint, model, and streaming=true** as the benchmark to ensure the connection pool is pre-established.
|
||||
|
||||
### Option C: Accept Bias for Comparative Benchmarks
|
||||
|
||||
If comparing **N models on the same infra with the same settings**, the cold-start bias applies equally to all. The absolute numbers are slightly off, but **rankings and relative ratios remain valid**. Use this only when:
|
||||
- Absolute SLA numbers are not the goal
|
||||
- All runs use identical connection topology (same client → same endpoint)
|
||||
- Sample size is large enough that one cold request is <5% of total
|
||||
|
||||
## When It Matters Most
|
||||
|
||||
| Scenario | Severity | Why |
|
||||
|----------|----------|-----|
|
||||
| conc=1, sessions=1 | 🔴 High | First request is 100% of traffic |
|
||||
| conc=5, sessions=5 | 🟡 Medium | 1 of 5 sessions starts cold |
|
||||
| conc=50, sessions=50 | 🟢 Low | Many sessions, cold-start amortized |
|
||||
| Capacity (10→100+) | 🟢 Low | Many sessions, cold-start amortized |
|
||||
| Steady-state (10min) | 🟢 Low | Thousands of requests, negligible |
|
||||
|
||||
## Per-Request Data Location
|
||||
|
||||
Raw per-request data is in `profile_export.jsonl` (NOT `profile_export_aiperf.json` which is aggregated). Each line is a JSON object with:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"session_num": 0,
|
||||
"turn_index": 0,
|
||||
"worker_id": "worker_766ae0c2",
|
||||
"conversation_id": "...",
|
||||
"was_cancelled": false
|
||||
},
|
||||
"metrics": {
|
||||
"time_to_first_token": {"value": 1699.2, "unit": "ms"},
|
||||
"prefill_throughput_per_user": {"value": 4.1, "unit": "tokens/sec/user"},
|
||||
"request_latency": {"value": 26372.4, "unit": "ms"},
|
||||
"http_req_connection_reused": {"value": 0, "unit": "ratio"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Key indicators of a cold request:
|
||||
- `"http_req_connection_reused": {"value": 0}` — fresh connection
|
||||
- `"http_req_dns_lookup"` > 500ms — DNS resolution happened
|
||||
- `"http_req_connecting"` > 500ms — TCP/TLS handshake happened
|
||||
|
||||
A reused connection has `"http_req_connection_reused": {"value": 1}` and near-zero DNS/connect times.
|
||||
@@ -0,0 +1,120 @@
|
||||
# AIPerf Suite Orchestrator Pattern
|
||||
|
||||
Reference: building a 12-run benchmark suite (4 models × 3 scenarios) against ai.noris.de.
|
||||
|
||||
## Directory Convention
|
||||
|
||||
```
|
||||
aiperf-benchmark-suite/
|
||||
├── configs/
|
||||
│ ├── gemma-4-31b-it/
|
||||
│ │ ├── sales.yaml # ShareGPT dataset, requests: 50, sessions: 1
|
||||
│ │ ├── capacity.yaml # Synthetic 512/128, requests: 100
|
||||
│ │ └── steady.yaml # ShareGPT, duration: 600
|
||||
│ ├── gpt-oss-120b/...
|
||||
│ └── ... (one dir per model)
|
||||
├── scripts/
|
||||
│ ├── run-suite.sh # Orchestrator
|
||||
│ ├── generate-summary.py # Markdown table from JSON
|
||||
│ └── plot-results.py # Matplotlib charts + CSV
|
||||
├── results/
|
||||
│ └── YYYY-MM-DD-HHMMSS/
|
||||
│ ├── gemma-4-31b-it/
|
||||
│ │ ├── sales/conc1/
|
||||
│ │ ├── sales/conc5/
|
||||
│ │ ├── sales/conc10/
|
||||
│ │ ├── capacity/conc10/
|
||||
│ │ ├── capacity/conc25/
|
||||
│ │ ├── capacity/conc50/
|
||||
│ │ ├── capacity/conc100/
|
||||
│ │ └── steady/
|
||||
│ └── ...
|
||||
└── venv/ # AIPerf installed here
|
||||
```
|
||||
|
||||
## Orchestrator Structure
|
||||
|
||||
The orchestrator is a bash script that:
|
||||
|
||||
1. **Sets absolute paths** — `AIPERF="$BASE_DIR/venv/bin/aiperf"` (never rely on `$PATH`)
|
||||
2. **Validates env** — checks `NORIS_API_KEY` and connectivity
|
||||
3. **Loops models × scenarios**
|
||||
4. **Per scenario, loops concurrency levels** (for count-based scenarios)
|
||||
5. **Calls AIPerf with CLI overrides** for each sub-run
|
||||
6. **Pauses between runs** (`sleep 5-30`) to avoid rate limits
|
||||
7. **Generates summary + plots** after all runs complete
|
||||
8. **Continues on error** (`|| true`) — one failed sub-run doesn't kill the suite
|
||||
|
||||
## Critical: Venv Path Fix
|
||||
|
||||
When running in background (cron, detached shell, Telegram-triggered runs), `$PATH` does NOT include the venv. The orchestrator MUST:
|
||||
|
||||
```bash
|
||||
AIPERF="$BASE_DIR/venv/bin/aiperf"
|
||||
if [ ! -x "$AIPERF" ]; then
|
||||
echo "ERROR: AIPerf not found at $AIPERF"
|
||||
exit 1
|
||||
fi
|
||||
# Then call "$AIPERF" profile ... everywhere
|
||||
```
|
||||
|
||||
Failure mode: `aiperf: command not found` on every run, producing empty results silently.
|
||||
|
||||
## CLI Override Pattern for Suite Runs
|
||||
|
||||
The YAML configs define the base structure. CLI flags sweep parameters:
|
||||
|
||||
```bash
|
||||
# Sales scenario: sweep concurrencies
|
||||
for conc in 1 5 10; do
|
||||
"$AIPERF" profile --config "$CONFIG" \
|
||||
--concurrency "$conc" \
|
||||
--request-count 50 \
|
||||
--artifact-dir "results/sales/conc${conc}"
|
||||
done
|
||||
|
||||
# Capacity scenario: stepped ramp
|
||||
for conc in 10 25 50 100; do
|
||||
"$AIPERF" profile --config "$CONFIG" \
|
||||
--concurrency "$conc" \
|
||||
--request-count 100 \
|
||||
--artifact-dir "results/capacity/conc${conc}"
|
||||
done
|
||||
|
||||
# Steady scenario: time-based
|
||||
"$AIPERF" profile --config "$CONFIG" \
|
||||
--concurrency 20 \
|
||||
--artifact-dir "results/steady"
|
||||
```
|
||||
|
||||
## Pre-Download Strategy for ShareGPT
|
||||
|
||||
First sales run downloads ShareGPT from HuggingFace (~2–5 min). To avoid this during production:
|
||||
|
||||
```bash
|
||||
# One-time warm-up (any model config with public dataset works)
|
||||
venv/bin/aiperf profile --config configs/any-model/sales.yaml \
|
||||
--concurrency 1 --request-count 1 \
|
||||
--artifact-dir /tmp/warmup --no-gpu-telemetry --no-server-metrics
|
||||
```
|
||||
|
||||
Subsequent runs use `.cache/aiperf/datasets/` instantly.
|
||||
|
||||
## Post-Processing Sequence
|
||||
|
||||
After the orchestrator finishes all runs:
|
||||
|
||||
```bash
|
||||
# 1. Markdown summary
|
||||
python3 scripts/generate-summary.py results/YYYY-MM-DD-HHMMSS
|
||||
|
||||
# 2. Plots + CSV
|
||||
python3 scripts/plot-results.py results/YYYY-MM-DD-HHMMSS
|
||||
```
|
||||
|
||||
Outputs:
|
||||
- `results/.../README.md` – human-readable markdown table
|
||||
- `results/.../plots/summary.csv` – Excel / Google Sheets import
|
||||
- `results/.../plots/capacity_curve.png` – throughput vs concurrency
|
||||
- `results/.../plots/latency_curve.png` – TTFT p99 vs concurrency
|
||||
- `results/.../plots/sales_comparison.png` – low-concurrency latency
|
||||
@@ -0,0 +1,82 @@
|
||||
# AIPerf Token-Metriken: Einheit & Interpretation
|
||||
|
||||
AIPerf exportiert mehrere Metriken die alle "tokens/sec" im Namen tragen, aber unterschiedliche Einheiten haben. Das führt zu Fehlinterpretationen wenn man die falsche Metrik für die falsche Frage verwendet.
|
||||
|
||||
## Metrik-Übersicht
|
||||
|
||||
| Metrik (JSON-Key) | Einheit | Bedeutung |
|
||||
|-------------------|---------|-----------|
|
||||
| `prefill_throughput_per_user` | **tokens/sec/user** | Wie schnell der Prompt **eines Users** verarbeitet wird (Prompt → erstes Token) |
|
||||
| `output_token_throughput` | **tokens/sec** | **Gesamter Output-Durchsatz** aller parallelen Sessions zusammen |
|
||||
| `output_token_throughput_per_user` | **tokens/sec/user** | Output-Generierungsgeschwindigkeit, **pro einzelnem User** |
|
||||
| `e2e_output_token_throughput` | **tokens/sec/user** | End-to-End Throughput: Gesamtzeit (inkl. TTFT) / Output-Tokens, **pro User** |
|
||||
| `total_token_throughput` | **tokens/sec** | (Input + Output)-Tokens gesamt, alle Sessions |
|
||||
| `request_throughput` | **requests/sec** | Anzahl abgeschlossener Requests pro Sekunde |
|
||||
|
||||
## Kritische Unterscheidung
|
||||
|
||||
### „Per User" = Session-Rate
|
||||
Diese Metriken beschreiben das Erlebnis **eines einzelnen Endnutzers**:
|
||||
|
||||
- `prefill_throughput_per_user: 505` → Jeder User bekommt seinen Prompt mit 505 tok/s verarbeitet
|
||||
- `output_token_throughput_per_user: 13.4` → Jeder User sieht ~13.4 neue Tokens pro Sekunde (die "Schreibgeschwindigkeit" des Modells)
|
||||
- `e2e_output_token_throughput: 11.9` → Inklusive Wartezeit bis zum ersten Token braucht der Request pro Output-Token länger
|
||||
|
||||
**Wichtig:** Bei `prefill_throughput_per_user` ist der Einheitshinweis irreführend. Es ist keine Durchschnittsbildung über alle Users, sondern tatsächlich die Rate **pro Session**. Bei 100 parallelen Sessions mit je 505 tok/s benötigt der Node also insgesamt **50.500 tok/s Prefill-Kapazität** (was nicht direkt gemessen wird).
|
||||
|
||||
### „Gesamt" = Node-Rate
|
||||
Diese Metriken beschreiben die **gesamte Leistung der Inferenz-Instanz**:
|
||||
|
||||
- `output_token_throughput: 1083` → Alle 100 Sessions zusammen generieren 1083 Output-Tokens pro Sekunde
|
||||
- `total_token_throughput: 5468` → (Input + Output) zusammen → inklusive Prefill-Arbeit
|
||||
|
||||
## Praktische Berechnungen
|
||||
|
||||
### Input Token/s (nicht direkt gemessen)
|
||||
"Input" oder "Prefill" in Gesamtzahlen ist nicht separat exportiert. Annäherung:
|
||||
|
||||
```
|
||||
input_tokens/sec ≈ total_token_throughput - output_token_throughput
|
||||
```
|
||||
|
||||
Beispiel (gemma @ conc=100):
|
||||
- total: 5468 tok/s
|
||||
- output: 1083 tok/s
|
||||
- → input ≈ 4385 tok/s
|
||||
|
||||
### Gesamtleistungs-Ranking
|
||||
Für Kapazitätsplanung (wie viele Tokens kann der Node pro Sekunde verarbeiten?) verwende `total_token_throughput`:
|
||||
|
||||
| Modell @ conc=100 | Input t/s | Output t/s | **Total t/s** |
|
||||
|-------------------|-----------|------------|---------------|
|
||||
| moonshotai-kimi-k2.6 | ~5115 | 1270 | **6385** |
|
||||
| gemma-4-31b-it | ~4385 | 1083 | **5468** |
|
||||
| qwen3.6-27b-nvfp4 | ~4191 | 992 | **5183** |
|
||||
| gpt-oss-120b | ~3300 | 804 | **4104** |
|
||||
|
||||
### Erlebnis-Ranking
|
||||
Für Kundenerfahrung (was spürt ein einzelner User?) verwende die _per_user-Metriken:
|
||||
|
||||
| Modell @ conc=100 | Prefill/User | Output/User |
|
||||
|-------------------|-------------|-------------|
|
||||
| moonshotai-kimi-k2.6 | 276 | 17.9 |
|
||||
| qwen3.6-27b-nvfp4 | 226 | 15.7 |
|
||||
| gemma-4-31b-it | 505 | 13.4 |
|
||||
| gpt-oss-120b | 137 | 12.3 |
|
||||
|
||||
## Fehlerquellen
|
||||
|
||||
1. **Prefill mit Output vergleichen** – Prefill ist `tok/s/user`, Output-Gesamt ist `tok/s`. Ein Modell mit hohem `prefill_throughput_per_user` und niedrigem `output_token_throughput` ist ein "Prompt-Monster" aber ein "Generation-Schnarcher".
|
||||
|
||||
2. **Gesamtleistung mit User-Erlebnis verwechseln** – Ein Node mit 10.000 tok/s Gesamtleistung bei 500 concurrent Sessions gibt jedem User nur 20 tok/s. Das ist unbrauchbar für Chat.
|
||||
|
||||
3. **Input/Prefill vernachlässigen** – Bei RAG-Anwendungen (lange Contexte) dominiert der Prefill die Gesamtlaufzeit. Ein Modell mit 505 tok/s Prefill braucht für 4K Context ~8 Sekunden bis zum ersten Token.
|
||||
|
||||
## Empfohlene KPIs pro Use-Case
|
||||
|
||||
| Use Case | Primäre Metrik | Sekundäre Metrik | Warnschwelle |
|
||||
|----------|---------------|------------------|-------------|
|
||||
| Chat-UI (Echtzeit) | `output_token_throughput_per_user` | `ttft_p99` | < 20 tok/s pro User |
|
||||
| API-Burst (Batch) | `output_token_throughput` (gesamt) | `request_throughput` | Steigt nicht mehr mit Conc |
|
||||
| RAG / Lang-Context | `prefill_throughput_per_user` | `ttft_p99` | > 10s für 4K Input |
|
||||
| Kapazitätsplanung | `total_token_throughput` | `ttft_p99` | Degradation > 2× |
|
||||
Reference in New Issue
Block a user