Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,342 @@
+---
+name: llm-inference-benchmarking
+description: "Benchmark LLM inference endpoints for throughput, latency, and capacity. Covers TTFT, ITL, concurrency scaling, and load patterns using AIPerf or equivalent tools."
+---
+
+# LLM Inference Benchmarking
+
+Measure the performance of generative AI inference endpoints under realistic and stress loads. This skill covers inference performance (speed, throughput, concurrency limits), NOT accuracy benchmarks like MMLU or GSM8K—use evaluating-llms-harness for those.
+
+## Trigger
+
+Use when:
+- Setting up performance benchmarks for LLM APIs (OpenAI-compatible, vLLM, TGI, etc.)
+- Determining capacity limits ("how many concurrent users can this model handle?")
+- Producing sales/reliability metrics (latency percentiles, throughput numbers)
+- Comparing model deployments or hardware configurations
+- Regression testing inference endpoints after updates
+
+## Primary Tool: AIPerf
+
+AIPerf (NVIDIA/ai-dynamo) is the recommended tool. **Version 0.10.0+ uses a new Schema 2.0 YAML format**—see Config Format section below.
+
+It supports:
+- OpenAI chat/completions/embeddings/audio/vision endpoints
+- Concurrency, request-rate, trace-replay, and ramping modes
+- ShareGPT and custom datasets
+- Real-time dashboard, CSV/JSON exports, and automatic plotting
+- Multiprocess architecture via ZMQ
+
+Installation:
+```bash
+pip install aiperf
+# Optional extras:
+pip install "aiperf[wandb]"   # historical trend tracking
+pip install "aiperf[otel]"    # live telemetry streaming
+```
+
+Platform notes:
+- **Linux aarch64**: `crick` dependency needs build tools: `sudo apt install build-essential`
+- **Older x86_64 VMs without X86_V2**: NumPy 2.x crashes with `RuntimeError: NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support: (X86_V2)`. Fix: `pip install "numpy<2.0" --force-reinstall`. This applies to older cloud VMs, Proxmox containers, or legacy hardware that lacks AVX2/BMI2/MOVBE/POPCNT/FMA.
+
+## AIPerf 0.10.0+ Config Format (Schema 2.0)
+
+> **Critical**: AIPerf 0.10.0 introduced a new YAML envelope. Old flat configs are rejected.
+
+### Envelope Structure
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+    strategy: "round_robin"
+  endpoint:
+    urls:
+      - "https://ai.noris.de/v1"
+    type: "chat"
+    api_key: "${API_KEY}"          # Can reference env var; plaintext also works
+    streaming: true
+    headers:
+      Authorization: "Bearer ${API_KEY}"
+  datasets:
+    - type: "synthetic"            # or "public" for ShareGPT
+      name: "main"
+      entries: 100
+      isl:
+        type: "fixed"
+        value: 512
+      osl:
+        type: "fixed"
+        value: 128
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      # One of requests, duration, or sessions MUST be set
+      requests: 50                  # total requests for this phase
+      sessions: 1                   # concurrent sessions (default: 1)
+      # duration: 600               # alternative to requests
+  tokenizer:
+    name: "builtin"                 # or huggingface model name
+```
+
+### Key fields explained
+- `models.items[].name`: The model identifier (must match endpoint's `/v1/models` list)
+- `endpoint.api_key`: Can be `${ENV_VAR}` reference. CLI `--api-key` overrides this.
+- `datasets.type`: `"public"` downloads ShareGPT from HuggingFace (first run ~2–5 min). `"synthetic"` generates locally with no download.
+- `datasets.isl` / `datasets.osl`: Input/Output Sequence Length distributions. `type: "fixed"` with `value: N` for deterministic lengths.
+- `phases`: Must specify at least one of `requests`, `duration`, or `sessions`. Default `concurrency` inside phase is 1.
+
+### CLI Overrides
+AIPerf allows overriding YAML values at CLI time. The most useful for suite orchestration:
+
+| CLI flag | Overrides YAML field | Typical use |
+|----------|----------------------|-------------|
+| `--concurrency N` | phase.sessions (indirectly) | Sweep concurrency levels |
+| `--request-count N` | phase.requests | Control total requests per sub-run |
+| `--duration N` | phase.duration | Time-based runs (steady-state) |
+| `--artifact-dir PATH` | output directory | Organize results per run |
+| `--ui simple` | dashboard mode | Less overhead, no TUI |
+| `--no-gpu-telemetry` | - | Skip DCGM (endpoints without GPU metrics) |
+| `--no-server-metrics` | - | Skip Prometheus scraping (non-vLLM endpoints) |
+
+**Orchestration pattern**: Define base structure in YAML, then use CLI flags in a shell loop for parameter sweeps:
+```bash
+for conc in 1 5 10 25 50 100; do
+  aiperf profile --config capacity.yaml --concurrency "$conc" --request-count 100 \
+    --artifact-dir "results/capacity/conc${conc}"
+done
+```
+
+## The 3-Scenario Suite Pattern
+
+Design benchmark suites around three complementary scenarios:
+
+### 1. Sales Scenario (Representative Load)
+- **Goal**: Clean numbers for pitch decks and SLAs
+- **Concurrency**: 1, 5, 10 (low, moderate, realistic)
+- **Dataset**: ShareGPT (realistic chat transcripts)
+- **Count**: ~50 requests per run
+- **Output**: avg/p99 TTFT, throughput, request latency
+
+### 2. Capacity Scenario (Breakpoint Discovery)
+- **Goal**: Find the concurrency ceiling before degradation
+- **Concurrency**: 10 → 25 → 50 → 100 (stepped ramp)
+- **Dataset**: Synthetic with fixed ISL/OSL (e.g., 512 IN / 128 OUT)
+- **Count**: ~100 requests per level
+- **Output**: Performance curve (throughput vs. concurrency, latency inflection points)
+
+**When to stop scaling concurrency**: Before proposing conc=200+ tests, extrapolate the existing curve. If TTFT p99 exceeds ~5s at conc=100, further scaling is operationally irrelevant—users won't accept 15s+ wait times. Better to test **longer inputs** (1024/2048 tokens) to isolate the memory-bandwidth prefill bottleneck, or implement queueing + load balancing.
+
+### 2b. Long-Context Prefill Test (Memory Bandwidth Bottleneck)
+- **Goal**: Isolate the memory-bandwidth prefill bottleneck (relevant for RAG, long-context agents)
+- **Concurrency**: Fixed low (e.g., 10) to avoid queueing artifacts
+- **Dataset**: Synthetic with **long input** (1024 / 2048 / 4096 tokens), short output (128)
+- **Duration**: 300s (long enough to amortize cold-start)
+- **Output**: Prefill throughput vs. input length curve
+
+This is more informative than conc=200 when the real workload involves reading long documents.
+
+### 3. Steady-State Scenario (Long-Term Stability)
+- **Goal**: Detect drift, memory leaks, or throttling over time
+- **Concurrency**: Sweet spot from Capacity scenario (e.g., 20)
+- **Duration**: 600 seconds (10 minutes) or longer
+- **Dataset**: Mixed (50% ShareGPT + 50% synthetic short/long)
+- **Output**: P50/P90/P99 trends over time, failed request rate
+
+### Universal Tokenizer for Heterogeneous Endpoints
+
+When benchmarking multiple models with different vocabularies (and downloading the correct HuggingFace tokenizer for each is impractical), use:
+
+```bash
+--tokenizer builtin
+```
+
+This uses tiktoken's `o200k_base` encoding universally. It avoids HF downloads, gated-repo access issues, and rate limits. **Trade-off:** Token counts are approximate for non-OpenAI models (different vocabularies), but **TTFT, ITL, and latency metrics remain exact**. Recommended when:
+- Tokenizer mapping is complicated (4+ models, gated repos, missing HF entries)
+- Running in air-gapped or rate-limited environments
+- Sales/capacity benchmark where relative comparisons matter more than absolute token counts
+
+Before benchmarking, discover available models from the target endpoint:
+
+```bash
+curl -s -H "Authorization: Bearer $API_KEY" \
+  https://<endpoint>/v1/models | \
+  python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin).get('data',[])]"
+```
+
+See `scripts/discover-models.sh` for a ready-to-run version.
+
+## Authentication & Security
+
+- **NEVER hardcode API keys** in YAML configs, shell scripts, or git
+- Always use environment variables: `${API_KEY}` or `$API_KEY`
+- In AIPerf YAML, reference via `--extra-headers` or templating:
+  ```yaml
+  extra-headers:
+    Authorization: "Bearer ${API_KEY}"
+  ```
+- Export in shell before running: `export API_KEY=sk-...`
+
+## AIPerf Quick Reference
+
+Basic profile run:
+```bash
+aiperf profile \
+  --model "vllm/qwen3.6-27b-nvfp4" \
+  --streaming \
+  --endpoint-type chat \
+  --tokenizer <huggingface-tokenizer-name> \
+  --url https://ai.noris.de \
+  --extra-headers "Authorization: Bearer ${API_KEY}" \
+  --concurrency 10 \
+  --request-count 50
+```
+
+Key flags:
+- `--concurrency N` – parallel requests
+- `--request-rate N` – requests per second (alternative to concurrency)
+- `--request-count N` – total requests to send
+- `--duration N` – time-based run (seconds)
+- `--streaming` – measure streaming metrics (TTFT, ITL)
+- `--output-tokens-mean N` – target output length (with caveats; see AIPerf docs)
+- `--dataset <file>` – custom dataset (ShareGPT format or synthetic)
+- `--random-seed N` – reproducibility
+
+## YAML Configs
+
+AIPerf supports YAML-driven runs for reproducibility. See `templates/aiperf-suite-config.yaml` for a complete 3-scenario suite template.
+
+Run from YAML:
+```bash
+aiperf profile -c configs/sales-scenario.yaml
+```
+
+## Results & Visualization
+
+AIPerf produces:
+- **CSV** – per-request metrics (latency, token counts, timestamps)
+- **JSON** – aggregated statistics with percentile data (`avg`, `min`, `max`, `p50`, `p90`, `p99`, `std`)
+- **Logs** – `aiperf.log` with detailed debug info
+- **Plots** – PNG charts via `aiperf plot <results-dir>`
+
+Results should be organized as:
+```
+results/YYYY-MM-DD/
+  sales/
+  capacity/
+  steady-state/
+```
+
+### Custom Post-Processing for Multi-Model Suites
+
+For sales/capacity comparison across multiple models and concurrency levels, AIPerf's built-in `plot` subcommand works on single runs. Use a custom Python post-processing script to generate cross-model comparison charts:
+
+1. **Load all** `profile_export_aiperf.json` files recursively from `results/YYYY-MM-DD/`
+2. **Extract key metrics** per `(model, scenario, concurrency)`:
+   - `output_token_throughput.avg` = throughput (tok/s) **TOTAL**
+   - `time_to_first_token.p99` = tail latency (ms)
+   - `inter_token_latency.p99` = streaming smoothness (ms)
+   - `prefill_throughput_per_user.avg` = prefill speed **per user session**
+   - `output_token_throughput_per_user.avg` = output speed **per user**
+   - `total_token_throughput.avg` = (input+output) tok/s **TOTAL** across all sessions
+3. **Generate plots** with matplotlib:
+   - `capacity_curve.png` – Throughput vs Concurrency (log scale on x-axis)
+   - `latency_curve.png` – TTFT p99 vs Concurrency (log scale on x-axis)
+   - `sales_comparison.png` – Latency at low concurrency (1/5/10)
+4. **Export** `summary.csv` for Excel / Google Sheets import
+
+A working implementation: `scripts/plot-suite-results.py`
+
+**Rule:** `*_per_user` metrics are for end-user SLA/UX. Total metrics (`output_token_throughput`, `total_token_throughput`) are for infrastructure capacity planning. Input throughput is approximately `total - output`.
+
+### MSP / Service Provider Perspective
+
+When presenting results for infrastructure or commercial decision-making (not model research):
+
+**Don't:** Compare Model A vs Model B on token-per-token quality.  
+**Do:** Report each model as an independent service tier with its own SLA envelope.
+
+**Structure the analysis per model:**
+- **Single-User Experience** (conc=1): TTFT p99, Output/User, SLA class (Gold/Silver/Bronze)
+- **Multi-Tenant Load** (conc=10→100): Throughput scaling, TTFT degradation, recommended max concurrency
+- **Steady-State**: Long-run stability, completed requests, throughput drift
+- **Service Recommendation**: Positioning (Entry/Premium), suitable use cases, limitations
+
+**Key metrics by business question:**
+
+| Business Question | Metric to Use | Why |
+|-------------------|---------------|-----|
+| "How many users per node?" | `total_token_throughput` (Input + Output) | Node capacity |
+| "What does each user feel?" | `output_token_throughput_per_user` | UX/SLA |
+| "Is chat real-time responsive?" | `ttft_p99` + `itl_p99` | Interaktivität |
+| "Can we handle batch jobs?" | `request_throughput` + `output_token_throughput` | Durchsatz |
+| "What breaks first under load?" | TTFT p99 scaling curve | Bottleneck |
+
+**Conc ceiling analysis:** Before proposing conc=200+ tests, extrapolate the existing capacity curve. If TTFT p99 already exceeds 7s at conc=100, further scaling is operationally irrelevant—users won't accept 15s+ wait times. Better to test longer inputs (1024/2048/4096 tokens) to find the prefill bottleneck, or implement queueing + load balancing.
+
+See `references/aiperf-token-metrics-explained.md` for the full unit-reference table.
+
+## Known Pitfalls
+
+1. **AIPerf 0.10.0+ Config Format** — The new `schema_version: "2.0"` envelope is required. Old flat configs will be rejected with `1 validation error for AIPerfConfig`. See Config Format section above. Always include `requests`, `duration`, or `sessions` in phases.
+2. **NumPy X86_V2 crash on older x86_64** — If you see `RuntimeError: NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support: (X86_V2)`, downgrade: `pip install "numpy<2.0" --force-reinstall`
+3. **ShareGPT first-run delay** — The `public` dataset type downloads ShareGPT from HuggingFace on first use (~2–5 minutes). Subsequent runs use a local cache. For quick validation, use `type: "synthetic"`. For production suites, **pre-download once** before the full orchestrated run:
+   ```bash
+   venv/bin/aiperf profile --config configs/any-model/sales.yaml \
+       --concurrency 1 --request-count 1 \
+       --artifact-dir /tmp/warmup --no-gpu-telemetry --no-server-metrics
+   ```
+   After this, `.cache/aiperf/datasets/` contains ShareGPT and all sales runs start instantly.
+4. **Tokenizer mismatch** — Always specify the correct tokenizer. AIPerf auto-detects for some models, but explicit is safer.
+5. **`--output-tokens-mean` is not guaranteed** — Unless the server supports `ignore_eos` and `min_tokens` via `--extra-inputs`.
+6. **Rate limiting** — AIPerf has built-in retry/backoff, but aggressive concurrency on shared endpoints may trigger 429s. Monitor `failed_request_count`.
+7. **Port exhaustion** — Very high concurrency (>15,000) can exhaust local ports. Reduce or adjust system limits.
+10. **Orchestrator path resolution** — When running benchmarks in background processes (cron, systemd, detached shells), `PATH` may not include a Python venv. Always use the absolute path: `AIPERF="$BASE_DIR/venv/bin/aiperf"` in shell scripts, never rely on `aiperf` being in `$PATH`. The failure mode is silent: every run produces `aiperf: command not found` but the orchestrator continues, producing empty results.
+    See `references/aiperf-background-process-pitfalls.md`.
+11. **Prefill throughput bias from cold-start requests** — The first request in any AIPerf session includes DNS resolution + TCP/TLS handshake time, which is counted toward "prefill throughput". At conc=1 this can deflate prefill metrics by 50%+. The aggregate `profile_export_aiperf.json` cannot filter this out. Use `profile_export.jsonl` (per-request data) to discard `turn_index=0` requests, or pre-warm connections with a throwaway run before the measured benchmark.
+    See `references/aiperf-prefill-warmup-bias.md`.
+12. **Token metric unit confusion** — AIPerf exports multiple "tokens/sec" metrics with different units (per-user vs. total). `prefill_throughput_per_user` is tok/s **per session**, while `output_token_throughput` is tok/s **total across all sessions**. Comparing them directly or using the wrong one for capacity planning leads to off-by-100× errors.
+    - Use `*_per_user` metrics for **end-user experience** (SLA, UX)
+    - Use `output_token_throughput` / `total_token_throughput` for **node capacity** (infrastructure sizing)
+    - Approximate input throughput: `total - output`
+    See `references/aiperf-token-metrics-explained.md`.
+
+## Orchestration
+
+For multi-scenario suites, wrap AIPerf calls in a shell script that:
+1. Validates connectivity and auth (health check)
+2. Runs scenarios sequentially
+3. Organizes results in timestamped folders
+4. Generates plots after each run
+5. Detects failures (exit codes, empty CSVs, high error rates) and continues
+
+## Session References
+
+- `references/aiperf-0.10.0-migration.md` — complete config format migration guide with validation error fixes
+- `references/aiperf-long-input-benchmarking.md` — long-input prefill bottleneck testing (replaces conc=200+ tests when memory bandwidth is the real limiter)
+- `templates/aiperf-long-input-config.yaml` — working Schema 2.0 template for 1024-token input tests
+- `references/aiperf-token-metrics-explained.md` — unit reference table: per-user vs. total tok/s, input throughput approximation
+- `references/aiperf-suite-orchestrator-pattern.md` — complete suite orchestrator pattern: directory layout, venv path fix, CLI sweep pattern, ShareGPT pre-download, post-processing sequence
+- `references/aiperf-prefill-warmup-bias.md` — cold-start bias in prefill metrics, mitigation options, per-request JSONL filtering
+- `references/aiperf-token-metrics-explained.md` — unit reference: per-user vs total metrics, input calculation, business-KPI mapping
+- `references/aiperf-capacity-curve-analysis.md` — interpreting capacity curves: when to stop scaling concurrency, breakpoint detection, extrapolation rules
+- `references/aiperf-long-context-prefill-test.md` — isolating the memory-bandwidth bottleneck with 1024/2048/4096-token inputs
+- `references/ai-noris-de-endpoint.md` — discovered model list, URL, auth pattern for ai.noris.de
+- `references/aiperf-kurzreferenz.md` — German quick reference summary
+- `templates/aiperf-suite-config-v2.yaml` — Schema 2.0 starter template with all options documented
+- `templates/long-context-prefill-test.yaml` — 1024-token input test template for RAG/memory-bandwidth analysis
+- `templates/3-scenario-suite.md` — reusable Sales/Capacity/Steady-State suite structure
+
+- **GenAI-Perf** (NVIDIA, predecessor to AIPerf) – AIPerf is the migration target
+- **locust + OpenAI client** – custom Python load testing, more flexible but more work
+- **k6 + xk6-openai** – if already using k6 for HTTP load testing
+
+- **GenAI-Perf** (NVIDIA, predecessor to AIPerf) – AIPerf is the migration target
+- **locust + OpenAI client** – custom Python load testing, more flexible but more work
+- **k6 + xk6-openai** – if already using k6 for HTTP load testing
+
+## Related Skills
+
+- evaluating-llms-harness – for accuracy benchmarks (MMLU, GSM8K, etc.)
+- serving-llms-vllm – for setting up the inference server being benchmarked