Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,111 @@
+# AIPerf 0.10.0+ Kurzreferenz
+
+## Installation
+
+```bash
+pip install aiperf
+# Extras: [mlflow] [otel] [wandb]
+```
+
+**NumPy compatibility warning:** On older x86_64 VMs without X86_V2 CPU features, NumPy 2.x (pulled by AIPerf) crashes. Pre-fix with:
+```bash
+pip install "numpy<2.0" --force-reinstall --no-cache-dir
+pip install aiperf
+```
+
+## CLI Quickstart
+
+```bash
+aiperf profile \
+  --config config.yaml \
+  --api-key "$API_KEY" \
+  --concurrency 10 \
+  --request-count 50 \
+  --artifact-dir results/
+```
+
+## Key CLI Flags (0.10.0+)
+
+| Flag | YAML Override | Description |
+|------|---------------|-------------|
+| `--config FILE` | - | Load Schema 2.0 YAML config |
+| `--api-key KEY` | `endpoint.api_key` | Auth key (overrides YAML) |
+| `--concurrency N` | phase.sessions | Concurrent requests/sessions |
+| `--request-rate N` | - | Requests/sec (alt. to concurrency) |
+| `--request-count N` | phase.requests | Total requests |
+| `--duration N` | phase.duration | Time-based run (seconds) |
+| `--streaming` | `endpoint.streaming` | Enable streaming (TTFT/ITL) |
+| `--ui simple` | `ui` | Progress bars (vs. `dashboard` TUI) |
+| `--ui none` | `ui` | Headless |
+| `--artifact-dir DIR` | `output_dir` | Results directory |
+| `--no-gpu-telemetry` | - | Skip DCGM/GPU metrics |
+| `--no-server-metrics` | - | Skip Prometheus scraping |
+| `--tokenizer builtin` | `tokenizer.name` | tiktoken (no HF download) |
+| `--random-seed N` | `random_seed` | Reproducibility |
+
+## Schema 2.0 YAML Config
+
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+    strategy: "round_robin"
+  endpoint:
+    urls:
+      - "https://ai.noris.de/v1"
+    type: "chat"
+    api_key: "${API_KEY}"
+    streaming: true
+    headers:
+      Authorization: "Bearer ${API_KEY}"
+  datasets:
+    - type: "public"        # "public" = HF download; "synthetic" = local gen
+      name: "main"
+      dataset: "sharegpt"
+      sampling: "shuffle"
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      requests: 50
+      sessions: 1
+  tokenizer:
+    name: "builtin"
+```
+
+Run from YAML:
+```bash
+aiperf profile --config config.yaml
+```
+
+## Outputs
+- **CSV** (`profile_export_aiperf.csv`): Per-request metrics
+- **JSON** (`profile_export_aiperf.json`): Aggregated stats (avg, p50, p90, p99)
+- **Log** (`logs/aiperf.log`): Detailed debug
+- **Plots**: `aiperf plot <artifact-dir>`
+
+## Dataset Types Quick Reference
+
+| Type | Speed | Source | Use |
+|------|-------|--------|-----|
+| `public` (ShareGPT) | Slow first-run (~2-5 min) | HuggingFace download | Realistic workload |
+| `synthetic` (fixed ISL/OSL) | Instant | Local generation | Capacity testing, reproducibility |
+
+## Orchestration Pattern
+
+YAML base + CLI sweeps:
+```bash
+for conc in 1 5 10 25 50 100; do
+  aiperf profile --config capacity.yaml --concurrency "$conc" \
+    --request-count 100 --artifact-dir "results/conc${conc}"
+done
+```
+
+## Validation Errors & Quick Fixes
+
+| Error | Fix |
+|-------|-----|
+| `1 validation error for AIPerfConfig` | Wrap YAML in `schema_version: "2.0"` / `benchmark:` envelope |
+| `Phase 'profiling': at least one of 'requests', 'duration', or 'sessions' must be specified` | Add `requests: N` or `duration: N` to phase |
+| `NumPy: baseline optimizations (X86_V2) not supported` | `pip install "numpy<2.0" --force-reinstall` |