Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,190 @@
+# AIPerf 0.10.0 Schema 2.0 Migration Guide
+
+> Session artifact: migrated 12 configs from old flat format to new envelope format. All errors encountered are documented here with fixes.
+
+## The Old Format (DEPRECATED)
+
+```yaml
+url: "https://ai.noris.de/v1"
+endpoint_type: "chat"
+model: "vllm/gemma-4-31b-it"
+extra-headers:
+  Authorization: "Bearer ${API_KEY}"
+streaming: true
+tokenizer: "builtin"
+request_count: 50
+concurrency: 1
+```
+
+**Result with AIPerf 0.10.0:** `1 validation error for AIPerfConfig`
+
+## The New Format (Schema 2.0)
+
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+    strategy: "round_robin"
+  endpoint:
+    urls:
+      - "https://ai.noris.de/v1"
+    type: "chat"
+    api_key: "${API_KEY}"
+    streaming: true
+    headers:
+      Authorization: "Bearer ${API_KEY}"
+  datasets:
+    - type: "synthetic"         # or "public" for ShareGPT
+      name: "main"
+      entries: 100
+      isl:
+        type: "fixed"
+        value: 512
+      osl:
+        type: "fixed"
+        value: 128
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      # ONE OF THESE IS MANDATORY:
+      requests: 50           # total requests
+      # OR duration: 600     # seconds
+      # OR sessions: 1       # concurrent sessions
+  tokenizer:
+    name: "builtin"
+```
+
+## Critical Validation Errors and Fixes
+
+### Error: `Phase 'profiling': at least one of 'requests', 'duration', or 'sessions' must be specified`
+
+**Cause:** Phase definition without termination condition.
+
+**Fix:** Add `requests: N` or `duration: N` or `sessions: N`.
+
+```yaml
+phases:
+  - type: "concurrency"
+    name: "profiling"
+    requests: 50     # <-- REQUIRED
+```
+
+### Error: `api_key must be a valid string`
+
+**Cause:** AIPerf's Pydantic model validates `api_key` eagerly. Even though `--api-key` CLI flag is present, the YAML's `api_key` field still gets parsed.
+
+**Fix:** Either:
+1. Set a dummy value: `api_key: "dummy"` (CLI `--api-key` overrides it)
+2. Use env var syntax: `api_key: "${API_KEY}"` (works if env var is set during config load)
+3. Use `headers` dict instead for auth
+
+**Best practice:** Use both:
+```yaml
+endpoint:
+  api_key: "${API_KEY}"          # satisfies validation
+  headers:
+    Authorization: "Bearer ${API_KEY}"
+```
+
+### Error: `model` not found in schema
+
+**Cause:** Old flat `model` key at top level. New format nests under `benchmark.models.items[].name`.
+
+**Fix:**
+```yaml
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+```
+
+### Error: `extra-headers` not found
+
+**Cause:** Old key name. New format uses `endpoint.headers`.
+
+**Fix:**
+```yaml
+benchmark:
+  endpoint:
+    headers:
+      Authorization: "Bearer ${API_KEY}"
+```
+
+### Error: `endpoint_type` not found
+
+**Cause:** Old key name. New format uses `endpoint.type`.
+
+**Fix:**
+```yaml
+benchmark:
+  endpoint:
+    type: "chat"
+```
+
+### Error: NumPy X86_V2 crash
+
+```
+RuntimeError: NumPy was built with baseline optimizations:
+(X86_V2) but your machine doesn't support: (X86_V2).
+```
+
+**Cause:** AIPerf installs NumPy 2.x, which requires CPU features not present on older x86_64 VMs.
+
+**Fix:**
+```bash
+pip install "numpy<2.0" --force-reinstall
+```
+
+### Error: ShareGPT download timeout
+
+**Cause:** First run with `type: "public"` dataset downloads ShareGPT from HuggingFace (~500MB, 2–5 min).
+
+**Fix:** Wait, or use `type: "synthetic"` for quick validation.
+
+## Synthetic Dataset Format
+
+For fixed ISL/OSL (reproducible capacity tests):
+```yaml
+datasets:
+  - type: "synthetic"
+    name: "main"
+    entries: 100
+    isl:
+      type: "fixed"
+      value: 512
+    osl:
+      type: "fixed"
+      value: 128
+```
+
+Distribution types: `fixed`, `normal`, `lognormal`, `multimodal`, `empirical`.
+
+## CLI Override Compatibility
+
+Flags that work with Schema 2.0 YAML configs:
+- `--concurrency N` → overrides phase concurrency
+- `--request-count N` → overrides phase.requests
+- `--duration N` → overrides phase.duration
+- `--artifact-dir PATH` → sets output directory
+- `--ui simple|dashboard` → UI mode
+- `--api-key KEY` → overrides endpoint.api_key
+- `--no-gpu-telemetry` → skips DCGM
+- `--no-server-metrics` → skips Prometheus scraping
+
+## Orchestrator Pattern: YAML + CLI Sweeps
+
+```bash
+AIPERF="./venv/bin/aiperf"  # always use venv path, never rely on PATH
+
+for conc in 1 5 10; do
+  "$AIPERF" profile \
+    --config sales.yaml \
+    --concurrency "$conc" \
+    --request-count 50 \
+    --artifact-dir "results/sales/conc${conc}"
+done
+```
+
+**Critical:** In background processes (cron, systemd, nohup), `PATH` may not include the venv. Always use absolute path to `venv/bin/aiperf`.