Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,111 @@
+# AIPerf Background-Process & Orchestrator Pitfalls
+
+Session artifact: debugging why a full 12-run benchmark suite silently failed with `aiperf: command not found` despite successful manual runs.
+
+## The Problem: `aiperf: command not found` in Background
+
+The background process (started via `bash -lic`) did not inherit the venv-activated PATH. All 12 runs produced:
+```
+./scripts/run-suite.sh: line 90: aiperf: command not found
+```
+
+**Root cause:** `aiperf` was installed into a Python venv, but the background shell did not activate the venv before running the orchestrator.
+
+## The Fix: Absolute Path to venv Binary
+
+In the orchestrator script, define an absolute path variable:
+```bash
+AIPERF="$BASE_DIR/venv/bin/aiperf"
+if [ ! -x "$AIPERF" ]; then
+    echo "ERROR: AIPerf not found at $AIPERF"
+    exit 1
+fi
+```
+
+Then call `"$AIPERF" profile ...` everywhere instead of plain `aiperf profile`.
+
+## ShareGPT Pre-Download Strategy
+
+`type: "public"` datasets download ShareGPT from HuggingFace on first use (~2–5 minutes). To avoid this delay during production benchmark runs, perform a pre-download:
+
+```bash
+# Pre-download ShareGPT into AIPerf's local cache
+venv/bin/aiperf profile \
+    --config configs/<any-model>/sales.yaml \
+    --concurrency 1 --request-count 1 \
+    --no-gpu-telemetry --no-server-metrics \
+    --artifact-dir /tmp/predownload
+```
+
+Subsequent sales-scenario runs will use the cached dataset instantly.
+
+## AIPerf 0.10.0 CLI Flag Reference
+
+Critical flags used in this session:
+
+| Flag | What it does | Status |
+|------|-------------|--------|
+| `--ui simple` | Minimal UI mode (no TUI) | ✅ Works |
+| `--ui dashboard` | Full TUI dashboard | ✅ Works |
+| `--artifact-dir PATH` | Output directory for CSV/JSON/logs | ✅ Works |
+| `--no-gpu-telemetry` | Skip DCGM/GPU metrics | ✅ Works |
+| `--no-server-metrics` | Skip Prometheus scraping | ✅ Works |
+| `--request-count N` | Total requests (overrides YAML `requests`) | ✅ Works |
+| `--concurrency N` | Concurrent sessions | ✅ Works |
+| `--simple` | ❌ NOT a flag; use `--ui simple` | ❌ Rejected |
+| `--output PATH` | ❌ NOT a flag; use `--artifact-dir` | ❌ Rejected |
+| `--output-artifact-dir PATH` | Alias for `--artifact-dir` | ✅ Works |
+| `--api-key KEY` | Overrides YAML `api_key` | ✅ Works |
+
+## The `api_key` + `requests`/`duration`/`sessions` Interaction
+
+Sequence of errors encountered:
+
+1. `api_key must be a valid string` — this appeared when `phases[].requests` was missing.
+2. With `requests: 50` and `sessions: 1` added to phases → config validated.
+3. **Lesson:** The `api_key` error was misleading. The real issue was always a missing `requests`/`duration`/`sessions` in the phase definition.
+
+Pattern that works:
+```yaml
+phases:
+  - type: "concurrency"
+    name: "profiling"
+    requests: 50          # REQUIRED
+    sessions: 1           # optional, default 1
+```
+
+## Complete Working Config (Single File)
+
+```yaml
+schema_version: "2.0"
+benchmark:
+  models:
+    items:
+      - name: "vllm/gemma-4-31b-it"
+    strategy: "round_robin"
+  endpoint:
+    urls:
+      - "https://ai.noris.de/v1"
+    type: "chat"
+    api_key: "${NORIS_API_KEY}"
+    streaming: true
+    headers:
+      Authorization: "Bearer ${NORIS_API_KEY}"
+  datasets:
+    - type: "synthetic"
+      name: "main"
+      entries: 100
+      isl:
+        type: "fixed"
+        value: 512
+      osl:
+        type: "fixed"
+        value: 128
+  phases:
+    - type: "concurrency"
+      name: "profiling"
+      requests: 50
+      sessions: 1
+  tokenizer:
+    name: "builtin"
+```