Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,120 @@
+# AIPerf Suite Orchestrator Pattern
+
+Reference: building a 12-run benchmark suite (4 models × 3 scenarios) against ai.noris.de.
+
+## Directory Convention
+
+```
+aiperf-benchmark-suite/
+├── configs/
+│   ├── gemma-4-31b-it/
+│   │   ├── sales.yaml      # ShareGPT dataset, requests: 50, sessions: 1
+│   │   ├── capacity.yaml   # Synthetic 512/128, requests: 100
+│   │   └── steady.yaml     # ShareGPT, duration: 600
+│   ├── gpt-oss-120b/...
+│   └── ... (one dir per model)
+├── scripts/
+│   ├── run-suite.sh        # Orchestrator
+│   ├── generate-summary.py # Markdown table from JSON
+│   └── plot-results.py     # Matplotlib charts + CSV
+├── results/
+│   └── YYYY-MM-DD-HHMMSS/
+│       ├── gemma-4-31b-it/
+│       │   ├── sales/conc1/
+│       │   ├── sales/conc5/
+│       │   ├── sales/conc10/
+│       │   ├── capacity/conc10/
+│       │   ├── capacity/conc25/
+│       │   ├── capacity/conc50/
+│       │   ├── capacity/conc100/
+│       │   └── steady/
+│       └── ...
+└── venv/                   # AIPerf installed here
+```
+
+## Orchestrator Structure
+
+The orchestrator is a bash script that:
+
+1. **Sets absolute paths** — `AIPERF="$BASE_DIR/venv/bin/aiperf"` (never rely on `$PATH`)
+2. **Validates env** — checks `NORIS_API_KEY` and connectivity
+3. **Loops models × scenarios**
+4. **Per scenario, loops concurrency levels** (for count-based scenarios)
+5. **Calls AIPerf with CLI overrides** for each sub-run
+6. **Pauses between runs** (`sleep 5-30`) to avoid rate limits
+7. **Generates summary + plots** after all runs complete
+8. **Continues on error** (`|| true`) — one failed sub-run doesn't kill the suite
+
+## Critical: Venv Path Fix
+
+When running in background (cron, detached shell, Telegram-triggered runs), `$PATH` does NOT include the venv. The orchestrator MUST:
+
+```bash
+AIPERF="$BASE_DIR/venv/bin/aiperf"
+if [ ! -x "$AIPERF" ]; then
+    echo "ERROR: AIPerf not found at $AIPERF"
+    exit 1
+fi
+# Then call "$AIPERF" profile ... everywhere
+```
+
+Failure mode: `aiperf: command not found` on every run, producing empty results silently.
+
+## CLI Override Pattern for Suite Runs
+
+The YAML configs define the base structure. CLI flags sweep parameters:
+
+```bash
+# Sales scenario: sweep concurrencies
+for conc in 1 5 10; do
+    "$AIPERF" profile --config "$CONFIG" \
+        --concurrency "$conc" \
+        --request-count 50 \
+        --artifact-dir "results/sales/conc${conc}"
+done
+
+# Capacity scenario: stepped ramp
+for conc in 10 25 50 100; do
+    "$AIPERF" profile --config "$CONFIG" \
+        --concurrency "$conc" \
+        --request-count 100 \
+        --artifact-dir "results/capacity/conc${conc}"
+done
+
+# Steady scenario: time-based
+"$AIPERF" profile --config "$CONFIG" \
+    --concurrency 20 \
+    --artifact-dir "results/steady"
+```
+
+## Pre-Download Strategy for ShareGPT
+
+First sales run downloads ShareGPT from HuggingFace (~2–5 min). To avoid this during production:
+
+```bash
+# One-time warm-up (any model config with public dataset works)
+venv/bin/aiperf profile --config configs/any-model/sales.yaml \
+    --concurrency 1 --request-count 1 \
+    --artifact-dir /tmp/warmup --no-gpu-telemetry --no-server-metrics
+```
+
+Subsequent runs use `.cache/aiperf/datasets/` instantly.
+
+## Post-Processing Sequence
+
+After the orchestrator finishes all runs:
+
+```bash
+# 1. Markdown summary
+python3 scripts/generate-summary.py results/YYYY-MM-DD-HHMMSS
+
+# 2. Plots + CSV
+python3 scripts/plot-results.py results/YYYY-MM-DD-HHMMSS
+```
+
+Outputs:
+- `results/.../README.md` – human-readable markdown table
+- `results/.../plots/summary.csv` – Excel / Google Sheets import
+- `results/.../plots/capacity_curve.png` – throughput vs concurrency
+- `results/.../plots/latency_curve.png` – TTFT p99 vs concurrency
+- `results/.../plots/sales_comparison.png` – low-concurrency latency