Initial commit: Hermes Agent Skills collection

2026-07-12 19:02:59 +00:00
commit e9cc106625
789 changed files with 233126 additions and 0 deletions
@@ -0,0 +1,67 @@
+# AIPerf 3-Scenario Suite Template
+
+Reusable structure for benchmarking N models with Sales + Capacity + Steady-State.
+
+## Directory Layout
+
+```
+aiperf-benchmark-suite/
+├── configs/
+│   └── <model-safe-name>/
+│       ├── sales.yaml
+│       ├── capacity.yaml
+│       └── steady.yaml
+├── templates/
+│   └── base-config.yaml
+├── scripts/
+│   ├── run-suite.sh
+│   └── generate-summary.py
+├── results/          # gitignored
+└── README.md
+```
+
+## Base Config Template
+
+```yaml
+url: "https://<endpoint>/v1"
+endpoint_type: "chat"
+extra-headers:
+  Authorization: "Bearer ${API_KEY}"
+streaming: true
+tokenizer: "builtin"
+random_seed: 42
+```
+
+## Scenario Definitions
+
+### Sales
+- Concurrency: 1, 5, 10 (sequentially)
+- Request count: 50 per level
+- Dataset: `sharegpt`
+- Display: `simple` or `dashboard`
+- Flags: `--concurrency N --request-count 50 --dataset sharegpt`
+
+### Capacity
+- Concurrency: 10, 25, 50, 100 (stepped ramp)
+- Request count: 100 per level
+- Dataset: synthetic fixed lengths
+- Flags: `--concurrency N --request-count 100 --input-tokens-mean 512 --output-tokens-mean 128`
+
+### Steady-State
+- Duration: 600 seconds
+- Concurrency: sweet spot from Capacity (e.g., 20)
+- Dataset: mixed or `sharegpt`
+- Flags: `--concurrency 20 --time 600 --dataset sharegpt`
+
+## Orchestrator Pattern
+
+Iterate models × scenarios, with `--output $RESULTS_DIR/$MODEL/$SCENARIO/conc${CONC}`.
+Pause 30s between major runs to avoid rate limits.
+
+## Result Summary
+
+After all runs, parse `profile_export_aiperf.json` files to generate markdown table:
+
+| Model | Scenario | Concurrency | TTFT avg | TTFT p99 | Throughput |
+
+See `scripts/generate-summary.py` for a reference implementation.