Prefill Throughput Bias from Cold-Start Requests

Problem

AIPerf's per-run profile_export_aiperf.json aggregates ALL requests, including the very first one which has to establish a fresh HTTP connection. This biases the prefill_throughput_per_user metric significantly:

Request	Input Tokens	Prefill tok/s	TTFT	Connection
0	7	4 tok/s	1699 ms	DNS (1002ms) + TCP (1144ms)
1	371	2003 tok/s	185 ms	Reused ✓
2	676	3596 tok/s	188 ms	Reused ✓
5	1488	7571 tok/s	197 ms	Reused ✓

The first request's DNS lookup + TCP handshake (2+ seconds) is counted toward "prefill time", deflating the metric. The true prefill throughput (requests 1+) is ~2K–7.5K tok/s, not the ~4K aggregate.

Impact

Prefill metrics: Inflated by ~1.5–2× for short runs (<20 requests). Improves as sample size grows.
TTFT p99: May be driven by the warm-up request's elevated TTFT.
Low-concurrency runs (conc=1, single session) are most affected because only ONE session exists and its first request is always cold.

Mitigation

Option A: Discard First Request (Recommended for Prescriptive Benchmarks)

Parse profile_export.jsonl (line-delimited JSON, one request per line) and exclude session_num=0 / turn_index=0:

import json

metrics = []
with open("profile_export.jsonl") as f:
    for line in f:
        req = json.loads(line)
        # Skip first turn of first session (cold-start)
        if req["metadata"]["turn_index"] == 0 and req["metadata"]["session_num"] == 0:
            continue
        m = req["metrics"]
        metrics.append({
            "ttft": m["time_to_first_token"]["value"],
            "prefill": m["prefill_throughput_per_user"]["value"],
        })

# Now metrics[] excludes the warm-up request

Option B: Use Connection Warm-Up Before Benchmarking

Send 1–2 throwaway requests to establish the TCP + TLS + HTTP/2 connection pool before the measured phase. AIPerf does not have a built-in warm-up phase, so this must be done externally:

# Pre-warm the connection
venv/bin/aiperf profile --config warmup.yaml \
    --concurrency 1 --request-count 1 \
    --artifact-dir /tmp/warmup

# Now run the real benchmark — connections are hot
venv/bin/aiperf profile --config benchmark.yaml \
    --concurrency 1 --request-count 50 \
    --artifact-dir results/benchmark

The warmup run should use the same endpoint, model, and streaming=true as the benchmark to ensure the connection pool is pre-established.

Option C: Accept Bias for Comparative Benchmarks

If comparing N models on the same infra with the same settings, the cold-start bias applies equally to all. The absolute numbers are slightly off, but rankings and relative ratios remain valid. Use this only when:

Absolute SLA numbers are not the goal
All runs use identical connection topology (same client → same endpoint)
Sample size is large enough that one cold request is <5% of total

When It Matters Most

Scenario	Severity	Why
conc=1, sessions=1	🔴 High	First request is 100% of traffic
conc=5, sessions=5	🟡 Medium	1 of 5 sessions starts cold
conc=50, sessions=50	🟢 Low	Many sessions, cold-start amortized
Capacity (10→100+)	🟢 Low	Many sessions, cold-start amortized
Steady-state (10min)	🟢 Low	Thousands of requests, negligible

Per-Request Data Location

Raw per-request data is in profile_export.jsonl (NOT profile_export_aiperf.json which is aggregated). Each line is a JSON object with:

{
  "metadata": {
    "session_num": 0,
    "turn_index": 0,
    "worker_id": "worker_766ae0c2",
    "conversation_id": "...",
    "was_cancelled": false
  },
  "metrics": {
    "time_to_first_token": {"value": 1699.2, "unit": "ms"},
    "prefill_throughput_per_user": {"value": 4.1, "unit": "tokens/sec/user"},
    "request_latency": {"value": 26372.4, "unit": "ms"},
    "http_req_connection_reused": {"value": 0, "unit": "ratio"}
  }
}

Key indicators of a cold request:

"http_req_connection_reused": {"value": 0} — fresh connection
"http_req_dns_lookup" > 500ms — DNS resolution happened
"http_req_connecting" > 500ms — TCP/TLS handshake happened

A reused connection has "http_req_connection_reused": {"value": 1} and near-zero DNS/connect times.

4.4 KiB Raw Blame History Unescape Escape