vault

Benchmarking LLM Inference

32 min read

Benchmarking LLM Inference

Two teams benchmarking the same server with two popular tools will report different inter-token latencies, and neither will be wrong — they are measuring different things under the same name. This is not sloppiness; it is a genuine definitional fork, and it is verifiable from source: LLMPerf’s inter_token_latency_s includes time to first token (its client appends TTFT as the first element of time_to_next_token, then divides the sum by the output-token count), while GenAI-Perf’s inter-token latency is explicitly the time between intermediate responses divided by the tokens in the later response — excluding the first token and normalising for multi-token chunks. Layered on top of that is a second fork: vLLM’s benchmark client records one latency sample per streamed chunk, not per token, so a server configured to bundle tokens per chunk changes the reported number without changing the model’s speed. And beneath both sits the fact that dominates every result — the input and output length distribution of the dataset. This note covers MLPerf Inference as the standardised answer, the practical harnesses as the everyday answer, and the specific traps that make published numbers incomparable.

The definitions of the metrics themselves are in Time to First Token and Inter-Token Latency; the SLO-conditioned capacity number a benchmark sweep produces is in Goodput for LLM Serving; the live production counterpart is Observability for LLM Serving. This note is about methodology — how to run the experiment so the number means something.

Version and tag pins (as of 2026-08-08)

MLPerf Inference rules were read from mlcommons/inference_policies, branch master, file inference_rules.adoc. The benchmark list was read from mlcommons/inference, branch master, README.md, which documents v6.1 (submission deadline 31 July 2026) as the in-flight round and v6.0 (submission deadline 13 February 2026) as the round before it. Probing the results repositories, mlcommons/inference_results_v5.0, _v5.1 and _v6.0 all resolve while _v6.1 returns HTTP 404 — so v6.0 is the most recent round with published results, and v6.1 results were not yet posted at the time of writing. vLLM harness code is from tag v0.26.0. LLMPerf, GenAI-Perf, AIPerf, inference-perf and SGLang code are from their main branches, fetched 2026-08-08.

Mental Model: A Benchmark Answers Exactly One of Three Questions

The most common benchmarking mistake is not a bad measurement — it is running one experiment and reading three answers out of it. There are three distinct questions, they require different configurations, and a setup that answers one actively misreports the others.

flowchart TD
    Q{"What do you<br/>actually want to know?"}
    Q -->|"How much work can<br/>this hardware do?"| OFF["OFFLINE / saturation test"]
    Q -->|"How many users can I serve<br/>at an acceptable latency?"| SRV["SERVER / open-loop sweep"]
    Q -->|"How fast is one request<br/>on an idle machine?"| SS["SINGLE STREAM / floor test"]

    OFF --> OFFC["All requests at t=0<br/>batch is always full<br/>no arrival process"]
    SRV --> SRVC["Poisson arrivals at rate R<br/>sweep R, check p99 constraints<br/>report max R that passes"]
    SS --> SSC["One request at a time<br/>no queueing, no batching<br/>gives the SLO denominator"]

    OFFC --> OFFM(["Metric: tokens/sec<br/>Use: cost per million tokens,<br/>batch jobs"])
    SRVC --> SRVM(["Metric: max sustained QPS<br/>Use: capacity planning,<br/>the honest serving test"])
    SSC --> SSM(["Metric: unloaded TTFT / ITL<br/>Use: reference for<br/>slowdown-relative SLOs"])

    OFFM -.->|"reading this as<br/>serving capacity<br/>OVERSTATES by 2-10x"| SRVM
    SSM -.->|"reading this as<br/>production latency<br/>is the classic demo lie"| SRVM

The three benchmark questions and the configuration each requires. What it shows: the offline and single-stream configurations are not “the serving benchmark with a knob turned” — they suppress the arrival process and the queueing that the serving question is entirely about. The insight to take: the two dotted arrows are the errors you will actually encounter in vendor material and internal slide decks. An offline throughput number quoted as serving capacity ignores that every request in an offline run waits arbitrarily long, and a single-stream latency number quoted as production latency was measured on a machine with nobody else on it. MLPerf keeps these as separate, separately-reported scenarios precisely so they cannot be conflated.

MLPerf Inference: The Standardised Answer

MLPerf Inference, from MLCommons, is the only benchmark in this space with an enforced rulebook, a mandatory load generator, an accuracy gate, and an audit process. Its design is described in the original benchmark paper (Reddi et al. 2019), and the operative rules live in inference_rules.adoc. Three properties make it worth studying even if you never submit.

A single load generator is mandatory. The rules state the LoadGen “is provided in C++ with Python bindings and must be used by all submissions”, and is responsible for generating queries per scenario, tracking latency, validating accuracy, and computing final metrics. Removing the harness as a variable is the single largest source of comparability, and it is exactly the variable that makes internal benchmarks incomparable.

Latency is defined against the schedule, not the send. The rule reads: “Latency is defined as the time from when the LoadGen was scheduled to pass a query to the SUT, to the time it receives a reply.” That word “scheduled” is load-bearing. If the client itself falls behind — thread starvation, GIL contention, a slow tokenizer in the request path — the delay it caused still counts against the system under test. A naive harness that timestamps at send() hides its own backpressure and reports a system as faster than it is.

Accuracy is a gate, not a reported metric. Every LLM benchmark in the Closed division carries a quality target: Llama3.1-8B must reach 99% (or 99.9%) of the FP32 reference ROUGE scores and produce a total generation length above 90% of the reference; Llama2-70b likewise on OpenOrca with tokens-per-sample ≥ 90% of a 294.45 reference; DeepSeek-R1 must hit 99% of an 81.9132% exact-match reference. This is what stops the obvious cheat. Without a length floor, a system can “win” by generating shorter answers — fewer decode steps, higher tokens/second, worse product.

The v6.1 LLM benchmark set and its latency tiers

The datacenter LLM benchmarks, with the Server-scenario constraints read directly from the rules table:

BenchmarkTaskDatasetConversational / Server TTFT · TPOTInteractive TTFT · TPOT
Llama3.1-8BSummarizationCNN/DailyMail v3.0.0, max_seq_len=20482000 ms · 100 ms500 ms · 30 ms
Llama2-70bQuestion answeringOpenOrca, max_seq_len=10242000 ms · 200 ms450 ms · 40 ms
DeepSeek-R1Reasoningmlperf_deepseek_r12000 ms · 80 ms1500 ms · 15 ms
GPT-OSS-120BText gen (QA, math, code)AIME25, GPQA Diamond, LiveCodeBench v63000 ms · 80 ms2000 ms · 20 ms
Qwen3-VL-235B-A22BVision-languageShopify Product Catalogue12 s (single constraint)—
WhisperSpeech to textLibriSpeechN/AN/A
Llama3.1-405B (v6.0; dropped in v6.1)Long-context text genLongBench / Ruler / GovReport subset6000 ms · 175 ms4500 ms · 80 ms
Mixtral-8x7B (v6.0; dropped in v6.1)Mixed QA/math/codeOpenOrca + GSM8K + MBXP2000 ms · 200 ms—

MLPerf Inference LLM benchmarks and their per-benchmark latency constraints, from inference_rules.adoc on master. What it shows: MLPerf does not have “a latency target” — it has a per-model, per-category pair of targets, and the same model is benchmarked twice under two different strictness tiers. The insight to take: the Conversational→Interactive gap is where all the engineering difficulty lives. Llama2-70b’s TPOT budget tightens from 200 ms to 40 ms, a 5× constraint, and a system’s throughput under the Interactive tier is typically a small fraction of its Conversational throughput — because tighter TPOT forces smaller decode batches. That trade-off is the whole subject of Goodput for LLM Serving, made concrete and adversarial. The mlcommons/inference README notes explicitly that “pointpainting, llama3.1-405b and mixtral-8x7b are not part of the v6.1 model list”, which is why cross-round comparisons of an “MLPerf LLM score” are meaningless — the benchmark set itself churns.

Scenarios, and why Server is the honest one

MLPerf defines four scenarios. Their query-generation rules, taken verbatim from the rules table, differ in exactly the way that matters:

ScenarioQuery generationDurationTail percentilePerformance metric
Single streamNext query sent as soon as the SUT completes the previous600 s90%*90th-percentile early-stopping latency estimate
Server / InteractiveNew queries per a Poisson distribution600 s99%*Maximum Poisson throughput parameter supported
OfflineAll samples sent at start in a single query≥1 query and 600 sN/AMeasured throughput
MultistreamNext query as soon as the previous completes, 8 samples/query600 s99%*99th-percentile early-stopping latency estimate

The Server scenario is the honest serving test for three compounding reasons.

First, the arrival process is exogenous. Poisson arrivals mean the load generator does not wait for the system; requests pile up if the system is slow, and that queueing shows up inside the measured latency. Single-stream and multistream are closed loops — the client’s send rate is throttled by the server’s completion rate, which is a self-limiting feedback loop that structurally cannot produce a queue. Offline has no arrival process at all.

Second, the metric is a constrained maximum, not an unconstrained observation. The rules describe the search: “LoadGen determines the system throughput using multiple test runs. Each test run evaluates a specific throughput value in queries-per-second (QPS)… LoadGen will use a binary search to find a candidate value. If a run fails, it will reduce the value by a small delta then try again.” The reported number is the largest arrival rate at which the 99th-percentile constraints still hold. That is the definition of goodput implemented as a harness.

Third, the constraint is at p99, not the mean. Averages are nearly free to satisfy; a 99th-percentile constraint on both TTFT and TPOT simultaneously is not. The rules also publish the query-count arithmetic behind that choice — to bound a 99th-percentile estimate to a 0.05% margin of error at 99% confidence takes 262,742 inferences, rounded to 33·2¹³ = 270,336; a 90th percentile needs only 24,576. An “early stopping” criterion permits shorter runs at the cost of a slightly inflated effective percentile, an explicit penalty for the higher variance of small samples.

sequenceDiagram
    autonumber
    participant LG as LoadGen
    participant SUT as System Under Test

    rect rgba(120,160,200,0.15)
    Note over LG,SUT: OFFLINE — one query, all samples
    LG->>SUT: all N samples at t=0
    SUT-->>LG: results (whenever)
    Note over LG: metric = N / elapsed<br/>no latency constraint at all
    end

    rect rgba(200,160,120,0.15)
    Note over LG,SUT: SERVER — Poisson arrivals, binary search on R
    loop trial at rate R
        LG->>SUT: query (interarrival ~ Exp(1/R))
        LG->>SUT: query
        LG->>SUT: query
        SUT-->>LG: replies
        Note over LG: check p99 TTFT ≤ target<br/>AND p99 TPOT ≤ target
    end
    alt constraints hold
        Note over LG: raise R, retry
    else constraints violated
        Note over LG: lower R by delta, retry
    end
    Note over LG: report max passing R
    end

The Offline and Server scenarios as the load generator actually drives them. What it shows: Offline is one shot with no feedback; Server is an outer binary search wrapped around repeated timed trials, each of which is pass/fail against two simultaneous percentile constraints. The insight to take: the Server number is expensive — many full 600-second trials — and that cost is the reason internal benchmarks skip it and report a single-load-point measurement instead. A single load point is SLO attainment, not capacity; the distinction is developed in Goodput for LLM Serving.

The Closed division additionally pins the generation parameters. The rules mandate greedy search for Llama2-70b, Llama3.1-405B, Mixtral-8x7B, DeepSeek-R1 and Llama3.1-8B, while GPT-OSS-120B and Qwen3-VL use sampling with temperature=1.0, top_p=1.0, top_k disabled; min_new_tokens and max_new_tokens are fixed per benchmark (Llama3.1-8B: 1 and 128; Llama2-70b: max 1024; DeepSeek-R1: 1 and 20,000). Speculative decoding is permitted only for a named subset of workloads and only under the constraints of a dedicated appendix, with specific draft heads and settings enumerated. And a rule exists specifically to forbid the most tempting cheat: “Techniques that only improve performance by artificially creating batches of identical samples. Sorting samples beyond the boundary of a dataset is prohibited for all (L|V)LM workloads.” Sorting by length would let a system pack perfectly uniform batches — enormously faster, and nothing like real traffic.

Resolved 2026-08-15

v6.1 results are still unpublished, and inference_policies genuinely has no versioned rules artifact — the rolling-branch caveat is a property of the repository, not a gap in this research. Three checks, all re-run 2026-08-15. (1) Results. mlcommons/inference_results_v6.1 returns HTTP 404 on both main and master, while _v5.1 and _v6.0 return 200 on both. There is no MLCommons announcement post for v6.1 results; the most recent v6.1 item on the site is a July 2026 call for submissions to the new edge-agentic benchmark (MLCommons). The latest round with published results remains v6.0, announced 2026-04-01 with 24 submitting organizations (MLCommons). Since v6.0’s deadline was 13 February 2026 and its results landed 1 April, the ~7-week lag puts v6.1 results — deadline 31 July 2026 — around September 2026; they were not out at the time of writing, two weeks past the deadline. (2) Rules versioning. mlcommons/inference_policies was probed for a round-specific ref: master returns 200 for inference_rules.adoc, while main, v6.0, v6.1 and r6.1 all return 404. MLCommons does not branch or tag its rules per round — the only citable state is “master as of a date,” which is exactly how this note cites it. (3) Site corroboration. mlcommons.org/benchmarks/inference-datacenter/ still renders with “V3.1” in its page title on re-fetch, so the results site remains unable to corroborate the current round. The round identification in this note — v6.1 in flight, v6.0 the newest published — therefore rests on the mlcommons/inference README’s dated deadlines plus the results-repository probe, and both were independently re-confirmed.

The Practical Harnesses

Nobody runs MLPerf to tune a config on a Tuesday. Four tools cover the everyday case, and they differ in ways that matter more than their feature lists suggest.

ToolOriginLoad modelDistinctive strengthDistinctive trap
vllm bench servevLLM, in-treeOpen loop: Poisson via --request-rate, shaped by --burstiness, ramps via --ramp-up-strategyBuilt-in --goodput; huge dataset registry; deep vLLM integrationITL sampled per streamed chunk, not per token
GenAI-PerfNVIDIA (Triton perf_analyzer)Concurrency or request-rateCleanest metric definitions; Time to Second Token; auto-collects GPU telemetry from a DCGM endpointDefinitions differ from vLLM’s — numbers are not interchangeable
AIPerfNVIDIA (ai-dynamo/aiperf)Successor generation to GenAI-PerfBuilt for higher client-side scaleNewer; check its metric definitions independently
LLMPerfRay projectClosed loop: num_concurrent_requests threadsSimple, widely quoted in vendor comparisonsinter_token_latency_s includes TTFT; closed loop suppresses queueing
inference-perfKubernetes SIG (kubernetes-sigs)Constant rate, Poisson, or concurrent users; multi-stage sweepsAutomatic saturation detection; engine-agnostic (vLLM, SGLang, TGI); pulls server-side metrics tooYounger project; fewer published baselines

The four practical harnesses. What it shows: they disagree on the load model (open vs closed loop), on the metric definitions, and on whether server-side telemetry is collected alongside client-side timing. The insight to take: inference-perf is the one to know about if you are on Kubernetes, because its README states it “was founded as a part of the inference benchmarking and metrics standardization effort in wg-serving” — the same working group whose model-server metric standardisation the Gateway API Inference Extension implements. It is the harness designed to produce numbers that compare across engines, which is precisely the property the other three lack.

vLLM’s client, read from source

vllm bench serve (the packaged form of the historical benchmark_serving.py) generates load with a configurable arrival process. --request-rate sets the mean rate; the flag’s documentation notes that inf means “all the requests are sent at time 0”, which converts the run into a burst test. --burstiness reshapes the process: its help text says “Default value is 1, which follows Poisson process. Otherwise, the request intervals follow a gamma distribution. A lower burstiness value (0 < burstiness < 1) results in more bursty requests.” And --ramp-up-strategy {linear,exponential} with --ramp-up-start-rps / --ramp-up-end-rps sweeps the rate within one run — a cheap approximation of the binary search MLPerf performs across runs.

Its per-request time-per-output-token is computed honestly, from real token counts reported in the response’s usage field:

output_len = outputs[i].output_tokens        # from the usage field, real tokens
tpot = 0.0
if output_len > 1:
    latency_minus_ttft = outputs[i].latency - outputs[i].ttft
    tpot = latency_minus_ttft / (output_len - 1)   # excludes the first token
    tpots.append(tpot)
# Note: if output_len <= 1, we regard tpot as 0 for goodput
all_tpots.append(tpot)

(output_len - 1) correctly excludes the prefill-produced first token. The trailing comment documents a real edge case — a single-token response scores TPOT 0 and therefore always satisfies a TPOT goodput constraint — which matters for classification and routing workloads and is discussed in Goodput for LLM Serving.

Where the Definitions Fork

Everything above concerns how load is applied. This section concerns what is recorded, and it is where published numbers quietly stop being comparable.

Fork 1: does inter-token latency include the first token?

GenAI-Perf’s metric table defines inter-token latency as “Time between intermediate responses for a single request divided by the number of generated tokens of the latter response, one value per response per request”, and separately defines Output Token Throughput Per User as “Total number of output tokens (excluding the first token) divided by the total duration of the generation phase of each request”. Both explicitly exclude the prefill-produced token.

LLMPerf does the opposite, and the code makes it unambiguous. In its OpenAI chat client, on the first content-bearing chunk:

if not ttft:
    ttft = time.monotonic() - start_time
    time_to_next_token.append(ttft)          # <-- TTFT pushed into the ITL list
else:
    time_to_next_token.append(time.monotonic() - most_recent_received_token_time)
...
metrics[common_metrics.INTER_TOKEN_LAT] = sum(time_to_next_token)
# comment in source: "This should be same as metrics[common_metrics.E2E_LAT]. Leave it here for now"

and then in the benchmark driver:

request_metrics[common_metrics.INTER_TOKEN_LAT] /= request_metrics[common_metrics.NUM_OUTPUT_TOKENS]

So LLMPerf’s inter_token_latency_s is end-to-end latency divided by output-token count — the source’s own comment concedes the sum equals E2E latency. For a request with a 2-second TTFT and 200 output tokens at a genuine 20 ms per token, GenAI-Perf reports ≈20 ms and LLMPerf reports ≈30 ms. Neither is a bug; they are different statistics with the same name, and the 50% gap is entirely the amortised prefill.

Worse, LLMPerf’s two code paths divide by different denominators. The main loop divides by request_metrics[NUM_OUTPUT_TOKENS], which at that moment still holds tokens_received — the count of SSE chunks — because it is overwritten with the re-tokenized length on the very next line. The final drain path divides by num_output_tokens, the re-tokenized count. If chunks and tokens are not 1:1, those two paths disagree with each other within a single run.

flowchart LR
    subgraph REQ["One request: TTFT = 2000 ms, then 200 tokens at 20 ms"]
        T0["first token<br/>at t = 2000 ms"]
        TN["tokens 2..200<br/>19 × 20 ms ... 199 × 20 ms"]
    end
    REQ --> GP["GenAI-Perf ITL<br/>mean over INTERMEDIATE gaps<br/>÷ tokens in each response"]
    REQ --> LP["LLMPerf inter_token_latency_s<br/>(TTFT + all gaps) ÷ output tokens"]
    REQ --> VL["vLLM tpot<br/>(latency − ttft) ÷ (output_len − 1)"]
    REQ --> VI["vLLM itl[]<br/>one sample PER SSE CHUNK"]
    GP --> R1["≈ 20 ms"]
    LP --> R2["≈ 30 ms<br/>(2000 + 3980) / 200"]
    VL --> R3["≈ 20 ms"]
    VI --> R4["20 ms if 1 token/chunk<br/>160 ms if 8 tokens/chunk"]

The same request scored by four definitions. What it shows: GenAI-Perf and vLLM’s TPOT agree; LLMPerf is 50% higher on this example purely because it amortises TTFT; vLLM’s raw ITL array depends on the server’s streaming granularity. The insight to take: the divergence scales with the TTFT-to-decode-time ratio, so it is worst exactly where it matters most — long prompts with short answers, the retrieval-augmented-generation shape. A RAG workload with an 8,000-token prompt and a 50-token answer will show LLMPerf ITL several times GenAI-Perf ITL for an identical server.

Fork 2: a streamed chunk is not a token

vLLM’s benchmark client, reading vllm/benchmarks/lib/endpoint_request_func.py at v0.26.0, appends exactly one interval per Server-Sent Events data frame:

timestamp = time.perf_counter()
data = json.loads(chunk)
if choices := data.get("choices"):
    content = choices[0]["delta"].get("content")
    if ttft == 0.0:                       # first content-bearing chunk
        ttft = timestamp - st
        output.ttft = ttft
    else:                                 # Decoding phase
        output.itl.append(timestamp - most_recent_timestamp)
    generated_text += content or ""
most_recent_timestamp = timestamp

There is no division by the number of tokens carried in content. If the server emits one token per frame, itl is a per-token interval array. If the server buffers several tokens per frame, each itl entry covers all of them and the array is correspondingly shorter.

Servers do buffer, and both major open-source engines expose the knob under the same name. vLLM has --stream-interval (SchedulerConfig.stream_interval, default 1) at v0.26.0 — see the resolved callout below. SGLang exposes exactly this control too; reading python/sglang/srt/server_args.py on main, the parameter is documented as “The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher”, with a default of 1. Raising it is a legitimate throughput optimisation — fewer HTTP frames, less per-token serialisation and event-loop work — and it genuinely speeds the server up.

The measurement consequence is subtle and runs in two directions at once. With k tokens per frame, each of vLLM’s itl samples is roughly k× a single-token interval, so mean ITL inflates — but there are k× fewer samples, and any stall that happens inside a bundle is averaged with the k−1 fast tokens beside it, so the p99 over ITL samples loses its ability to see short stalls. Meanwhile vLLM’s tpot, computed from the usage token count rather than from frames, tracks the real per-token rate and improves, because the server really did get faster. A tool that reports TPOT and a tool that reports chunk-interval ITL will therefore move in opposite directions in response to the same configuration change. GenAI-Perf is immune to the first half of this by construction — its definition divides by “the number of generated tokens of the latter response” — but no client-side tool can recover a stall that the server hid inside a bundle.

Resolved 2026-08-15 (second half); tail shape still unmeasured

vLLM has the same control, and it is called the same thing. SchedulerConfig.stream_interval exists at v0.26.0 with Field(default=1, ge=1) and the docstring “The interval (or buffer size) for streaming in terms of token length. A smaller value (1) makes streaming smoother by sending each token immediately, while a larger value (e.g., 10) reduces host overhead and may increase throughput by batching multiple tokens before sending” (vllm/config/scheduler.py). It is exposed on the CLI as --stream-interval (vllm/engine/arg_utils.py, line 1489) and enforced in vllm/v1/engine/output_processor.py, which suppresses an output unless the request “has finished, or … is the first token, or … has reached the stream interval number of tokens.” Two consequences follow from that exact condition. TTFT is untouched — the first token always flushes immediately, by an explicit special case. And the flag is server-side and engine-wide, not a per-request OpenAI parameter, so a client cannot detect or opt out of it.

vLLM’s own benchmark code confirms the asymmetry this note describes, in a comment: the harness counts output tokens with the tokenizer “instead of looking at len(outputs[i].itl) since multiple output tokens may be bundled together” (vllm/benchmarks/serve.py). TPOT is then computed as (latency − ttft) / (output_len − 1) against that true token count, while itls += outputs[i].itl concatenates one raw frame-gap per SSE frame. TPOT is structurally immune to bundling; ITL is structurally exposed to it. That is not an empirical finding to be measured — it is a property of the two formulas.

The quantitative shape also follows without an experiment, and it is not a simple k×. With stream_interval = k, each reported ITL sample is the sum of k consecutive per-token gaps. Summing concentrates: if single-token gaps have mean μ and standard deviation σ, a k-token bundle has mean kμ and standard deviation σ√k. So mean ITL inflates by exactly k×, but the p99 inflates by only about (kμ + 2.33σ√k) / (μ + 2.33σ) — strictly less than k× whenever σ > 0, and the p99-to-mean ratio falls toward 1 as k grows. Concretely: a distribution with μ = 10 ms, σ = 20 ms has p99 ≈ 57 ms unbundled; at k = 8 the mean becomes 80 ms and the p99 about 212 ms — a 3.7× rise in p99 against an 8× rise in the mean. Bundling therefore makes a server’s ITL tail look relatively healthier while making its absolute ITL look worse, which is the worst possible combination for anyone comparing two configurations.

What is still not measured is the residual: real per-token gaps are not independent (a preemption or a long prefill chunk stalls several consecutive tokens together), so the true √k concentration is weaker than the independent case above and the real p99 inflation sits somewhere between the √k bound and k×. Settling that needs the experiment the original flag proposed — vllm bench serve against the same engine at --stream-interval 1 and 8, comparing mean ITL, p99 ITL, sample count and TPOT — which was not run here. uncertain

Fork 3: the arrival process

This one is covered in depth by the sibling notes and only summarised here: a closed-loop harness issues request n+1 only after n completes, so under load it self-throttles and the queue it would have created never forms. LLMPerf is closed-loop by construction — it spawns num_concurrent_requests threads that each loop — while vllm bench serve with a finite --request-rate, GenAI-Perf in request-rate mode, inference-perf in Poisson mode, and MLPerf’s Server scenario are all open-loop. Closed-loop numbers are not wrong, they answer a different question (“what does this system do at fixed concurrency c?”), and that question is legitimate. They simply cannot be compared to open-loop numbers, and they systematically overstate goodput. See Utilization Targets and the Latency Knee for why the queueing term they suppress is the one that explodes.

The Dataset Dominates Every Result

If you change nothing but the dataset, every number in the report changes — often by more than any optimisation you are trying to measure. This is because prefill cost scales with prompt length, decode cost scales with output length, and the two have opposite hardware characteristics (Prefill and Decode as Two Different Workloads).

vLLM’s harness ships a large dataset registry — ShareGPTDataset, RandomDataset, RandomMultiModalDataset, CustomDataset, TimedTrace, plus HuggingFace-backed loaders for VisionArena, MMVU, InstructCoder, MTBench, HumanEval, GSM8K, AIMO, ASR, MMStar, BFCL and an MLPerfDataset. Two of them account for the overwhelming majority of published numbers, and both carry defaults that are easy to miss.

ShareGPT, and the pruning nobody mentions

ShareGPTDataset reads a JSON conversation dump, keeps entries with at least two turns, shuffles under random_seed, and takes the first turn as prompt and the second as the reference completion — tokenizing both to get real lengths. It then filters each candidate through is_valid_sequence, whose defaults are:

def is_valid_sequence(
    prompt_len: int, output_len: int,
    min_len: int = 4,
    max_prompt_len: int = 1024,      # <-- prompts over 1024 tokens are DROPPED
    max_total_len: int = 2048,       # <-- prompt + output over 2048 are DROPPED
    skip_min_output_len_check: bool = False,
) -> bool:

with a docstring noting these criteria are “copied from the original sample_hf_requests and sample_sharegpt_requests functions in benchmark_serving.py”. So a default ShareGPT run contains no long-context requests at all — nothing over 1,024 prompt tokens, nothing whose total exceeds 2,048. That is a defensible choice for reproducibility on small-context models, and a serious problem for anyone benchmarking a long-context deployment: the KV cache never fills, prefill never dominates, chunked prefill never engages, and preemption never happens. The benchmark exercises a regime your production traffic may never visit.

Random, and the constants that make it unrepresentative

RandomDataset defaults, read from the class body, are DEFAULT_PREFIX_LEN = 0, DEFAULT_RANGE_RATIO = 0.0, DEFAULT_INPUT_LEN = 1024, DEFAULT_OUTPUT_LEN = 128. Each of the three matters:

  • range_ratio = 0.0 means zero variance — every request has an identical input and output length. Uniform lengths make the scheduler’s job trivial: batches pack perfectly, no request is stuck behind a much longer one, and head-of-line blocking cannot occur. Real traffic has a heavy-tailed length distribution, and it is that tail that produces the p99.
  • prefix_len = 0 means no shared prefix, so automatic prefix caching gets a 0% hit rate. Since enable_prefix_caching defaults to True in vLLM v0.26.0, a synthetic-random benchmark measures a feature-complete server with one of its most valuable features contributing nothing. Conversely, a benchmark with a large --random-prefix-len measures mostly the cache. Neither is your production hit rate. See Automatic Prefix Caching.
  • output_len = 128 fixes decode work per request, which fixes the decode/prefill ratio, which fixes which phase your bottleneck is in.
flowchart TD
    D{"Dataset choice"}
    D --> SG["ShareGPT (default filters)"]
    D --> RD["Random, range_ratio=0"]
    D --> RP["Random + large prefix_len"]
    D --> TR["Trace replay / timed_trace"]

    SG --> SG1["realistic length MIX<br/>but capped at 1024 / 2048 tokens"]
    SG1 --> SG2["never tests long context,<br/>KV pressure, or preemption"]

    RD --> RD1["identical lengths<br/>perfect batch packing"]
    RD1 --> RD2["p99 ≈ p50<br/>flatters the scheduler"]
    RD --> RD3["prefix_len = 0<br/>→ 0% prefix cache hits"]
    RD3 --> RD4["understates a feature<br/>that is ON by default"]

    RP --> RP1["high cache hit rate<br/>prefill nearly free"]
    RP1 --> RP2["overstates throughput"]

    TR --> TR1["real lengths AND<br/>real arrival timing"]
    TR1 --> TR2["closest to truth;<br/>hardest to obtain"]

How each dataset choice biases the result, and in which direction. What it shows: every synthetic option distorts something, and the distortions point in different directions — uniform-random flatters the scheduler and penalises the cache, while large-prefix-random does the reverse. The insight to take: always report the input and output length distributions alongside the latency numbers, not just the dataset name. “ShareGPT” without the filter parameters is not a reproducible specification, and two teams both saying “we benchmarked on ShareGPT” may have measured workloads with no overlap in prompt length.

The papers in this area know it. DistServe, Splitwise and Sarathi-Serve all report length distributions per workload and construct arrival processes explicitly, because the datasets carry no timestamps — Splitwise’s headline finding that its coding trace and conversation trace behave completely differently, with the conversation trace showing better TTFT and inter-token latency but 5.8× the end-to-end latency, is a statement about output-length distribution and nothing else (Patel et al. 2023; Zhong et al. 2024; Agrawal et al. 2024).

A Protocol That Produces Trustworthy Numbers

Pulling the above together into an actual procedure.

1. Fix the question first. Offline throughput, Server capacity, or unloaded latency — pick one per experiment. If you need all three, run three experiments.

2. Measure the unloaded floor. Single-stream TTFT and ITL on an idle server give the denominator for slowdown-relative and hardware-relative SLOs, and they tell you whether an absolute SLO is even achievable on this model and hardware before you start tuning.

3. Warm up, and discard the warm-up. The first requests hit an empty prefix cache, uncaptured CUDA graphs, and cold kernel autotuning. vllm bench serve provides --num-warmups for exactly this.

4. Sweep the rate, do not pick one. A single load point yields SLO attainment; capacity requires the sweep. Use --ramp-up-strategy for a cheap first pass and discrete fixed-rate runs for the number you will publish.

5. Fix and publish the workload specification. Dataset, the filter parameters, range_ratio, prefix_len, input and output length percentiles, seed, arrival distribution and --burstiness, streaming on/off, and ignore_eos.

6. Collect server-side metrics in the same window. The client cannot see queue time, KV cache utilisation or preemption. inference-perf does this natively; otherwise scrape /metrics around the run. A throughput result obtained while vllm:num_preemptions was climbing is a measurement of a thrashing server (see Observability for LLM Serving and Preemption and Recomputation in LLM Serving).

7. Report percentiles, per metric, with the definition named. p99 for ITL, p95–p99 for TTFT, p50 only as a regression tripwire — and state which tool and which definition produced them.

8. Repeat. Poisson arrivals are stochastic; a single 60-second run at a rate near saturation has enormous variance. MLPerf’s 600-second minimum and its query-count table exist for this reason.

# Step 2 — unloaded floor: one request at a time
vllm bench serve --model $MODEL --dataset-name sharegpt --dataset-path $SHAREGPT \
  --num-prompts 200 --max-concurrency 1 --percentile-metrics ttft,tpot,itl,e2el
 
# Step 4 — coarse ramp to locate the knee cheaply
vllm bench serve --model $MODEL --dataset-name random \
  --random-input-len 2048 --random-output-len 256 --random-range-ratio 0.3 \
  --random-prefix-len 512 \
  --ramp-up-strategy linear --ramp-up-start-rps 1 --ramp-up-end-rps 20 \
  --num-prompts 5000 --num-warmups 100
 
# Step 4/7 — the number you publish: fixed rate, enough samples for p99
vllm bench serve --model $MODEL --dataset-name random \
  --random-input-len 2048 --random-output-len 256 --random-range-ratio 0.3 \
  --random-prefix-len 512 --seed 0 \
  --request-rate 9 --burstiness 1.0 --num-prompts 20000 --num-warmups 200 \
  --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,95,99 \
  --goodput ttft:500 tpot:30 e2el:20000 \
  --save-result --result-filename run-rate9.json

Line by line on the third command: --random-range-ratio 0.3 restores length variance, without which p99 is meaningless; --random-prefix-len 512 gives the prefix cache something realistic to hit rather than 0% or 100%; --seed 0 makes the sampled lengths reproducible; --burstiness 1.0 states the Poisson assumption explicitly rather than leaving it implicit; --num-prompts 20000 is chosen against MLPerf’s confidence-interval table rather than by feel; --goodput adds an e2el bound so that single-token responses cannot dodge the TPOT constraint; and --save-result produces the artefact that makes the run auditable later.

Failure Modes and Common Misunderstandings

“Both tools report ITL, so the numbers are comparable.” They are not. LLMPerf includes TTFT, GenAI-Perf excludes it and normalises per-chunk tokens, vLLM’s itl array is per-frame while its tpot is per-token. Name the tool and the metric definition in every result.

“We ran at --request-rate inf to find the maximum.” That is an Offline burst test wearing a Server test’s clothes. It measures the engine with the arrival process removed and reports a throughput that no latency-constrained deployment can sustain.

“We benchmarked on ShareGPT, which is realistic.” At vLLM’s defaults it is realistic up to 1,024 prompt tokens and 2,048 total. Long-context behaviour, KV cache pressure and preemption are structurally excluded.

“Synthetic fixed-length inputs isolate the variable.” They isolate it by removing the thing that generates tail latency. range_ratio = 0.0 makes p99 ≈ p50 and hides head-of-line blocking entirely.

“Throughput went up, so the change was good.” Check whether preemption also went up — recomputed prefill tokens count as prompt tokens. And check whether output length went down; shorter answers are cheaper and worse. MLPerf’s generation-length floors exist because of exactly this.

“We got 30% more tokens per second by raising the streaming interval.” Possibly true and possibly good — but verify it against a per-token metric, because chunk-interval ITL and per-token TPOT move in opposite directions under that change.

“One 60-second run is enough.” MLPerf mandates 600 seconds and publishes the query counts required for a 99th-percentile estimate at a stated confidence. Near saturation the variance between runs can exceed the effect you are measuring.

“The vendor’s MLPerf number applies to our deployment.” MLPerf Closed submissions are heavily tuned, run on a fixed dataset with fixed generation parameters, and are audited against a quality target. That is what makes them comparable to each other. It does not make them predictive of your prompt distribution, your quantization, or your multi-tenant traffic mix.

Production Notes

Three habits separate benchmarking that informs decisions from benchmarking that generates slides.

Benchmark the thing you will deploy, not the engine. The full request path includes the gateway, the endpoint picker, tokenization, any guardrail model, and the network. Benchmarking the model server’s port directly measures a component; benchmarking through the gateway measures the product, and prefix-cache-aware routing in particular can only be evaluated end to end, because its entire benefit is landing a follow-up turn on the replica that already holds its prefix (Prefix-Cache-Aware Request Routing).

Keep a standing regression run and treat drift as a bug. Fix a dataset, a seed, a rate and an SLO, run it on every release, and store the JSON. This is the case where p50 earns its keep — as a tripwire that detects a whole-distribution shift, per Time to First Token and Inter-Token Latency. A regression suite that changes its workload between runs detects nothing.

Re-run the sweep when the traffic changes, not on a calendar. Goodput is a function of the input and output length distribution, so it decays as your product changes. DistServe’s system automates this with a workload profiler that watches “the average input and output length of the requests, the average arrival rate” and triggers a replan on a significant shift, noting the search “runs in seconds” while reloading weights “can be completed within minutes — far shorter than the hourly scale at which real-world workload variations tend to occur” (Zhong et al. 2024). Without that machinery, the manual equivalent is to alert on prompt-length distribution drift from your production vllm:request_prompt_tokens histogram and re-benchmark when it fires.

Finally, a note on why this area is so unsettled. Benchmarking standardisation for LLM serving is an active effort, not a solved problem: inference-perf exists because the Kubernetes Serving Working Group concluded that neither the existing harnesses nor the existing metric definitions were comparable across model servers, and vLLM’s own metrics design document links that group’s “Benchmarking LLM Workloads for Performance Evaluation and Autoscaling in Kubernetes” proposal while conceding the underlying question — where the saturation point of a model server actually is — remains open. Treat any single tool’s output as one tool’s opinion, cross-check with a second, and always publish the workload specification alongside the number.

See Also

Graph View

Backlinks