ttft

One forward pass reads every weight. Whether that read is the bottleneck depends on how many tokens ride along with it, and the answer is different for the two halves of a request.

Tokens per second flatters a system that is failing its users. The metrics that matter are two latencies at a percentile and the fraction of requests that meet both.

One 20k-token prompt can freeze every active stream for most of a second. The fix slices it into per-step budgets, and the budget number is a trade between two SLOs you can compute.

Two hundred milliseconds is about four thousand prompt tokens of prefill on a full node, before any queueing. Every design choice is about not spending that budget twice.

A tail that wide is queueing or prompt-length variance, and the two are distinguishable in one measurement. Four causes ranked by how often they are it, the decomposition that attributes the wait, and the fix that is usually a flag rather than hardware.

Prefill is the compute-bound half of serving: 2 × N × tokens FLOPs over the effective TFLOPS of the replica. The chain that gives a 0.24 s floor on a node, the single-card version, and what it means for time to first token.

Four objectives, each measured at a percentile because averages hide the experience you are promising. The error budget in minutes per month, the burn-rate arithmetic that catches a fast failure in an hour and a slow one in a day, and why two of the four need separate targets per traffic class.

200 milliseconds at p99 is a budget, and prefill alone spends it on a long prompt. The decomposition into network, queue, routing, prefill and first token, the prompt length the budget allows, chunked prefill in the tail, admission control that rejects what it cannot serve in time, and the utilization the SLO costs.