prefill

One forward pass reads every weight. Whether that read is the bottleneck depends on how many tokens ride along with it, and the answer is different for the two halves of a request.

Prefill and decode fight over the same GPU and each ruins the other's latency. Putting them on separate pools ends the fight, at the price of shipping every request's cache across the network. The break-even is a number you can derive.

At 128k tokens one request's cache is 43 gigabytes and its prefill is measured in seconds. Capacity breaks first, then TTFT, then the scheduler. Each has a fix, and the numbers say which fix you need at which length.

An image is not one token; it is hundreds to thousands of them after a second model has run. That changes the prefill budget, the batch shape and where the first bottleneck appears.

The two phases want opposite machines, so running them together makes every setting a compromise. What splitting buys, the KV transfer that couples the pools, and the arithmetic that says whether the coupling is affordable for this model.

Four things break in order and only one of them is the attention math everyone expects. What a single million-token sequence costs in memory, why the first token takes minutes, and what the model's own architecture already did about the parts that would otherwise be impossible.

An image becomes a large number of tokens before the language model sees anything, so a request that looks small carries a prefill the size of a long document. The token arithmetic, the encoder that sits outside the usual parallelism, and the two capacity numbers that move.

Prefill is the compute-bound half of serving: 2 × N × tokens FLOPs over the effective TFLOPS of the replica. The chain that gives a 0.24 s floor on a node, the single-card version, and what it means for time to first token.

Two bounds, bandwidth and compute, and five questions about what the number counts. For a 70B on an H100 the claim exceeds the bf16 compute peak and is impossible; for an 8B at high batch it is routine; for input tokens it is easy. The chain that tells the cases apart.

Ten thousand queries a second through an embedder, a sharded vector index, a reranker and a 70B answer model. The latency budget per stage, the two fleets sized from tokens and pairs rather than queries, and the cache that decides whether the LLM tier is 60 nodes or 200.

200 milliseconds at p99 is a budget, and prefill alone spends it on a long prompt. The decomposition into network, queue, routing, prefill and first token, the prompt length the budget allows, chunked prefill in the tail, admission control that rejects what it cannot serve in time, and the utilization the SLO costs.

Prefill is compute-bound and decode is bandwidth-bound, and one GPU running both does each worse. The pool ratio from the traffic's prompt and output lengths, the KV transfer cost per request and the link it needs, the router that pairs a prefill worker with a decode worker, and the load at which it beats a monolith.