Prefill vs Decode
Follow one prompt through its first prediction and later decode steps, then separate GPU work from the wait a user sees. Learn why batch size and context change the likely bottleneck.
TL;DR: Prefill processes the prompt and produces the first output token. Ordinary decode then consumes one new token per sequence to predict the next, reusing the KV cache. Large prefills often benefit from more compute; small decode batches often benefit from more memory bandwidth. Those are useful starting hypotheses, not guarantees about a whole request.
Watch one response appear
You send a chat message. Nothing appears for a moment, then the answer streams out. Two kinds of model work sit behind that experience. A token is a piece of text; the model predicts tokens, which the application turns into readable text.
During prefill, the model processes your prompt, saves attention state in the KV Cache, and predicts the first output token. The cache holds key and value vectors: information later tokens can attend to without recomputing the prompt. A forward pass is one run of the model over its input, from the first layer to the last. During decode, the model consumes the token it just predicted, adds that token's state to the cache, and predicts another. See the Hugging Face cache walkthrough for the corresponding generation loop.
Here is a hypothetical 1,024-token prompt that produces four output tokens, A through D. Assume ordinary autoregressive generation, no speculative decoding and no reused prefix.
Four output tokens required one prefill and three decode steps. The newest output token enters the cache only if another forward pass consumes it. This distinction prevents an off-by-one error when tracing generation.
A serving engine can split a long prompt across several iterations using Chunked Prefill. In that case, finishing the final chunk makes the first output prediction possible. Prefill names the prompt-processing work; it does not require one uninterrupted scheduler iteration.
Why the same model behaves differently
The GPU must both move data from its high-bandwidth memory (HBM) and perform arithmetic. Memory-bound means data movement limits progress; compute-bound means arithmetic does. Think of these as limits to investigate, not permanent labels attached to a model.
A long prefill supplies many tokens to the model's linear layers together. (Linear layers are the large weight-matrix multiplies that hold most of a model's parameters.) Loaded weight tiles can serve multiple tokens, giving the GPU more arithmetic for the data it moves. At decode batch size one, each linear layer handles just one new token. There is much less opportunity to share weight movement across tokens. NVIDIA's matrix multiplication guide explains this relationship between matrix shape, reuse and performance.
Put numbers on it with one weight matrix of 4,096 × 4,096 values in BF16 (2 bytes each):
weight bytes = 4,096 × 4,096 × 2 = 33,554,432 bytes (about 33.5 MB)
work for 1 token = 2 × 4,096 × 4,096 = 33,554,432 FLOPs
FLOPs per byte, 1 token = 33,554,432 ÷ 33,554,432 = 1
work for 1,024 tokens = 1,024 × 33,554,432 = 34,359,738,368 FLOPs
FLOPs per byte, 1,024 = 34,359,738,368 ÷ 33,554,432 = 1,024
sanity: the bytes stay the same, so FLOPs per byte grows with the number of tokens sharing them
A GPU stays busy only above a certain ratio, its ridge point: peak FLOPs ÷ memory bandwidth. For an that is about 989 trillion ÷ 3.35 trillion ≈ 295 FLOPs per byte. One decode token at 1 FLOP per byte is far below it, so the GPU waits on memory. A 1,024-token prefill at 1,024 FLOPs per byte is above it, so arithmetic becomes the limit. The Roofline Model draws this picture. Real kernels also move activations and attention state, so treat these as the weight-only part of the story.
Batching several decoding sequences increases that reuse. It also adds their attention work and cache traffic. A longer context increases attention work without increasing the number of new tokens in a decode step. That is why batch size alone cannot predict the bottleneck.
| Situation | Starting hypothesis | What could change it? |
|---|---|---|
| Long uncached prompt | Linear-layer compute matters | Attention cost, kernel efficiency or communication may dominate |
| One decoding sequence | Weight traffic matters | Small models may instead spend much of the step on launches or sampling |
| Large decode batch | Better weight reuse | KV reads, compute and memory capacity grow with the batch |
| Mostly cached prompt | Less prefill work remains | The uncached suffix still needs processing; cache lookup and scheduling remain |
Measure the wait the user actually sees
Time to first token (TTFT) includes more than prefill: the request can queue, undergo tokenization, and wait for its first token to reach the client. Time per output token (TPOT) describes subsequent generation speed; client-observed gaps also include scheduling and delivery. State where timestamps were taken, especially when comparing two serving systems. Latency Metrics: TTFT, TPOT and Goodput explains the measurement choices.
For a hypothetical request with no overlapping stages:
| Event | Duration | Time since arrival |
|---|---|---|
| Queue and request preparation | 20 ms | 20 ms |
| Prefill and first-token selection | 80 ms | 100 ms |
| First-token delivery | 5 ms | 105 ms |
| Three more observed token gaps | 3 × 25 ms | 180 ms |
TTFT is 105 ms. The four-token answer finishes at 180 ms, and its three post-first-token gaps average 25 ms. These are invented teaching inputs, not a hardware benchmark. Sanity check: four tokens have only three gaps between them.
Choose a change from evidence
If long prompts interrupt active streams, inspect the scheduler timeline and try a smaller prefill chunk budget. If queueing dominates TTFT, faster prefill may leave the main problem untouched. If a GPU trace shows weight movement dominating decode, test higher bandwidth or weight quantization while checking output quality and latency under the same workload.
More compute is a plausible improvement for compute-limited prefill. Reverse that choice when measurements show time going elsewhere. Likewise, larger decode batches can raise total throughput while making individual users wait longer between tokens. Keep the latency target in the experiment; there is no universally free batch increase.
Check your understanding
A request asks for exactly one output token. How many ordinary decode steps follow prefill? Zero: prefill already provides that token's prediction. If its TTFT is poor, investigate the work before that first token rather than tuning subsequent decode steps.
Now suppose a 1,024-token prompt has a reusable 1,000-token prefix. Do you still perform the original full prefill? No. Reuse can reduce the prompt work, subject to the engine's cache granularity and matching rules. It does not make queueing, suffix processing or output generation disappear.
Key takeaways
- Prefill predicts the first output; four ordinary output tokens need three subsequent decode steps.
- One 4,096 × 4,096 BF16 matrix does 1 FLOP per byte for one token and 1,024 for 1,024 tokens, against a ridge near 295 on an H100.
- Weight reuse explains the common phase difference. Context, batch size and implementation decide the actual limit.
- In the timing example, 80 ms of prefill contributes to 105 ms TTFT; the four-token answer ends at 180 ms.
- Measure the breached latency target before choosing hardware or a scheduling change.