TL;DR: Prefill processes the whole prompt in one pass, so thousands of tokens share each weight read and the GPU is compute-bound; it sets time to first token. Decode produces one token per sequence per step, so a small batch reads 141 GB of weights to do almost no math and the GPU is bandwidth-bound; it sets time per output token. The ridge point of the accelerator, about 295 FLOP per byte on an H100, is the line between them, and batch size is what moves decode toward it.
How to approach it
Open by naming the two phases and the metric each one owns: prefill produces the first token, decode produces every token after it. Ask which model and which card the interviewer has in mind, because the argument needs a weight size and a bandwidth figure; if they leave it open, pick Llama 3.1 70B on H100 and say so. Then say you will compute the arithmetic intensity of each phase and compare it against the card's ridge point, since that single comparison decides which resource is the bottleneck. Close by connecting each phase to the SLO it controls.
A strong answer
A typical situation: a fleet is sized on one throughput number and then misses its TTFT target on long prompts and its TPOT target on short ones. The two halves of a request want opposite hardware, and one number cannot describe both.
A transformer forward pass reads every weight once per step regardless of how many tokens are in the step. The useful work per weight read is what separates the phases. The Prefill vs Decode split is the first thing to draw:
inputs (Llama 3.1 70B, bf16, one H100 SXM)
weights W = 70.6e9 params × 2 B = 141.2 GB
FLOPs per token per pass = 2 × params = 2 × 70.6e9 = 1.41e11 FLOP
H100: 989 TFLOPS dense bf16, 3.35 TB/s HBM
ridge point = peak FLOPS ÷ bandwidth
= 989e12 ÷ 3.35e12 ≈ 295 FLOP per byte
arithmetic intensity of a step = FLOPs ÷ bytes read
prefill, 2,000-token prompt: (2,000 × 1.41e11) ÷ 141.2e9 ≈ 2,000 FLOP/B (above 295: compute-bound)
decode, batch 1: (1 × 1.41e11) ÷ 141.2e9 = 1 FLOP/B (far below 295: bandwidth-bound)
decode, batch 64: 64 FLOP/B (still bandwidth-bound)
sanity: intensity for bf16 decode equals the batch size, so decode only reaches the ridge near batch 295
That table is the whole mechanism. During prefill every token in the prompt multiplies against the same weight tile while it sits in registers, so the tensor cores are the limit and the phase scales with prompt length. During decode there is one new token per sequence, so the step is a pass over 141 GB of HBM to compute a handful of matrix-vector products.
The two phases therefore have different time formulas:
TTFT ≈ prompt tokens × 2 × params ÷ (GPUs × peak × MFU)
= 2,000 × 1.41e11 ÷ (8 × 989e12 × 0.4) (8 H100s, MFU 0.4 for prefill)
= 2.82e14 ÷ 3.16e15 ≈ 89 ms
TPOT at batch B ≈ (W + B × KV per sequence) ÷ aggregate bandwidth
batch 1: 141.2 GB ÷ (8 × 3.35 TB/s = 26.8 TB/s) ≈ 5.3 ms per token
batch 64, 4k context (KV per seq = 328 KB × 4,096 ≈ 1.34 GB):
(141.2 + 64 × 1.34) GB ÷ 26.8 TB/s = 227 GB ÷ 26.8 TB/s ≈ 8.5 ms per token
sanity: 64 users each get a token every 8.5 ms, so the fleet emits about 7,500 tokens/s;
at batch 1 it emits 190 tokens/s from the same eight cards
The consequence for hardware is that the two phases want different things. Prefill wants FLOPS, and an H100 and an H200 are the same die, so they prefill at the same speed. Decode wants bytes per second and capacity: the H200's 4.8 TB/s against the H100's 3.35 TB/s cuts the batch-1 step from 5.3 ms to 3.7 ms without changing the compute at all, and its 141 GB lets more sequences ride on each weight read. A shop whose traffic is short prompts and long generations buys bandwidth. A shop doing document summarization with 30k-token prompts and 200-token answers buys FLOPS. That is why Disaggregated Prefill and Decode exists as a design at all: the phases are cheapest on different pools.
The decision: serve on the higher-bandwidth part when decode dominates the token count. The reversal condition: when the prompt-to-output ratio climbs past roughly 20 to 1, because at that point prefill compute is most of the GPU-seconds.
What interviewers probe next
- "Why does adding batch help decode but not prefill?" Because decode intensity equals the batch size in bf16, so every added sequence is free until the step reaches the ridge, while prefill is already above the ridge at a few hundred tokens and extra tokens cost proportional compute.
- "What happens to decode at 32k context?" The KV read joins the weight read: 32,768 × 328 KB ≈ 10.7 GB per sequence, so a batch of 16 reads 172 GB of KV on top of 141 GB of weights, and the step time more than doubles.
- "Does fp8 change which phase is bound?" It halves the bytes per weight, so decode intensity doubles per unit of batch and the ridge in batch terms stays near 295 on an H100 because fp8 peak also doubles; it mostly buys capacity and a faster step, not a change in regime.
Common mistakes
- Saying "inference is memory-bound" without splitting the phases; a 20k-token prefill is as compute-bound as training.
- Quoting sparse tensor-core peaks (1,979 bf16 TFLOPS "with sparsity") so the ridge point comes out at 590 and the batch math is wrong by two.
- Treating TTFT as a queueing number and forgetting the prefill compute term, which is most of it for long prompts.
- Assuming the H200 speeds up prefill.
Key takeaways
- Intensity of bf16 decode equals batch size; the H100 ridge is about 295 FLOP/B, so decode is bandwidth-bound until the batch is in the hundreds.
- Batch-1 decode of a 70B model on 8 H100s is about 5.3 ms per token, set by 141 GB over 26.8 TB/s.
- TTFT scales with prompt tokens times 2 × params over available FLOPS; TPOT scales with bytes per step over bandwidth.
- Bandwidth and capacity buy decode; FLOPS buy prefill; the prompt-to-output ratio decides which you are buying.