Prefill is a matrix multiply; decode is a memory read wearing a matmul's clothes

The two phases are separated by one number: how many tokens share each weight read. Prefill shares it across the whole prompt and lands compute-bound; decode shares it across one token and lands memory-bound. This lesson does the arithmetic that makes the difference concrete.

14 MIN

TL;DR: Both phases perform the same operations against the same weights. The only difference is how many tokens ride along with each weight read, and that single number moves the work from one side of the roofline to the other. Batching decode is not a trick, it is the act of turning decode back into prefill's shape.

Where you are. Middle of the second module. You know the two phases exist. This lesson does the arithmetic that explains why they behave so differently, because the numbers are what you will be asked for.

The same work, a different divisor

Take a dense model with N parameters at b bytes each. Both phases read all of it.

bytes read per pass         = N x b                      (the whole weight set, both phases)
arithmetic per token        ~ 2N                         (one multiply-accumulate per parameter)

PREFILL, prompt of T tokens processed together
  bytes read                = N x b
  operations                = 2N x T
  intensity I               = 2NT / Nb = 2T / b

DECODE, one token per pass, batch of B sequences
  bytes read                = N x b   + KV traffic
  operations                = 2N x B
  intensity I               ~ 2B / b

The weight read is the same in both. What changes is the numerator, and it changes by a factor of the prompt length or the batch size.

Put numbers on it with b = 2 bytes per parameter:

Situation	Tokens sharing the read	Intensity	Side of the ridge
Prefill, 2,000-token prompt	2,000	~2,000	far compute-bound
Decode, one user	1	~1	far memory-bound
Decode, batch 32	32	~32	still memory-bound
Decode, batch 256	256	~256	approaching the ridge

The ridge point on a current accelerator sits in the low hundreds of operations per byte, so the table says something specific and useful: batching moves decode toward compute-bound but needs a batch in the hundreds to get there, and a serving system's entire scheduling design is downstream of that fact.

It also explains a result that surprises people. Doubling the batch from 1 to 2 roughly doubles throughput at almost no latency cost, because the weight read is paid once either way. Doubling from 256 to 512 buys much less, because by then the read is amortised and you have arrived at the compute ceiling. The returns are steeply diminishing and the shape is knowable in advance.

What decode reads besides the weights

The weight read is the large term but it is not the only one. Each decode step also reads the KV cache for every sequence in the batch, and that term grows with context length and with batch size while the weight term stays fixed.

bytes per decode step = N x b            (weights, fixed)
                      + B x T x kv       (cache, grows with batch and context)

At short contexts the weight term dominates and batching looks almost free. At long contexts the cache term takes over, and then adding sequences to the batch adds proportional traffic rather than sharing existing traffic. That is the point where throughput stops improving with batch size, and it arrives earlier the longer your contexts are.

This is worth stating precisely because it is the most common wrong intuition in serving: batching is not uniformly good. It is close to free while the fixed weight read dominates, and it stops being free once the per-sequence cache traffic catches up. Where that crossover sits is a property of your context lengths, which is another reason the workload distribution has to come first.

Do this before moving on

Write the decode step-time expression for a model you can name, with symbols rather than numbers:

step time ~ ( N x b + B x T x kv ) / bandwidth
tokens per second = B / step time

Then ask what happens to tokens per second as B rises, holding T fixed. For small B the numerator barely changes, so throughput rises almost linearly. For large B the second term dominates, so throughput flattens. You have just derived the curve in the diagram from two terms, and you can now say where the knee is for any context length rather than remembering a number.

Go deeper

Prefill vs Decode carries the same arithmetic with the model dimensions filled in.
Continuous Batching is what a real system does with the batching result above, and it is the first gated concept worth your time here.
KV Cache Sizing gives you the kv term as a formula, which the next module turns into a capacity plan.
Arithmetic Intensity by Operation tabulates where these operations land so you can check your reasoning.
How big can a batch be before decode is compute-bound? is exactly the derivation above, asked as an interview question.
Tokens per second for a 70B single stream is the batch-one case worked end to end.

Key takeaways

Both phases read the whole weight set; the only difference is how many tokens share that read.
Prefill divides the read across the prompt and lands compute-bound; decode at batch one divides it across a single token and lands memory-bound.
Batching moves decode toward the ridge, but needs a batch in the hundreds to arrive, which shapes every serving scheduler.
Decode also reads the KV cache, and that term grows with batch and context while the weight term stays fixed.
Batching is nearly free while the weight read dominates and stops being free once cache traffic catches it; the crossover depends on your context lengths.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1Decode throughput improves sharply from batch 1 to batch 8, then barely improves from batch 256 to batch 512. Why?
2Two deployments run the same model at the same batch size, but one serves 500-token contexts and the other 100,000-token contexts. Why does batching help the first far more?
3An engineer proposes moving a decode-heavy service to an accelerator with the same memory bandwidth but far higher arithmetic throughput, at batch 16. What should you predict?