A request becomes tokens, and tokens become two different workloads

TL;DR: A request is tokenised, then processed in two phases. Prefill runs the whole prompt through the model at once, which is wide parallel work. Decode then produces one token at a time, each pass reading every weight in the model. The two have opposite hardware profiles, they compete for the same device, and nearly every optimisation in serving exists to manage that conflict.

Where you are. Second module of the trunk course. You know the machine is bandwidth-limited and how to name a binding resource. Now the workload: what a request actually asks the machine to do.

Tokens first, because the unit matters

Nothing in a serving system counts requests. It counts tokens, and a request is a variable number of them.

Text arrives, a tokeniser splits it into token identifiers, and those become the model's input. The output is generated token by token and detokenised on the way back. This matters for infrastructure for one reason: cost, memory and latency all scale with tokens, and a request tells you nothing about how many there are. A one-line prompt referencing a pasted document is not a small request.

So the first question about any workload is never "how many requests per second". It is the distribution of prompt lengths and output lengths. A fleet sized on request rate alone is sized on a number that does not determine the cost.

The two phases

Prefill takes the whole prompt and runs it through the model once. Every token in the prompt is processed together, so the work is a set of large matrix-matrix multiplications. Lots of parallel work, high arithmetic intensity, and it lands on the compute-bound side of the roofline from the first lesson.

Decode produces the output one token at a time. Each pass takes a single new token, runs it through every layer, and emits the next one. Because there is only one token in flight, the matrix multiplications collapse to matrix-vector products: the machine reads the entire weight set to do a small amount of arithmetic. Intensity near 2, deeply memory-bound.

Between them sits the thing that makes decode possible at all. Attention lets each token look back at every previous token, and recomputing all of that history for every new token would be quadratic and absurd. Instead the per-token key and value vectors are kept, and each new token attends against the stored ones. That store is the KV cache, it grows by one entry per token per layer, and it is why memory pressure in a serving system rises as conversations get longer rather than staying flat.

Why the split is the whole subject

Two phases on one device, wanting opposite things, is a scheduling problem with no clean solution. That is not a flaw in any particular system; it is the shape of the problem.

Read the last row twice. Prefill determines when a user sees anything at all. Decode determines whether the text then arrives smoothly. They are different experiences with different fixes, and a system tuned hard for one will usually be worse at the other.

That single tension generates most of the serving techniques you will meet later, and it is worth seeing the list now even though each gets its own treatment:

• Batching groups decode passes so one weight read serves many sequences.
• Paging stops the cache from reserving memory it will not use.
• Prefix reuse avoids redoing prefill for text the system has already seen.
• Chunking stops one long prefill from blocking every decode behind it.
• Splitting the phases across separate machines lets each be tuned alone.

Every one of those is a response to the same asymmetry. When you meet them individually, the useful question is always which side of the split they act on.

What this buys you in an interview

Asked to design a serving system, the weak opening is a diagram with a load balancer. The strong opening establishes the workload: prompt length distribution, output length distribution, concurrency, and which of the two latency experiences the product actually cares about.

That is not stalling. Those four answers determine whether the system is prefill-dominated or decode-dominated, and those two situations have different architectures. A candidate who asks is doing the job. A candidate who starts drawing has skipped it.

Do this before moving on

Take two workloads and decide which phase dominates each:

1. A code assistant that receives 8,000 tokens of context and returns a 40-token completion.
2. A chat product that receives 200 tokens and streams back 800.

Compute the ratio of prompt tokens to output tokens for each, then say which phase you would optimise first and what you would expect to gain. The first is prefill-heavy by a factor of 200; the second is decode-heavy by 4. The architectures that suit them are not the same, and noticing that is the entire point of the exercise.

Go deeper

• Prefill vs Decode is the canonical treatment of the split, with the arithmetic for each phase written out.
• The KV Cache explains what is stored and why it grows the way it does, which is the next thing you need.
• Continuous Batching is the first and largest lever on the decode side, and shows why static batching wastes the machine.
• Chunked Prefill is the answer to prefill blocking decode, and worth reading once you can state why the two interfere.
• Prefill vs decode: why they need different hardware is the interview version of this lesson.
• What is the KV cache and why does it grow? has the derivation you will be asked for.

Key takeaways

• Serving systems count tokens, not requests, and a request tells you nothing about how many tokens it carries.
• Prefill processes the whole prompt at once: matrix-matrix, compute-bound, and it determines time to first token.
• Decode produces one token per pass and reads every weight to do it: memory-bound, and it determines the gap between tokens.
• The KV cache exists so decode does not recompute history, and it grows with every token, so memory pressure rises as conversations lengthen.
• Batching, paging, prefix reuse, chunking and phase splitting are all responses to the same asymmetry.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1A product receives 8,000-token prompts and returns 40-token completions. Which phase dominates, and what follows for the architecture?

   Decode, because the KV cache is largest during generationDecode, because output tokens are generated one at a timePrefill, because there are 200 prompt tokens per output token, so the compute-bound phase carries almost all the workNeither; they are balanced because both run once per request

2. 2Why does decode read the entire weight set to produce a single token, when prefill reads it once for the whole prompt?

   Answer it, then reveal

3. 3An interviewer says 'design a system to serve 10,000 concurrent chat users' and the candidate immediately draws a load balancer and a pool of replicas. What has been skipped?

   The choice of serving engine, which should come firstThe quantisation decision, which determines the replica countThe workload: prompt and output length distributions, real concurrency, and which latency experience the product cares about, all of which decide whether the system is prefill- or decode-dominatedNothing; that is a reasonable starting point for any serving system