A serving engine is a scheduler with a memory allocator attached

TL;DR: Two components explain nearly everything a serving engine does. A scheduler picks, at each iteration, which sequences advance. An allocator decides which sequences can be resident at all. They are coupled: the scheduler cannot admit what the allocator cannot house. Learn the pair and engine behaviour becomes predictable rather than mysterious.

Where you are. First lesson of Course 2. Course 1 sized a service from outside. This course opens it up, and the useful first move is to reduce it to two components rather than a feature list.

Two components, one loop

The core loop is small enough to state in full:

loop forever:
    admit    ask the allocator what fits; take waiting requests while it says yes
    prefill  process prompts for newly admitted sequences (possibly in chunks)
    decode   advance every running sequence by one token
    evict    release sequences that finished, were capped, or were cancelled
    repeat

Everything else is a policy inside one of those five steps. Continuous batching is the admit and evict steps happening every iteration rather than per batch. Chunked prefill is the prefill step being allowed to do part of a prompt. Paging is the allocator's data structure. Prefix reuse is the allocator noticing two sequences want the same bytes. Speculation changes what "advance by one token" means.

That is the value of the reduction: a long list of features collapses into five steps and two components, and a new technique can be placed rather than memorised.

The scheduler decides who advances

Each iteration the scheduler picks the running set. Its decisions:

Who gets admitted. Not just "is there a slot" but "will the allocator house this sequence, and what does admitting it do to the requests already running". Admitting one more sequence slows every existing one slightly, because the step gets bigger.

Whether prefill preempts decode. A newly admitted sequence needs its prompt processed. Doing that in one piece delays every decode step behind it; doing it in chunks interleaves the two. This single decision produces most of the visible latency behaviour in a busy fleet.

What happens when the allocator says no. Queue, reject, or take memory from someone already running. The third option, preemption, is the interesting one and the next module treats it properly.

The allocator decides who is resident

The allocator's job is the cache from Course 1: bytes per token times context times sequences. Its decisions:

How memory is carved. Contiguous per-sequence blocks sized for the worst case, or fixed-size pages with an indirection table. The second wastes far less, which is why it won.

Whether two sequences can share. If two requests begin with the same tokens, their cache entries for that prefix are identical, so they can point at the same pages instead of holding two copies.

What to do when full. Refuse admission, or evict a running sequence and recompute its cache later. Recompute is a prefill, so eviction converts a memory problem into a compute problem rather than solving it.

The dashed arrow is where most incidents live. When the allocator says no, something has to give, and which thing gives is a policy choice that shows up as either rejections, queueing or a preempted request paying for a second prefill.

Reading engine behaviour through the pair

Three behaviours that look like bugs and are the pair working as designed:

Throughput rises with load until it collapses. More concurrent sequences share each weight read, so throughput improves. Then the allocator runs out, the scheduler starts preempting to make room, preempted sequences need re-prefilling, and that recompute competes with the work that caused the pressure. The collapse is preemption thrash, and the fix is admission control rather than more memory.

Latency degrades for everyone when one long prompt arrives. The scheduler chose to prefill it in one piece, and every decode step waited. Not a fairness bug in the queue; a chunking decision.

Two identical requests cost differently. One hit a shared prefix already resident and skipped most of its prefill. Correct behaviour, invisible without the accounting field from Course 1.

None of those are explicable from a feature list. All three fall out of the two components.

Do this before moving on

Take five things you know a serving engine can do and place each one in the loop: which of the five steps does it modify, and is it a scheduler decision or an allocator decision? Try continuous batching, paged cache, prefix reuse, chunked prefill and speculative decoding.

Two are allocator, two are scheduler, and one changes what a decode step means. Getting the split right is the point; it is also how you will place the next technique you meet, which is the durable skill here.

Go deeper

• Serving Engines is the architecture comparison, and worth reading once the two components are clear rather than before.
• Continuous Batching is the admit and evict steps in detail, and the largest single behaviour difference between engines.
• PagedAttention is the allocator's data structure, and the next module opens it up.
• Capacity and Backpressure is the answer to the dashed arrow: what happens when the allocator says no.
• How PagedAttention works is the interview version of the allocator half.
• GPU utilisation is 30% in serving: why? is a case that only resolves once you can see both components.

Key takeaways

• A serving engine reduces to a scheduler deciding who advances and an allocator deciding who is resident, coupled through admission.
• The loop is admit, prefill, decode, evict; every feature is a policy inside one of those steps.
• Throughput collapsing under load is usually preemption thrash, and the fix is admission control rather than memory.
• Eviction converts a memory problem into a compute problem, because recompute is a prefill.
• Placing a new technique in the loop beats memorising it, and is what makes engine behaviour predictable.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1A fleet's throughput rises with offered load, then collapses sharply past a point. What is the most likely mechanism?

   Preemption thrash: the allocator runs out, the scheduler evicts running sequences to admit new ones, and the evicted ones need re-prefilling, so recompute competes with the work that caused the pressureThe model is thermally throttling under sustained loadThe network saturates between replicasBatch sizes exceed what the kernels are tuned for

2. 2Why does eviction not solve a memory shortage?

   Answer it, then reveal

3. 3One long prompt arrives and latency degrades for every user, not just that one. Which component made the decision, and what was it?

   The allocator, which reserved too much memory for the long sequenceNeither; this is expected contention that no policy can changeThe allocator, which evicted running sequences to make roomThe scheduler, which chose to process the whole prompt in one piece, so every decode step for the running sequences waited behind it