TL;DR: Continuous batching lets finished requests leave and waiting work join between model iterations. It reuses capacity before the longest request ends. The gain depends on the workload and scheduler budgets; a fuller batch can increase throughput while also lengthening the gap between each user's tokens.
How to approach it
First ask whether outputs have different lengths and whether requests must stream interactively. Then trace one waiting request through a fixed batch and an iteration-level scheduler with the same concurrency cap. Close by separating the scheduling opportunity from the memory and latency limits that decide whether admitting more work helps.
A strong answer
A typical situation: A needs a short answer, B is writing a long one, and C is waiting. A finishes. Why should C have to wait for B too?
A fixed-membership batch admits a group and runs it until all members finish. It may stream A's completed answer immediately, but cannot fill A's place with C during that batch. Some implementations execute padded rows; others mask or compact finished work. The defining limitation is no replacement request, not mandatory padding.
Continuous Batching changes membership between iterations. An iteration is one scheduled model step, not an entire response. That allows C to start while B is still running, if the scheduler has room. The following deliberately small example isolates this difference.
Read the schedule: the same work, two slots
A and B start ready to decode. A needs 2 iterations, B needs 6. C is already waiting and needs 2 decode iterations after a 1-iteration prefill. Prefill means processing C's prompt before its later decode steps. These are synthetic work units, not measured milliseconds; assume C's prefill can run beside B's decode without changing iteration duration.
static: A+B for 6 iterations, then C prefill + decode for 1+2
total = 6 + 1 + 2 = 9 iterations
continuous: A finishes at 2; C prefill at 3; C decode at 4 and 5
total = max(B finishes at 6, C finishes at 5) = 6 iterations
completion-window speedup = 9 / 6 = 1.5× under equal iteration durations
sanity: both schedules do the same 10 decode iterations of useful work
(A:2 + B:6 + C:2) and the same one prefill; only overlap changes.
The picture shows an opportunity to overlap work. It does not establish a production speedup. If mixing prefill makes a step longer, B's token gaps grow. If C's prompt needs several chunks, C may take longer to enter decode. Comparing static cap 8 with continuous cap 32 would also change the concurrency cap, so it would not isolate this scheduling improvement. Orca introduced iteration-level scheduling; its reported results belong to its evaluated workloads.
Now change one assumption. Suppose ordinary iterations and C’s standalone prefill take 1 ms, but its prefill mixed with B takes 3 ms. Static completion stays at 9 ms. Continuous completion takes five ordinary iterations plus the mixed one: 5 × 1 + 3 = 8 ms. C finishes at 7 ms and B at 8 ms. If the allowed gap between B’s tokens is 2 ms, the mixed step violates it despite the shorter overall window. These durations are hypothetical, not an Orca result.
Why does batching help a GPU? In a bandwidth-limited dense-model decode step, multiple sequences can share loaded weight tiles. Filling available capacity amortizes those reads across more useful tokens. Larger batches also add arithmetic and cache traffic, so step duration need not stay fixed. The KV Cache explains retained state; Chunked Prefill explains how to limit the prompt work mixed into an iteration.
A scheduler checks several limits before bringing C into an iteration. It needs room under the running-sequence cap, enough token work allowed in that iteration, and KV blocks for the state the work will create. (KV blocks are fixed-size pieces of GPU memory that hold each request's attention cache; PagedAttention explains them.) Priority and admission policy matter too. An empty sequence slot is only one of those checks. C still needs prefill, and it joins at a boundary between iterations rather than interrupting a running kernel. When a request finishes or is cancelled, its references can be released, although reusable prefix blocks may remain cached.
| Observation under the same offered workload | What to investigate |
|---|---|
| Queue grows; few requests are running | Admission policy, unavailable KV blocks, or another bottleneck |
| Preemptions rise as contexts grow (the scheduler pauses a running request and frees its memory to make room) | Insufficient KV allocation for scheduled work; inspect per-rank capacity |
| Long token gaps align with large prefills | Reduce prefill work per iteration and remeasure |
| Tokens/s rises but p99 token gaps worsen | More aggregate work, worse streaming latency; check the service target |
Use continuous batching for variable-length interactive generation, with bounded admission and measured streaming latency. The reversal condition: a controlled offline workload whose requests have similar work and no queueing benefit from mid-batch refill; a simpler fixed batch may be sufficient. Evaluate both on the same inputs, cap and latency contract.
What interviewers probe next
- "Does A have to wait for B to receive its answer?" No. A fixed batch may stream or return A's result earlier; the refill limitation concerns admitting C.
- "Is an empty slot enough to admit C?" No. C needs permitted token work and KV allocation, including the scheduler's growth policy, before it can run.
- "Which metric would make you reject a throughput gain?" A breach of the streaming latency target, such as p99 inter-token latency (the gap between tokens that 99% of gaps beat), or fewer successfully completed requests meeting the full service contract.
Common mistakes
- Presenting a larger concurrency cap as proof of continuous batching's gain.
- Treating high GPU utilization as proof that all work is useful.
- Drawing C directly in decode without accounting for its prompt prefill.
- Promising a fixed latency increase independent of context, batch and hardware.
Key takeaways
- Batch membership can change between iterations; a running kernel is not interrupted.
- The toy schedule takes 9 versus 6 iterations under stated equal-duration assumptions.
- A free slot still needs token-work and KV capacity before admission.
- Compare throughput and per-request latency together on an unchanged workload.