TL;DR: Static batching groups requests at arrival and runs the group until its longest member stops, so slots go idle as short answers finish. Continuous batching re-forms the batch at every decode step: finished sequences leave, queued ones join, and the GPU runs one pass over whatever is active. Because each decode step reads all 141 GB of a 70B model's weights whether one or sixty-four sequences ride along, keeping the batch full raises throughput several times over while per-token latency moves by a millisecond or two.
How to approach it
Frame the problem first: LLM requests have wildly different output lengths, and a batch is only as short as its longest member. Ask about the engine and traffic shape (chat with variable outputs versus fixed-length classification), because static batching is fine for the latter. Then say you will show the cost of an idle slot with the decode bandwidth arithmetic, describe the per-step scheduler loop, and finish with the two things it introduces that the interviewer will ask about next: prefill interference and KV admission.
A strong answer
A typical situation: a serving deployment reports high GPU utilization and low throughput, and the two are both true. Seven of its eight batch slots are computing padding while the longest request finishes.
Picture 8 requests arriving together for Llama 3.1 70B on 4 H100s with tensor parallelism. One answers in 12 tokens, another in 900. Under static batching all 8 run for 900 steps. After step 12 one slot is padding, after step 100 half of them are, and the queue behind them waits the full 900 steps for a slot. The GPU reports high utilization the entire time because it is running the same kernels on padded rows.
Why the idle slot is expensive comes from the decode step cost:
inputs: Llama 3.1 70B bf16 on 4 × H100 SXM
W = 141.2 GB, aggregate HBM = 4 × 3.35 = 13.4 TB/s
KV per sequence at 2k context = 328 KB × 2,048 ≈ 0.67 GB
step time ≈ (W + B × KV) ÷ bandwidth
B = 1: 141.2 ÷ 13.4 ≈ 10.5 ms → 95 tokens/s
B = 4: (141.2 + 2.7) ÷ 13.4 ≈ 10.7 ms → 373 tokens/s
B = 32: (141.2 + 21.5) ÷ 13.4 ≈ 12.1 ms → 2,640 tokens/s
sanity: 32x the sequences cost 15% more per step, because the weight read dominates
and it is paid once per step regardless of batch
A static batch of 8 averages 3 to 4 live sequences over its lifetime, so it sits near the 373 tokens/s row. Continuous Batching keeps the running set near its cap whenever demand exists, so the same hardware sits near the 2,640 row. That gap, roughly 7x on this traffic, is the reason vLLM and TensorRT-LLM's in-flight batching displaced request-level batching.
The scheduler loop is simple to state. After every forward pass: free the KV blocks of sequences that emitted a stop token or hit max length; admit waiting requests while free blocks cover their prompt plus room to grow; run the next step over the new set. Admission is gated on memory, never on a slot count, which is where the KV cache enters. A cap of 128 sequences that the KV pool cannot honor at typical context produces preemption storms under load: the engine evicts a running sequence, swaps or drops its blocks, and recomputes later. A non-zero preemption counter is the signal that the ceiling is memory rather than scheduling.
The cost is prefill interference. A newly admitted request needs its prompt processed, and the naive policy runs that prefill as its own step. A 20k-token prompt then stalls every running stream for the length of that prefill (about 900 ms on this fleet at 40% MFU), and the inter-token latency histogram grows a second mode aligned with admissions. Chunked Prefill is the remedy: slice the prompt into a per-step token budget and mix it with decode.
The decision: use continuous batching for anything with variable output length, which is every chat and agent workload. The signal that the fix landed is a p99 inter-token latency that stops tracking the longest request in the batch. The reversal condition: fixed-length, offline scoring where every input costs the same, where static batching with packed inputs is simpler and equally fast.
What interviewers probe next
- "A request arrives while a batch is mid-decode. What happens?" It is admitted on the next step if KV blocks are free, its prompt is prefilled (in one step or in chunks), and it decodes alongside the others from then on; nothing waits for the batch to finish.
- "Does continuous batching change per-token latency?" Slightly upward, from 10.5 to 12 ms in the example, because each step now reads more KV; the throughput gain is what the extra millisecond buys.
- "Which metrics tell you it is working?" Running and waiting request counts, KV utilization percent, preemptions per second, and the inter-token latency histogram; a bimodal ITL says prefill is interfering.
Common mistakes
- Explaining the gain as "better GPU utilization" without the weight-read argument, which is the mechanism.
- Confusing it with Triton Inference Server's dynamic batching, which groups at arrival and still runs each group to completion.
- Setting the sequence cap by GPU count rather than by KV budget ÷ expected context.
- Not knowing that admission triggers prefill and therefore an ITL spike.
Key takeaways
- Unit of scheduling is one decode step; sequences leave when they finish and new ones join on the next step.
- A 70B decode step on 4 H100s costs about 10.5 ms at batch 1 and 12 ms at batch 32; the weight read is paid once per step.
- Admission is gated on free KV blocks; preemptions mean the KV pool, not compute, is the ceiling.
- Prefill of admitted requests is the main ITL spike; chunked prefill fixes it.