TL;DR: If the scheduler runs a long prefill before returning to decode, active streams wait for that work. Chunked Prefill divides the prompt into pieces that can share iterations with decoding. A token budget controls the amount scheduled; measured iteration and client-gap times determine whether it meets the latency targets.
How to approach it
Ask whether the replica shares prefill and decode, and inspect when the long prompt was actually admitted. Define the first-token and streaming-gap targets. Trace the token budget, then explain how you would choose a setting from measurements rather than derive a guaranteed millisecond limit from peak FLOPS.
A strong answer
Sixty users are watching answers stream. A new user submits a 20,000-token document. If one non-interleaved prefill occupies their shared execution resources, the existing requests can miss their next decode opportunity. That is head-of-line blocking: one large job at the front of the line holds up everything queued behind it. It is a plausible explanation for the pause, not proof from a latency histogram alone.
Continuous Batching means the scheduler can change membership at iteration boundaries. Chunked Prefill gives it more boundaries within prompt processing. Sarathi-Serve studies this mechanism, and the vLLM V1 guide describes a decode-prioritized token budget. Confirm the deployed scheduler's policy and available sequence/KV capacity before applying the example.
Assume sixty ordinary decoding sequences each contribute one token, all remain active, one uncached prompt is pending, and no speculative or other extra work consumes the budget:
total token budget = 2,048 per iteration
decode work = 60 tokens
available prefill work = 2,048 − 60 = 1,988 tokens
20,000 / 1,988 = 10 full chunks plus a remainder
10 × 1,988 = 19,880 tokens
last chunk = 20,000 − 19,880 = 120 tokens
prefill finishes in 11 iterations
The eleventh iteration schedules 60 decode tokens and only 120 prompt tokens. Sanity check: 10 × 1,988 + 120 = 20,000. Dividing by the total budget gives the wrong schedule, and pretending all eleven chunks are full overcounts prompt work.
At the last boundary, the scheduler should schedule only remaining work. Treating a budget as mandatory work would invent tokens the user never sent.
The new token cap does not bound a step to 95 ms or any other fixed time. More historical context changes attention work; different batch shapes change kernel efficiency; communication and runtime stalls can add delay. Chunking retains earlier prompt state in the KV Cache. It does not require recomputing every earlier chunk's keys and values from scratch.
The following hypothetical comparison shows how to choose a setting at the same offered load and length distribution. The prompt-count column follows from the arithmetic above. The latency percentiles are illustrative values; they cannot be calculated from the prompt count alone.
| Total budget (iterations for this prompt) | p99 client gap | p99 long-prompt TTFT |
|---|---|---|
| 512 (45 iterations) | 38 ms | 1,800 ms |
| 2,048 (11 iterations) | 95 ms | 1,050 ms |
| 4,096 (5 iterations) | 190 ms | 960 ms |
For targets of at most 100 ms p99 gap and 1,200 ms p99 TTFT, only the 2,048 setting meets both among these candidates. It has little gap headroom, so repeat at burst load and longer contexts before accepting it. If the gap target tightens to 50 ms with the same TTFT target, none of the measured candidates qualifies. That is the condition for revisiting capacity, admission, routing or phase separation rather than declaring the smallest chunk a solution.
Do not diagnose the stall from p99 alone. A single bad gap in a long answer may lie beyond the reported percentile, depending on the sample population. Correlate client gaps with request-level scheduling traces: did a large prefill run during that exact interval? Compare with server emission times to exclude transport buffering.
The current vLLM metrics reference includes vllm:inter_token_latency_seconds, vllm:num_requests_waiting and vllm:num_preemptions. Check names and boundaries for the deployed version. A rising preemption count suggests cache pressure deserves attention too; a smaller token cap may not resolve the actual cause.
What interviewers probe next
- “Why not make chunks tiny?” Scheduling and kernel overhead can increase, prompt progress may suffer, and existing gaps may already be near their achievable floor. Pick from measured feasible settings.
- “Does decode always go first?” The cited vLLM policy prioritizes it; other schedulers can differ. Memory availability, sequence caps and speculative work also affect what executes.
- “What if ten long prompts arrive together?” Their progress depends on admission and fairness across partial prefills. Measure per-request queueing and starvation; do not assume strict one-prompt-at-a-time service.
Common mistakes
- Forgetting the sixty decode tokens when calculating chunk count.
- Predicting a millisecond ceiling from a token cap or a peak-compute estimate.
- Claiming a fixed prefill penalty, guaranteed throughput increase or a particular histogram shape.
- Confusing average TPOT with the individual gaps that reveal pauses.
Key takeaways
- A 20,000-token prompt with sixty decode tokens and a 2,048 total budget needs eleven iterations; its last chunk has 120 tokens.
- Earlier prompt state is retained; scheduling boundaries change, not the requested attention context.
- In the hypothetical sweep, 2,048 meets both stated targets; a tighter gap target leaves no feasible tested setting.
- Verify the cause with traces and choose the budget from measured TTFT, individual gaps and prompt progress.