Your p99 time to first token is four times p50. Find out why.

A tail that wide is queueing or prompt-length variance, and the two are distinguishable in one measurement. Four causes ranked by how often they are it, the decomposition that attributes the wait, and the fix that is usually a flag rather than hardware.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🚀 Inference & Serving🔒 Premium

Chunked PrefillA long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.

Core

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency objective is met or missed by the sum of a chain of delays, and the way to design for it is to write the chain down with a number on every link, find the links that dominate at the tail, and attack those. For an LLM request the chain is network, gateway, router, queue, prefill, then the decode loop, and the tail is shaped by queueing and by the size of the batch the request lands in. This page decomposes a 500 ms time-to-first-token budget link by link, derives how queueing turns a comfortable median into a broken p99, and gives the design moves (admission control, chunked prefill, priority lanes, hedging) that hold it.

Foundational

🧮 Open Weights & Serving Engines

vLLM Server Arguments That MatterA vLLM deployment is mostly decided by a dozen flags, and the ones that matter fall into four groups: how the model is split across GPUs, how memory is divided between weights and cache, how requests are batched, and which specialized backends the model needs. Getting the first two wrong produces an engine that will not start or that runs out of memory under load. Getting the third wrong produces an engine that starts, serves, and misses its latency target by a wide margin.

Foundational

🧮 Open Weights & Serving Engines

Serving Benchmarks That Do Not LieMost published serving numbers are not comparable to each other and not predictive of production, because they differ in the input distribution, the concurrency, whether the cache was warm, and which of several very different metrics is being reported. A benchmark that supports a decision has to fix all four, report a distribution rather than a mean, and be run against the traffic shape you actually serve. The single most useful discipline is to compute the bandwidth bound first, so you know what fraction of the possible you achieved.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYou have one week to get a newly released trillion-parameter model serving production traffic. Plan it.Next in this trackSelf-host a 753B open-weights model or call a hosted API? Work the crossover.Next in this trackOne replica in a serving fleet returns tokens slower than its siblings. Find out why.

FEDITOR'S NOTE

Scored on decomposing TTFT into queue wait and prefill, on prompt-length variance as the usual cause, and on chunked prefill and admission as the remedies.

DISCUSSION · 0

No comments yet — be the first to share your approach.