TL;DR: Follow the request: the gateway checks admission, the router chooses an eligible replica, and the engine prepares input, schedules model work and manages attention state. Prefill processes the uncached prompt; repeated decode iterations produce the continuation. Output streams back while telemetry measures the whole path. Size the deployment from a representative workload and measured service capacity, not a count of signed-in users.
How to approach it
First agree on the model, input/output lengths, arrival pattern and latency objectives. Trace one accepted request, then show the token loop inside its replica. Add cancellation and overload at their ownership boundaries before calculating capacity. Close with the measurement that could change the design.
A strong answer
A chat interface shows a spinner, then words begin to appear. The GPU is only part of that wait: the request must pass admission, reach a suitable replica, obtain state and get scheduled. Start with this path so each component has a reason to exist.
The boxes show responsibilities that may share a process. In this drawing, the chosen replica tokenizes the input. A router that needs token counts could move that work earlier. Output streams as generation proceeds, so the client does not wait for the whole answer. The vLLM architecture overview gives one concrete division between API processing, engine scheduling and GPU workers.
Gateway and router. The gateway authenticates callers and enforces request size, quota and deadline policy. A quota shared across gateway instances needs coordinated accounting. The router chooses among replicas that serve the required model and are eligible for the tenant. A replica is one serving deployment, possibly spread across several devices; it is not necessarily one GPU. LLM Request Routing explains why a cache hit can lose to a shorter queue elsewhere.
Input, scheduler and cache. A chat template arranges messages and control markers into model input. The tokenizer converts that input into vocabulary IDs; a word can span several tokens. The scheduler then chooses which work runs in each iteration, checking both memory and execution budgets. The KV Cache holds attention state for positions the model has already consumed. A valid prefix-cache match can reuse that state. Matching text alone is insufficient if the model revision, adapter or tenant boundary makes sharing incompatible.
What the model repeats. For an ordinary causal decoder, embeddings map token IDs into vectors. Transformer layers update the representation using attention and feed-forward computation, with the architecture's normalization and residual paths. The output head produces scores over possible next tokens; the decoding policy chooses the next ID. The diagram groups those internals because the platform question is about the request's progress, not every tensor operation.
A, B, C and D are symbolic token labels, not a claim about a real tokenizer. The first pass consumes the prompt and yields the first output token. Each ordinary subsequent iteration consumes the latest generated token, using previously retained state, and yields another. Prefill vs Decode explains the two workloads. Speculative decoding and other architectures can change this schedule; do not present the simple loop as every implementation's execution trace.
Output and cleanup. Detokenization and streaming convert generated IDs into text events. A network chunk need not equal one token or one complete character. Propagate disconnects, deadlines and cancellation to the engine; otherwise an abandoned stream can keep consuming capacity. Stop selecting new work for the request, then release its active references only after in-flight consumers are safe. A policy may retain unused prefix state for later reuse.
| Owner | Evidence when the request stalls |
|---|---|
| Gateway | Rejections, quota decisions and deadline expiry |
| Router | Eligible replicas, queue estimates and stale cache hints |
| Scheduler | Waiting work, admitted tokens and state reservations |
| GPU workers | Kernel, communication and device evidence |
| Output path | Buffered bytes, disconnects and cancellation completion |
Controls around the path. Observability follows a request identifier through those boundaries. Measure time to first token, individual token gaps, failures and qualified completions with explicit denominators. The autoscaler changes eligible serving capacity; it does not sit between two tokens. Record load, warm-up and readiness times for the actual deployment. An idle worker with unloaded weights is not equivalent to a ready replica.
Give the drawing a workload. Use a synthetic exercise: 2,000 sessions each submit one request per minute, averaging 500 uncached prompt tokens and 300 output tokens. Suppose a benchmark on this same combined prefill/decode workload establishes a planning rate of 5 requests/s per replica while meeting the required latency and failure criteria. This is a supplied assumption, not a measured result for a named GPU.
arrivals = 2,000 / 60 = 33.33 requests/s
uncached prompt load = 33.33 × 500 ≈ 16,667 tokens/s
output load = 33.33 × 300 = 10,000 tokens/s
healthy capacity candidate = ceil(33.33 / 5) = 7 replicas
one-replica-loss candidate = 7 + 1 = 8 replicas
surviving service capacity = 7 × 5 = 35 requests/s
sanity: the survivor margin is only 35 - 33.33 = 1.67 requests/s.
The calculation gives a candidate of eight replicas. Before choosing it, test bursts and the length distribution, check memory on every rank, and define which failures the service must survive. One rack failure could remove several replicas at once. The assumed 5 requests/s already includes prefill and decode, so adding another arbitrary prefill allowance would count that work twice. If the same-resource benchmark supports only 4 requests/s per replica, surviving one loss instead requires ceil(33.33/4) + 1 = 10 replicas.
Start with a combined serving pool and explicit admission limits. Add phase separation or remote cache tiers when measured interference, locality or capacity makes their extra transfer and recovery costs worthwhile. The reversal condition must name the observed problem and the outcome required; a fixed replica count alone does not justify either architecture.
What interviewers probe next
- "Traffic doubles before new workers become ready?" Bound admission and waiting time using surviving capacity; warm capacity helps only if it is actually ready for this model and workload.
- "Why not size from the active-user count?" Convert behavior into arrival rate and prompt/output work, then check concurrency and state separately.
- "The client disconnects after the first token?" Propagate cancellation, prevent more scheduled work and safely release active state; do not equate a closed socket with completed cleanup.
- "Where would you investigate a first-token regression?" Trace gateway, routing, queueing and prompt work for the affected slice before blaming GPU execution.
Common mistakes
- Drawing components without identifying what data moves or which component owns waiting.
- Routing solely for cache affinity while ignoring eligibility and queueing.
- Treating the KV cache as a separate output stage after model execution.
- Charging prefill twice after using a combined-workload capacity measurement.
- Freeing shared buffers as soon as cancellation arrives.
Key takeaways
- Trace admission, placement, state, execution and output; scaling and metrics surround that path.
- The emitted token becomes a later input; its cached state is created when it is consumed.
- The synthetic workload is 33.33 requests/s, about 16,667 prompt tokens/s and 10,000 output tokens/s.
- Eight replicas follows only from the assumed 5 requests/s planning rate and one-replica-loss requirement; 4 requests/s changes it to ten.