Sequential is fifty seconds; one batch is four. In between are the questions the interviewer is holding: how you pack ragged inputs, what a batch window costs, and where the memory stops you.
more free answers with an account · no card
← LLM Inference & Serving / 13
medium★ EssentialNewAnthropic
You have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.
Sequential is fifty seconds; one batch is four. In between are the questions the interviewer is holding: how you pack ragged inputs, what a batch window costs, and where the memory stops you.
Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.
The concepts behind this question
Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.
Core
Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.🚀 Inference & ServingSign in
Foundational
Multi-Region Serving and FailoverRunning inference in more than one region buys latency for distant users and survival when a region fails, and it costs a second fleet that must be capable of absorbing the first one's traffic. The design turns on three decisions: whether regions are active-active or active-passive, what state has to cross regions and what deliberately does not, and how much headroom each region carries so a failover does not simply move the outage.📐 AI Systems Design
Foundational
Latency Metrics: TTFT, TPOT and GoodputAn LLM request has two latencies, not one: time to first token, set by queueing and prefill, and time per output token, set by the decode loop. Reporting them as percentiles, and reporting goodput (requests that met both SLOs per second) rather than raw throughput, is what separates a serving engineer from a benchmark reader. The numbers a loop expects: about 24 tokens per second single-stream for a 70B model on one H100, TTFT floors in the hundreds of milliseconds for long prompts, and p99s that come from queueing, not from the GPU.🚀 Inference & Serving
Foundational
Control Plane and API Design for GPU PlatformsEvery GPU platform has a control plane, and its API is what the rest of the organization experiences as the platform. Three semantics decide whether it survives contact with a network: idempotent creation so a retried request does not launch a second job on sixty-four GPUs, cancellation modelled as intent because only the node agent can stop a running process, and cursor pagination that does not skip rows when work is created during a listing.📐 AI Systems Design
UP NEXT ON YOUR JOURNEY
Next in this trackYour p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.Next in this trackvLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?Next in this trackHow would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?
FEDITOR'S NOTE
Scored on doing the sequential-versus-batched arithmetic from first principles, on handling variable lengths without padding waste, and on knowing the memory check that bounds the batch.
DISCUSSION · 0
No comments yet — be the first to share your approach.