A flat median with a broken tail means one in a hundred requests is hitting something the others do not. There are five usual suspects, and the order you rule them out in is the answer.
more free answers with an account · no card
← LLM Inference & Serving / 14
hard★ EssentialNewBasetenAnthropicOpenAI
Your p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.
A flat median with a broken tail means one in a hundred requests is hitting something the others do not. There are five usual suspects, and the order you rule them out in is the answer.
Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.
The concepts behind this question
Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.
Core
Designing for Latency SLOsA latency objective is met or missed by the sum of a chain of delays, and the way to design for it is to write the chain down with a number on every link, find the links that dominate at the tail, and attack those. For an LLM request the chain is network, gateway, router, queue, prefill, then the decode loop, and the tail is shaped by queueing and by the size of the batch the request lands in. This page decomposes a 500 ms time-to-first-token budget link by link, derives how queueing turns a comfortable median into a broken p99, and gives the design moves (admission control, chunked prefill, priority lanes, hedging) that hold it.📐 AI Systems DesignSign in
Advanced
Capacity and BackpressureA system that accepts more work than it can finish does not degrade gracefully; it degrades completely, because every request it queues makes every other request slower until all of them time out. The defence is backpressure: bounded queues at every stage, admission control that rejects early when the expected wait exceeds the budget, load shedding by priority, and clients that back off. For LLM serving the stages are the gateway, the router, the engine's queue and its KV pool. This page works the arithmetic of why unbounded queues kill p99, designs the bounds per stage, and the client contract that keeps it stable under overload.📐 AI Systems Design🔒 Premium
Foundational
The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.🚀 Inference & Serving
Core
DCGM and GPU TelemetryNVIDIA's Data Center GPU Manager reads a GPU's counters, runs its diagnostics and exports both to the monitoring stack, and nearly every fleet's dashboards and alerts are built on it. The skill is knowing which of its hundreds of fields carry signal: the profiling metrics that say whether the tensor cores are busy (not the utilization number everyone reads first), the error counters that predict a failure, the throttle reasons that explain a slow step, and the diagnostic levels that decide whether a node returns to the pool. This page walks those fields, derives an MFU estimate from them, and gives a fleet's alert thresholds.🩺 Fleet Reliability & ObservabilitySign in
UP NEXT ON YOUR JOURNEY
Next in this trackvLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?Next in this trackHow would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?Next in this trackDesign an autoscaler for GPU inference replicas that reacts to load without thrashing.
FEDITOR'S NOTE
Scored on the isolation order (split by replica, then by request shape, then by engine counters), on naming the engine metric that confirms each cause, and on not guessing at a fix before the data says which one it is.
DISCUSSION · 0
No comments yet — be the first to share your approach.