Core

Designing for Latency SLOs

A latency objective is met or missed by the sum of a chain of delays, and the way to design for it is to write the chain down with a number on every link, find the links that dominate at the tail, and attack those. For an LLM request the chain is network, gateway, router, queue, prefill, then the decode loop, and the tail is shaped by queueing and by the size of the batch the request lands in. This page decomposes a 500 ms time-to-first-token budget link by link, derives how queueing turns a comfortable median into a broken p99, and gives the design moves (admission control, chunked prefill, priority lanes, hedging) that hold it.

a free account unlocks the core curriculum tier · no card

RELATED CONCEPTS

SLOs for AI Systems Latency Metrics: TTFT, TPOT and Goodput Chunked Prefill Capacity and Backpressure Continuous Batching Request Routing and Load Balancing for LLMs

LESSONS THAT TEACH THIS

Inference EngineeringBenchmarks lie unless the length distribution is real14 min→Foundations of AI InfrastructureLatency has three numbers and reporting one of them hides the problem13 min→Foundations of AI InfrastructureYou can size a serving fleet on a napkin, and you will be asked to15 min→

PRACTICE THIS IN REAL QUESTIONS

AI Infrastructure System DesignDesign an LLM service for a 200 ms time-to-first-token SLO at p99. Decompose the budget and say what you would give up.→Open-Weights Models & Serving EnginesYour p99 time to first token is four times p50. Find out why.→LLM Inference & ServingA long prompt arrives while sixty users are mid-generation. What happens, and how does chunked prefill fix it?→LLM Inference & ServingYour p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.→AI Infrastructure System DesignDesign an LLM batching system end to end: the queue, the batch, the KV cache and streaming. Give me numbers.→LLM Inference & ServingDefine TTFT, TPOT and goodput, and tell me how you would measure each one in production.→