Advanced

Capacity and Backpressure

A system that accepts more work than it can finish does not degrade gracefully; it degrades completely, because every request it queues makes every other request slower until all of them time out. The defence is backpressure: bounded queues at every stage, admission control that rejects early when the expected wait exceeds the budget, load shedding by priority, and clients that back off. For LLM serving the stages are the gateway, the router, the engine's queue and its KV pool. This page works the arithmetic of why unbounded queues kill p99, designs the bounds per stage, and the client contract that keeps it stable under overload.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Designing for Latency SLOs Inference Platform Architecture Continuous Batching Rate-Limiting Algorithms Retry, Backoff and Idempotency Inference Autoscaling and Cold Starts

LESSONS THAT TEACH THIS

Inference EngineeringA serving engine is a scheduler with a memory allocator attached13 min→Inference EngineeringBenchmarks lie unless the length distribution is real14 min→Foundations of AI InfrastructureLatency has three numbers and reporting one of them hides the problem13 min→

PRACTICE THIS IN REAL QUESTIONS

AI Infrastructure System DesignDesign an LLM service for a 200 ms time-to-first-token SLO at p99. Decompose the budget and say what you would give up.→AI Infrastructure System DesignDesign rate limiting for an LLM API. Why tokens instead of requests, and how does a bucket work when the cost is unknown until the end?→AI Infrastructure System DesignDesign an LLM batching system end to end: the queue, the batch, the KV cache and streaming. Give me numbers.→AI Infrastructure System DesignDesign a distributed search system with an LLM answer layer at 10,000 queries per second. Size both tiers and name the SLOs.→LLM Inference & ServingYour p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.→Coding for InfraWrite a producer-consumer pipeline with a bounded queue. What are the three bugs that show up in every first attempt?→