Advanced

Capacity and Backpressure

Backpressure limits accepted work when a service cannot keep up. Bounded queues and clear retry rules prevent overload from spreading through the request path.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Designing for Latency SLOs Inference Platform Architecture Continuous Batching Rate-Limiting Algorithms Retry, Backoff and Idempotency Inference Autoscaling and Cold Starts

LESSONS THAT TEACH THIS

From Model to ServiceRead the client clock before tuning the server12 min→Inference EngineeringFollow two requests through one loaded model21 min→Inference EngineeringA serving benchmark needs a workload and an arrival model17 min→

PRACTICE THIS IN REAL QUESTIONS

AI Infrastructure System DesignDesign an LLM service for a 200 ms time-to-first-token SLO at p99. Decompose the budget and say what you would give up.→AI Infrastructure System DesignDesign rate limiting for an LLM API. Why tokens instead of requests, and how does a bucket work when the cost is unknown until the end?→AI Infrastructure System DesignReview this eight-GPU inference design: what must change before it can meet its deadline?→AI Infrastructure System DesignDesign an LLM batching system end to end: the queue, the batch, the KV cache and streaming. Give me numbers.→AI Infrastructure System DesignDesign a distributed search system with an LLM answer layer at 10,000 queries per second. Size both tiers and name the SLOs.→LLM Inference & ServingYour p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.→