← AI Infrastructure System Design / 12

mediumBasetenModal

Design autoscaling for an LLM service whose traffic triples within a minute. What signal, what cold-start budget, what warm pool?

A 60-second demand ramp meets a 70-second startup delay. Compare the rate and ready-capacity curves, then design a controller that tracks pending launches without a blind cooldown.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

A 60-second demand ramp meets a 70-second startup delay. Compare the rate and ready-capacity curves, then design a controller that tracks pending launches without a blind cooldown.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🚀 Inference & Serving🔒 Premium

Inference Autoscaling and Cold StartsInference autoscaling adds or removes model replicas as demand changes. Queue pressure, latency and startup time help determine how much ready capacity to keep.

📐 AI Systems Design🔒 Premium

Serverless GPU PlatformsA serverless GPU platform hides allocation and startup from its users. The design balances cold-start latency, tenant isolation and the cost of idle capacity.

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency budget follows a request through networking, queueing and model execution. Use measurements to find which stage causes slow responses under load.

📐 AI Systems Design🔒 Premium

Model Routing and CascadesChoose a model against measured task quality, latency and cost. Distinguish routing before generation from cascades that pay for a first answer before escalating.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign a KV cache tier across GPU memory, host memory and NVMe so prefixes survive across sessions. Bandwidths, and when it pays.Next in this trackDesign the registry and deploy pipeline for model weights: 141 GB artifacts, versioning, distribution to a thousand nodes, integrity.Next in this trackDesign rate limiting for an LLM API. Why tokens instead of requests, and how does a bucket work when the cost is unknown until the end?

FEDITOR'S NOTE

Scored on explicit capacity assumptions, correct state and identity boundaries, failure handling and a justified design choice.

DISCUSSION · 0

No comments yet — be the first to share your approach.