Advanced

Inference Autoscaling and Cold Starts

Inference autoscaling adds or removes model replicas as demand changes. Queue pressure, latency and startup time help determine how much ready capacity to keep.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Latency Metrics: TTFT, TPOT and Goodput Capacity and Backpressure Serverless GPU Platforms Containers, Images and GPU Cold Starts Designing for Latency SLOs Spot, Preemption and Capacity Strategies

LESSONS THAT TEACH THIS

From Model to ServiceScale from evidence and keep the bill bounded17 min→Foundations of AI InfrastructureBound generation and make cancellation observable14 min→Foundations of AI InfrastructureStreaming is where correctness bugs hide13 min→

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingA new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?→LLM Inference & ServingDesign an autoscaler for GPU inference replicas that reacts to load without thrashing.→AI Infrastructure System DesignDesign a serverless GPU platform: a customer deploys a function with a model and pays per second. Where does the cold start budget go?→Kubernetes, Slurm & GPU SchedulingDesign a serverless GPU platform where a function that loads a 7B model cold-starts in under a second. Where does every second go today?→AI Infrastructure System DesignDesign autoscaling for an LLM service whose traffic triples within a minute. What signal, what cold-start budget, what warm pool?→Open-Weights Models & Serving EnginesDesign a platform that serves twenty open-weights models of varying size to internal teams.→