← 🚀 Inference & Serving
Advanced
Inference Autoscaling and Cold Starts
Scaling an LLM fleet is harder than scaling a web service because a replica takes minutes to become useful (pull an image, load 141 GB of weights, warm the cache) and costs several dollars an hour while idle. The signals that work are queue depth and TTFT against the SLO, not GPU utilization, which is misleading for memory-bound decode. The design is a warm pool sized for the burst, hysteresis so the fleet does not thrash, and a cold-start path measured in seconds through snapshots and weight streaming.
Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew
RELATED CONCEPTS
LESSONS THAT TEACH THIS
PRACTICE THIS IN REAL QUESTIONS
LLM Inference & ServingA new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?→LLM Inference & ServingDesign an autoscaler for GPU inference replicas that reacts to load without thrashing.→AI Infrastructure System DesignDesign a serverless GPU platform: a customer deploys a function with a model and pays per second. Where does the cold start budget go?→Kubernetes, Slurm & GPU SchedulingDesign a serverless GPU platform where a function that loads a 7B model cold-starts in under a second. Where does every second go today?→AI Infrastructure System DesignDesign autoscaling for an LLM service whose traffic triples within a minute. What signal, what cold-start budget, what warm pool?→Open-Weights Models & Serving EnginesDesign a platform that serves twenty open-weights models of varying size to internal teams.→