Advanced

Serverless GPU Platforms

A serverless GPU platform lets a customer deploy a function or a model and pay only while it runs, so the platform has to start a GPU workload in seconds, pack many customers onto shared hardware without letting them see each other, and keep enough capacity warm that a burst does not wait for a cold start. Each is a design problem with numbers: the cold-start chain and the snapshot that shortens it, bin-packing memory-sized workloads onto fixed-size GPUs, the isolation boundary and its cost, and the economics of idle capacity against cold starts. This page designs the platform and derives the trade-offs.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Containers, Images and GPU Cold Starts Inference Autoscaling and Cold Starts Multi-Tenancy, Quotas and Fair Share MIG, MPS and Time-Slicing Capacity and Backpressure Inference Platform Architecture

PRACTICE THIS IN REAL QUESTIONS

AI Infrastructure System DesignDesign a serverless GPU platform: a customer deploys a function with a model and pays per second. Where does the cold start budget go?→Kubernetes, Slurm & GPU SchedulingDesign a serverless GPU platform where a function that loads a 7B model cold-starts in under a second. Where does every second go today?→Kubernetes, Slurm & GPU SchedulingRun untrusted user code at 50,000 concurrent sessions, some on GPUs. Pick the isolation boundary and defend the density you lose.→LLM Inference & ServingA new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?→AI Infrastructure System DesignDesign autoscaling for an LLM service whose traffic triples within a minute. What signal, what cold-start budget, what warm pool?→LLM Inference & ServingDesign an autoscaler for GPU inference replicas that reacts to load without thrashing.→