Advanced

Capacity Planning and Utilization

Capacity planning connects demand, useful throughput and acceptable waiting time. Serving and training need different models of bursts, job shape and spare capacity.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Cost per Million Tokens TCO: Buy vs Rent Inference Autoscaling and Cold Starts GPU Job Scheduler Design Multi-Tenancy, Quotas and Fair Share Latency Metrics: TTFT, TPOT and Goodput

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureWeights are the floor, not the budget13 min→Foundations of AI InfrastructureSize the fleet from active work and failure requirements15 min→Foundations of AI InfrastructureCapstone: test the assumptions in a service capacity plan270 min→

PRACTICE THIS IN REAL QUESTIONS

Napkin Math, Cost & CapacityTraffic peaks at three times the daily average. Capacity-plan the serving fleet.→Napkin Math, Cost & CapacityModel our serving request queue with Little's law. What happens as we approach saturation?→Napkin Math, Cost & CapacitySize an inference fleet for a 70B model serving 1,000 concurrent users→Napkin Math, Cost & CapacityWhat does it cost per million output tokens to serve a 70B model on eight H100s?→Open-Weights Models & Serving EnginesYour product forecasts 12,800 output tokens per second at peak. Size the fleet.→Kubernetes, Slurm & GPU SchedulingWe rent GPUs. When should we buy committed capacity instead of paying on demand, and what do we do with the rest of the demand?→