Advanced

Cost per Million Tokens

Cost per million tokens connects fleet spending to useful output. Keep the time window, token definition and utilization assumptions consistent when comparing deployments.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Bandwidth-Bound Decode Throughput Latency Metrics: TTFT, TPOT and Goodput TCO: Buy vs Rent Capacity Planning and Utilization Quantization for Inference Inference Autoscaling and Cold Starts

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureSize the fleet from active work and failure requirements15 min→Inference EngineeringCapstone: a reproducible serving teardown240 min→

PRACTICE THIS IN REAL QUESTIONS

Napkin Math, Cost & CapacityWhat does it cost per million output tokens to serve a 70B model on eight H100s?→Napkin Math, Cost & CapacityWhat does one training token cost?→Napkin Math, Cost & CapacitySize an inference fleet for a 70B model serving 1,000 concurrent users→Napkin Math, Cost & CapacityRank the H100, MI300X and Trainium2 by cost per token for decode→Open-Weights Models & Serving EnginesYour model now serves 6,568 tokens per second. What does a million tokens cost, and what do you hand the gateway and eval teams?→Napkin Math, Cost & CapacityTraffic peaks at three times the daily average. Capacity-plan the serving fleet.→