Advanced

Bandwidth-Bound Decode Throughput

Estimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Memory-Bound vs Compute-Bound Kernels Arithmetic Intensity by Operation KV Cache Sizing Quantization for Inference Cost per Million Tokens Speculative Decoding

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureSize the fleet from active work and failure requirements15 min→

PRACTICE THIS IN REAL QUESTIONS

Napkin Math, Cost & CapacityHow many tokens per second can a 70B model generate for a single user on H100s?→Napkin Math, Cost & CapacitySize an inference fleet for a 70B model serving 1,000 concurrent users→Napkin Math, Cost & CapacityHow much does moving from bf16 to fp8 save in serving cost?→Open-Weights Models & Serving EnginesYour model decodes at a tenth of its bandwidth bound at batch one. Explain the gap.→Napkin Math, Cost & CapacityHow many tokens per second can one B200 decode for a 70B model?→Napkin Math, Cost & CapacityHow big can the decode batch get before the step becomes compute-bound?→