Advanced

Quantization for Inference

Quantization reduces weight, activation or cache precision. Calculate payload plus scales, estimate whole-step speedup, and test task quality before choosing a smaller representation.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Numerics: FP32, BF16, FP8 and FP4 Memory-Bound vs Compute-Bound Kernels Bandwidth-Bound Decode Throughput Tensor Cores and Matrix Units Model Memory Footprint Serving Engines: vLLM, SGLang and TensorRT-LLM

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureWeights are the floor, not the budget13 min→Training at ScaleMixed precision is a numerics contract, not a speed setting14 min→Inference EngineeringWeights, activations and the cache are three separate precision decisions20 min→

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingYou need to quantize a model for serving. Which method do you pick, and what do you measure before shipping it?→Open-Weights Models & Serving EnginesThe model ships in FP8. Should you requantize to four bits to fit more of it on fewer GPUs?→Open-Weights Models & Serving EnginesYou need a released bf16 model at half the footprint. Walk through quantizing it yourself.→Open-Weights Models & Serving EnginesWhich GPUs can run an FP8 or AWQ checkpoint, and what do you actually get on an older card?→GPU & Accelerator ArchitectureYou are moving a model to fp4 inference on Blackwell. What breaks first, and how would you measure whether the result is acceptable?→Napkin Math, Cost & CapacityHow much does moving from bf16 to fp8 save in serving cost?→