Advanced

Quantization for Inference

Quantization stores weights, and sometimes activations and the KV cache, in fewer bits, which cuts the bytes a decode step has to stream and the memory a model occupies. Weight-only int4 (GPTQ, AWQ) is a capacity and single-stream latency play; fp8 for weights and activations (W8A8) doubles tensor-core throughput and helps prefill and large batch; fp8 KV cache doubles context per GPU. Each has an accuracy cost you measure rather than assume, and knowing which one to reach for from the bottleneck is the interview question.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Numerics: FP32, BF16, FP8 and FP4 Memory-Bound vs Compute-Bound Kernels Bandwidth-Bound Decode Throughput Tensor Cores and Matrix Units Model Memory Footprint Serving Engines: vLLM, SGLang and TensorRT-LLM

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureWeights are the floor, not the budget13 min→Training at ScaleMixed precision is a numerics contract, not a speed setting14 min→Inference EngineeringWeights, activations and the cache are three separate precision decisions15 min→

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingYou need to quantize a model for serving. Which method do you pick, and what do you measure before shipping it?→Open-Weights Models & Serving EnginesThe model ships in FP8. Should you requantize to four bits to fit more of it on fewer GPUs?→Open-Weights Models & Serving EnginesYou need a released bf16 model at half the footprint. Walk through quantizing it yourself.→Napkin Math, Cost & CapacityHow much does moving from bf16 to fp8 save in serving cost?→GPU & Accelerator ArchitectureYou are moving a model to fp4 inference on Blackwell. What breaks first, and how would you measure whether the result is acceptable?→Napkin Math, Cost & CapacityHow much GPU memory does it take to run Llama 3.1 70B?→