Separate gradient range from update precision using a small numerical example, and make the training-state recipe explicit.
This answer is included with a free account. Sign in to read it and save your progress.
free account · no card needed
← Distributed Training & Parallelism / 16
mediumNVIDIAMeta
Explain mixed-precision training. Why does fp16 need loss scaling and bf16 not, why keep fp32 master weights, and what changes with fp8?
Separate gradient range from update precision using a small numerical example, and make the training-state recipe explicit.
Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.
Work through the numbers
Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.
The concepts behind this question
Use these concept explanations to review the ideas behind this question.
Core
Numerics: FP32, BF16, FP8 and FP4Floating-point formats trade range and precision for storage and compute cost. Understand exponent bits, significand bits, scaling and accumulation before choosing a format.🧩 GPU & Accelerator ArchitectureSign in
Foundational
Weight Formats: FP8 Blocks, MXFP4 and AWQDerive the storage cost of low-precision tensors, including scales, zero points and higher-precision exceptions. Distinguish formats from quantization methods, understand fused dequantization, and test whether a smaller checkpoint actually improves serving capacity without unacceptable quality loss.🧮 Open Weights & Serving Engines
Advanced
Quantization for InferenceQuantization reduces weight, activation or cache precision. Calculate payload plus scales, estimate whole-step speedup, and test task quality before choosing a smaller representation.🚀 Inference & Serving🔒 Premium
Foundational
Data Parallelism and DDPReplicas combine gradients from different data. Follow a two-worker update, fix unequal-token weighting, and see why queued gradient buckets can leave more than one reduction exposed.🕸️ Distributed Training
UP NEXT ON YOUR JOURNEY
Next in this trackYour training run delivers only 60% of the throughput you projected. How do you find out whether it is compute, memory, network or I/O?Next in this trackTraining loss went flat at step 40k after descending normally. Walk me through how you debug it.Next in this trackAt 16,000 GPUs something fails every few hours. How do you choose the checkpoint interval, and what does the write have to look like?
FEDITOR'S NOTE
Separate numerical range from the precision of small updates. Trace scaling and unscaling, then identify actual parameter, gradient and optimizer copies.
DISCUSSION · 0
No comments yet — be the first to share your approach.