Construct a 16384-GPU candidate layout from ownership and batch constraints, with explicit sharding recipes and pipeline idle time.
This answer is included with a free account. Sign in to read it and save your progress.
free account · no card needed
← Distributed Training & Parallelism / 15
hard★ EssentialMetaNVIDIA
You have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.
Construct a 16384-GPU candidate layout from ownership and batch constraints, with explicit sharding recipes and pipeline idle time.
Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.
Work through the numbers
Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.
The concepts behind this question
Use these concept explanations to review the ideas behind this question.
Advanced
Tensor ParallelismTensor parallelism splits operations within a layer across GPUs. Derive the column/row matrix split, count training and inference collectives, and choose a group size from memory, latency and actual interconnect performance.🕸️ Distributed Training🔒 Premium
Advanced
Pipeline Parallelism and the BubblePipeline parallelism assigns groups of layers to different devices. Microbatches keep stages busy, while the schedule controls idle periods and activation memory.🕸️ Distributed Training🔒 Premium
Core
ZeRO and FSDPZeRO and FSDP reduce replicated training state by distributing it across workers. The memory saving comes with communication and temporary-buffer costs.🕸️ Distributed TrainingSign in
Foundational
Training Uptime and Interruption StatisticsTraining interruption statistics connect failure frequency to lost progress. Use a stated rate, checkpoint cost and recovery model to estimate how much time a run can lose.🩺 Fleet Reliability & Observability
UP NEXT ON YOUR JOURNEY
Next in this trackExplain mixed-precision training. Why does fp16 need loss scaling and bf16 not, why keep fp32 master weights, and what changes with fp8?Next in this trackYour training run delivers only 60% of the throughput you projected. How do you find out whether it is compute, memory, network or I/O?Next in this trackTraining loss went flat at step 40k after descending normally. Walk me through how you debug it.
FEDITOR'S NOTE
Check the global batch after choosing DP. More replicas can leave too few microbatches to keep a deep pipeline busy.
DISCUSSION · 0
No comments yet — be the first to share your approach.