Your training run delivers only 60% of the throughput you projected. How do you find out whether it is compute, memory, network or I/O?

Use per-rank arrival times and controlled substitutions to explain a slow step without mistaking collective wait for network transfer.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Use per-rank arrival times and controlled substitutions to explain a slow step without mistaking collective wait for network transfer.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

GPUs and time-to-train calculator

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Core

🕸️ Distributed TrainingSign in

MFU and HFUMeasure useful model work, separate recomputation, and align token counts, timing and hardware peaks. Learn what activity counters can and cannot tell you.

Foundational

🖧 Hardware & Cluster Build-Out

Cluster Bring-Up: Firmware, Drivers and the StackBring-up validates the dependencies between firmware, drivers, networking and the runtime. Checking each layer makes faults easier to locate.

Advanced

🔌 Networking & Storage🔒 Premium

Data Loading Pipelines for TrainingTrace batches from storage to GPU readiness, budget workers and prefetch memory, and resume from the input position saved with training state. An executable example shows how a read-ahead cursor can silently skip samples.

Advanced

⚡ Kernels & Compilers🔒 Premium

CUDA Compilation: PTX, Cubins and SASSCUDA source, PTX and native GPU instructions describe different stages of execution. Inspect the selected binary and its resource use before explaining a performance change.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackTraining loss went flat at step 40k after descending normally. Walk me through how you debug it.Next in this trackAt 16,000 GPUs something fails every few hours. How do you choose the checkpoint interval, and what does the write have to look like?Next in this trackDesign a training system that survives losing a node without a human in the loop. What does elasticity cost you?

FEDITOR'S NOTE

Compare readiness and completion times across ranks. A long collective may be waiting for late compute or input rather than a slow link.

DISCUSSION · 0

No comments yet — be the first to share your approach.