Your training run is at 60% of the step time you projected. How do you find out whether it is compute, memory, network or I/O?

Compute the step time the formula predicts, measure the one you have, and the gap is the budget to explain. The isolation order with one metric per suspect: dataloader wait, SM and tensor-core activity, HBM throughput, time in collectives, and the per-rank spread that says it is one machine.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

🕸️ Distributed TrainingSign in

MFU and HFUModel FLOPs utilization is the fraction of a GPU's peak that goes into the model's own forward and backward math, computed from 6ND and the step time; hardware FLOPs utilization also counts recomputation. Production LLM training lands at 35 to 45% MFU, and knowing where the other 55% goes is the job.

Foundational

🖧 Hardware & Cluster Build-Out

Cluster Bring-Up: Firmware, Drivers and the StackBring-up is an ordered dependency chain and skipping a step produces a symptom that points somewhere else. Firmware first, then the operating system and kernel, then the GPU driver, then the fabric manager, then the network stack, then GPUDirect, then CUDA and NCCL, then the container and scheduling layer. On Blackwell HGX systems the fabric manager reaches the NVSwitches through a bridge device and therefore depends on the InfiniBand stack being present, which is a dependency that surprises almost everyone the first time.

Advanced

🔌 Networking & Storage🔒 Premium

Data Loading Pipelines for TrainingThe dataloader is the only part of a training job that runs on the CPU, the disk and the network at once, and it is the part most often found starving the GPUs. A pipeline that keeps 1,024 accelerators fed has to read sharded files sequentially, decode and tokenize in parallel workers, prefetch several batches ahead, pin memory for the PCIe copy, and do it deterministically enough to resume mid-epoch. The symptom of failure is a GPU at 30% utilization with nothing wrong on the GPU.

Advanced

⚡ Kernels & Compilers🔒 Premium

Profiling with NsightNsight Systems answers where wall-clock time goes across CPU, kernels and copies; Nsight Compute answers why one kernel is slow, from hardware counters. The skill interviewers test is the order: timeline first, then the Speed of Light section, then the two or three metrics that name the bottleneck, so that a memory-bound kernel is recognized from its profile in under a minute and the fix is bytes, not occupancy.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackTraining loss went flat at step 40k after descending normally. Walk me through how you debug it.Next in this trackAt 16,000 GPUs something fails every few hours. How do you choose the checkpoint interval, and what does the write have to look like?Next in this trackDesign a training system that survives losing a node without a human in the loop. What does elasticity cost you?

FEDITOR'S NOTE

Scored on starting from the projected step time rather than from a profiler, on having one metric per suspect, and on checking the per-rank spread before blaming any subsystem.

DISCUSSION · 0

No comments yet — be the first to share your approach.