expert★ EssentialMetaAnthropicOpenAI

A multi-GPU training job hangs at step 400 with every GPU at 100 percent utilization. Debug it.

Check useful progress before interpreting GPU activity. Run a bounded trace exercise that exposes a rank-conditional call, equal-count ordering errors and matching shapes with the wrong tensor meaning.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Stragglers and HangsTrace how a late worker reaches the critical path, distinguish longer steps from lower throughput, and separate ordinary jitter from a stalled job. Worked examples show when delay is hidden, how shared jitter changes the math, and what evidence supports a repair.

Foundational

🕸️ Distributed Training

Collective Communication PrimitivesFollow small vectors through all-reduce, reduce-scatter, all-gather and all-to-all. Separate the result each worker needs from the algorithm-specific byte count and measured completion time.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL implements GPU collectives using algorithms, protocols and channels over available hardware paths. Learn what each choice changes, how to read bandwidth results, and which evidence separates a communication problem from late application work.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceInterpret NCCL algbw and busbw, then isolate a slow collective with controlled tests. Distinguish rank delays, GPU-memory paths and shared-fabric limits without guessing from bandwidth ratios.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackImplement byte-pair encoding merges, then explain why token counts differ across languages Next in this trackImplement beam search without losing finished hypotheses or corrupting KV ownership Next in this trackImplement a LoRA forward pass and merge two adapters without inventing cross terms

FEDITOR'S NOTE

Look for scoped sequence evidence, compatible operations and semantics, a participation-aware repair, and regression tests that exercise the failing branch rather than just counting calls.

DISCUSSION · 0

No comments yet — be the first to share your approach.