← GPU Fleet Reliability & Observability / 20

hardMetaNVIDIA

Walk me through reading a collective flight-recorder dump. What is in it, and how do you find the rank that caused a hang?

Compare the same collective across the ranks that should participate. Missing records, mismatched arguments, and unfinished operations lead to different next checks.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Compare the same collective across the ranks that should participate. Missing records, mismatched arguments, and unfinished operations lead to different next checks.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🩺 Fleet Reliability & Observability🔒 Premium

Stragglers and HangsTrace how a late worker reaches the critical path, distinguish longer steps from lower throughput, and separate ordinary jitter from a stalled job. Worked examples show when delay is hidden, how shared jitter changes the math, and what evidence supports a repair.

🕸️ Distributed Training

Collective Communication PrimitivesFollow small vectors through all-reduce, reduce-scatter, all-gather and all-to-all. Separate the result each worker needs from the algorithm-specific byte count and measured completion time.

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL implements GPU collectives using algorithms, protocols and channels over available hardware paths. Learn what each choice changes, how to read bandwidth results, and which evidence separates a communication problem from late application work.

🖧 Hardware & Cluster Build-Out

NVLink Domains and the NVL72 RackAn NVLink domain connects a group of GPUs through a scale-up fabric. Its boundary helps determine where to place communication-heavy parallel groups.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackThe facility asks you to cap GPU power by 30 percent for the summer. What does that cost a training run, and how would you do it?Next in this trackA GPU computes the wrong answer and reports no error. How would you detect that, and what does it look like in a training run?Next in this trackDesign the alerting rules for a GPU platform so that a page is always worth waking for. What are the mechanics?

FEDITOR'S NOTE

Look for process-group alignment, incomplete-dump handling, and a distinction between evidence of a stalled collective and proof of a network fault.

DISCUSSION · 0

No comments yet — be the first to share your approach.