Advanced

Stragglers and Hangs

Trace how a late worker reaches the critical path, distinguish longer steps from lower throughput, and separate ordinary jitter from a stalled job. Worked examples show when delay is hidden, how shared jitter changes the math, and what evidence supports a repair.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

NVLink and Fabric Faults Thermal, Power and Cooling Events Collective Communication Primitives Debugging a Slow All-Reduce DCGM and GPU Telemetry Elastic and Fault-Tolerant Training

LESSONS THAT TEACH THIS

Training at ScaleMFU connects token throughput to an explicit compute model20 min→Training at ScaleFind the late dependency before blaming the rank that waits22 min→Fleet and ReliabilityDesign health gates with coverage, false positives and clear ownership14 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismYou see 'Watchdog caught collective operation timeout' in the logs. What happened, and how do you find which rank caused it?→Distributed Training & ParallelismA 16,384-GPU training job’s steps grow from 6.0 to 6.9 seconds. How would you isolate and fix the slowdown?→GPU Fleet Reliability & ObservabilityA training run hangs every few hours with no error, and the GPUs sit idle until the timeout fires. Find the cause.→Coding for InfraA multi-GPU training job hangs at step 400 with every GPU at 100 percent utilization. Debug it.→GPU Fleet Reliability & ObservabilityDesign observability for a large training cluster. What do you collect, what does each signal answer, and what pages someone?→GPU Fleet Reliability & ObservabilityWalk me through reading a collective flight-recorder dump. What is in it, and how do you find the rank that caused a hang?→