You see 'Watchdog caught collective operation timeout' in the logs. What happened, and how do you find which rank caused it?

Read a watchdog timeout without blaming the reporting rank. Compare group-scoped traces, separate missing evidence from mismatched calls, and distinguish CPU submission from GPU completion.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Read a watchdog timeout without blaming the reporting rank. Compare group-scoped traces, separate missing evidence from mismatched calls, and distinguish CPU submission from GPU completion.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

GPUs and time-to-train calculator

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Stragglers and HangsTrace how a late worker reaches the critical path, distinguish longer steps from lower throughput, and separate ordinary jitter from a stalled job. Worked examples show when delay is hidden, how shared jitter changes the math, and what evidence supports a repair.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceInterpret NCCL algbw and busbw, then isolate a slow collective with controlled tests. Distinguish rank delays, GPU-memory paths and shared-fabric limits without guessing from bandwidth ratios.

Advanced

💻 Coding for Infra🔒 Premium

Batching Queues and BackpressureA request batcher groups work to reduce processing overhead. Queue bounds, dispatch timing and failure handling keep that efficiency from becoming uncontrolled waiting.

Foundational

🕸️ Distributed Training

Collective Communication PrimitivesFollow small vectors through all-reduce, reduce-scatter, all-gather and all-to-all. Separate the result each worker needs from the algorithm-specific byte count and measured completion time.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow do you overlap communication with compute in training, and where does the overlap stop working?Next in this trackHow is training on TPUs with JAX different from training on GPUs with PyTorch? What do you stop doing by hand?Next in this trackFine-tune a 70B on one 80 GB card. What do NF4 and double quantization actually buy, and where does DoRA change the arithmetic?

FEDITOR'S NOTE

Compare matching groups and job attempts, then separate submission, GPU readiness and completion. Preserve missing-evidence uncertainty and distinguish the two timeout mechanisms.

DISCUSSION · 0

No comments yet — be the first to share your approach.