After a switch firmware update, all-reduce bandwidth across the cluster fell 20 percent. Every link is up and no errors are logged. Find it.

A uniform loss across every node rules out hardware and points at something that changed everywhere at once. The differential procedure that isolates a reset default from a real regression, the settings firmware resets on its own, and the change process that catches it in an hour.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

Congestion Control for AI FabricsCollective traffic is the worst case a network can see: hundreds of senders transmit to the same receiver at the same instant (incast), every flow is large and long-lived, and RDMA cannot tolerate a dropped packet. Congestion control is the set of mechanisms (PFC, ECN with DCQCN, adaptive routing, packet spraying) that keep queues from overflowing without stalling the fabric. On plain Ethernet a busy all-reduce can fall to about 60% of link rate; with a tuned control loop it holds above 90%. Reading the counters that show which one you have is the on-call skill.

Foundational

🖧 Hardware & Cluster Build-Out

Cluster Bring-Up: Firmware, Drivers and the StackBring-up is an ordered dependency chain and skipping a step produces a symptom that points somewhere else. Firmware first, then the operating system and kernel, then the GPU driver, then the fabric manager, then the network stack, then GPUDirect, then CUDA and NCCL, then the container and scheduling layer. On Blackwell HGX systems the fabric manager reaches the NVSwitches through a bridge device and therefore depends on the InfiniBand stack being present, which is a dependency that surprises almost everyone the first time.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceA training job reports its all-reduce at a third of what the fabric should deliver, every node passed its health check, and nothing is logged. This page is the isolation order that finds the cause in an hour instead of a day: measure the collective in isolation, split the job until the slow pair or rank appears, then check the specific things that make a link, a node or a placement slow. Most cases end at one NIC, one topology mismatch, or GPUDirect silently off.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Closest to this topicOne fabric link goes down and comes back once an hour. What does that do to a 1,024-GPU training job, and how would you find it?Closest to this topicAfter a maintenance window, two-node bandwidth halved. Single-node is unchanged. Diagnose it.Closest to this topicSeveral unrelated jobs slow down at once and nothing is dropping packets. How do you confirm a pause cascade and find where it started?

FEDITOR'S NOTE

Scored on treating uniformity as the key evidence, on differential diagnosis against a held-back node group rather than absolute measurement, on naming the specific settings firmware resets, and on the process fix.

DISCUSSION · 0

No comments yet — be the first to share your approach.