21One fabric link goes down and comes back once an hour. What does that do to a 1,024-GPU training job, and how would you find it?▼hardNewMetaCoreWeave4 replies◆ premiumA collective stops when one participant stops, and a job of a thousand ranks has no way to distinguish a link that will come back in two seconds from a rank that has died. What the job actually does during those seconds, the cost per flap, and why the fault passes every test you would run.Open full answer →
26Several unrelated jobs slow down at once and nothing is dropping packets. How do you confirm a pause cascade and find where it started?▼hardNewMetaxAI4 replies◆ premiumThe signature is unusual: many jobs degrade together, throughput collapses, and the drop counters stay at zero. Reading pause counters backward along the paths to find the origin, the three conditions that produce one, and the configuration change that stops it happening again.Open full answer →
30After a switch firmware update, all-reduce bandwidth across the cluster fell 20 percent. Every link is up and no errors are logged. Find it.▼expertNewMetaCoreWeave4 replies◆ premiumA uniform loss across every node rules out hardware and points at something that changed everywhere at once. The differential procedure that isolates a reset default from a real regression, the settings firmware resets on its own, and the change process that catches it in an hour.Open full answer →
