33A node fails burn-in twice and passes on the third attempt. Do you accept it?▼hardNewCoreWeaveMetaMicrosoft4 replies◆ premiumTwo failures and a pass is not a passing node, it is a node with a demonstrated intermittent fault and a small sample. The arithmetic that shows how weak the evidence for acceptance is, what an intermittent fault costs once it is in a training job, and the policy that removes the argument.Open full answer →
21One fabric link goes down and comes back once an hour. What does that do to a 1,024-GPU training job, and how would you find it?▼hardNewMetaCoreWeave4 replies◆ premiumA collective stops when one participant stops, and a job of a thousand ranks has no way to distinguish a link that will come back in two seconds from a rank that has died. What the job actually does during those seconds, the cost per flap, and why the fault passes every test you would run.Open full answer →
