05One GPU's correctable memory error rate has been climbing for a week. What does that predict and what do you do about it?▼mediumNewMetaCoreWeave4 repliesunlockedCorrectable errors are corrected, so nothing is wrong yet, and that is exactly why they are the most useful signal a fleet has. What rising rates predict, the remapping budget that decides whether a device can heal itself, and the policy that converts a trend into a scheduled drain instead of a failed training run.Open full answer →
14What do NVLink errors look like in telemetry, when is a link degrading rather than broken, and when do you drain the node?▼hardNewNVIDIAMeta4 replies○ sign inA link that corrupts occasionally does not fail; it retransmits, and each retransmission costs bandwidth that shows up as a slow job with no error anywhere. The counters that reveal it, the arithmetic turning a replay rate into lost bandwidth, and the threshold where watching becomes draining.Open full answer →
