13How would you detect that GPUs are thermally throttling, and what is the right response when they are?▼mediumNewCoreWeaveCrusoe4 replies○ sign inThrottling is the hardware protecting itself, so it produces no error and no failure, only a job that is quietly slower. The two telemetry fields that name it, the arithmetic linking clock to throughput, and why the response differs completely depending on whether the cause is one node or the room.Open full answer →
21The facility asks you to cap GPU power by 30 percent for the summer. What does that cost a training run, and how would you do it?▼hardNewMicrosoftCrusoe4 replies◆ premiumPower rises faster than clock speed does, so giving up thirty percent of the power costs closer to eleven percent of the throughput. The relationship behind that, the uniformity requirement that matters more than the level, and why an uncoordinated cap is worse than a deeper coordinated one.Open full answer →
21Given per-rank heartbeats from a training job, detect which rank has stopped and when. What produces false positives?▼mediumNewMetaAnthropic4 replies◆ premiumA few lines of comparison, and every difficulty is in the threshold and the clock. Why the collector's receive time rather than the sender's, the boundary case that decides whether an exactly-late rank is reported, and the arithmetic that turns a false-positive rate into a threshold.Open full answer →
