31Your 4,096-GPU run loses about 2% of every day to restarts. Fix it, and tell me where the floor is.▼expertMetaAnthropicxAI4 replies◆ premiumTurn the daily loss into recovery and replay costs, then identify which part is measured and which is assumed. Compare checkpoint cadence, recovery work and failure prevention without promising one universal floor.Open full answer →
26Several unrelated jobs slow down at once and nothing is dropping packets. How do you confirm a pause cascade and find where it started?▼hardMetaxAI4 replies◆ premiumFollow pause requests against the direction of data, estimate queue growth and distinguish cascade evidence from its possible causes.Open full answer →
28How would you know GPUDirect RDMA is not being used, given that nothing reports an error when it is disabled?▼mediumNVIDIALambda4 replies◆ premiumDistinguish GPU-memory capability, registration and selected transport. Compare direct and staged paths without relying on a half-speed rule or one log substring.Open full answer →
18Error rate on an inference fleet tripled ten minutes after a deploy. What do you do first, and what should have caught it?▼mediumBasetenOpenAI4 replies○ sign inStop rollout expansion, preserve evidence and use the safest available mitigation. Compare versions under representative load, and size canary exposure from the regression you need to detect rather than a fixed traffic percentage.Open full answer →
22A GPU computes the wrong answer and reports no error. How would you detect that, and what does it look like in a training run?▼expertMetaGoogle4 replies◆ premiumA successful operation can still produce a wrong value. Explain what comparisons can detect, what they miss, and how to investigate without mistaking ordinary numerical variation for corruption.Open full answer →
21Given per-rank heartbeats from a training job, detect which rank has stopped and when. What produces false positives?▼mediumMetaAnthropic4 replies◆ premiumDetect missing and silent ranks from a roster, distinguish liveness from progress, and derive thresholds without assuming independent delays.Open full answer →