nccl tests

Compare the same benchmark on a healthy node, then follow the evidence through NCCL transport choices, device topology and fabric health. Half the measured bandwidth is a symptom, not a unique diagnosis.

A bandwidth regression narrows the search only after the experiment is defined. Work through rank-count normalization, controlled path tests and the difference between collective time and training progress.

Separate host-buffer, GPU-buffer and collective tests, with explicit message-size models and bandwidth metric definitions.

Use a short admission check for recent changes and a deeper acceptance suite for new or repaired nodes. Set thresholds from matched healthy baselines, and make quarantine and return-to-service decisions explicit.

Build an acceptance suite around the node’s intended workload and supported hardware limits. Combine configuration, correctness, communication and sustained-load evidence, then record a baseline and an explicit return-to-service decision.