Separate host-buffer, GPU-buffer and collective tests, with explicit message-size models and bandwidth metric definitions.
This answer is included with a free account. Sign in to read it and save your progress.
free account · no card needed
← Networking, Interconnects & Storage / 16
mediumLambdaCrusoe
How do you establish that the link between two GPU nodes is healthy, and what numbers should each test return?
Separate host-buffer, GPU-buffer and collective tests, with explicit message-size models and bandwidth metric definitions.
Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.
The concepts behind this question
Use these concept explanations to review the ideas behind this question.
Foundational
Burn-In and Acceptance TestingAcceptance testing checks that a new cluster meets agreed performance and reliability criteria. Sustained tests and comparisons between nodes help reveal defects before production.🖧 Hardware & Cluster Build-Out
Foundational
Debugging a Slow All-ReduceInterpret NCCL algbw and busbw, then isolate a slow collective with controlled tests. Distinguish rank delays, GPU-memory paths and shared-fabric limits without guessing from bandwidth ratios.🔌 Networking & Storage
Advanced
Node Health Checks and Burn-InHealth checks look for faults before and during GPU workloads. Burn-in, pre-flight checks and continuous monitoring cover different conditions and need explicit acceptance criteria.🩺 Fleet Reliability & Observability🔒 Premium
Foundational
NCCL and Collective AlgorithmsNCCL implements GPU collectives using algorithms, protocols and channels over available hardware paths. Learn what each choice changes, how to read bandwidth results, and which evidence separates a communication problem from late application work.🔌 Networking & Storage
UP NEXT ON YOUR JOURNEY
Next in this trackWhy do large training clusters provision roughly 400 gigabits per second per GPU rather than more or less?Next in this trackWhat is Ultra Ethernet trying to fix, and does it change how you would choose a fabric today?Next in this trackDesign the complete network for a 1,024-GPU cluster: compute fabric, storage, management. What are the counts and where does the money go?
FEDITOR'S NOTE
Add host, GPU and collective dependencies one test at a time. Define the bandwidth units, sweep relevant message sizes and compare matched baselines.
DISCUSSION · 0
No comments yet — be the first to share your approach.