Construct concrete TP and DP groups, then verify physical devices and estimate traffic with per-rank rather than whole-model bytes.
This answer is included with a free account. Sign in to read it and save your progress.
free account · no card needed
← Networking, Interconnects & Storage / 14
hardNVIDIAMeta
Map a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?
Construct concrete TP and DP groups, then verify physical devices and estimate traffic with per-rank rather than whole-model bytes.
Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.
The concepts behind this question
Use these concept explanations to review the ideas behind this question.
Advanced
Topology-Aware CommunicationMap communication groups onto real GPU and network paths. A four-rank example shows how two tensor-parallel streams share one adapter, why per-rank bandwidth can mislead, and when locality is worth waiting for.🔌 Networking & Storage🔒 Premium
Advanced
NVLink, NVSwitch and PCIeNVLink, NVSwitch and PCIe provide different GPU communication paths. Learn how their topology and bandwidth affect the placement of parallel work.🧩 GPU & Accelerator Architecture🔒 Premium
Advanced
Communication Volume EstimatesEstimate communication by naming the tensor, collective, group and synchronization schedule. Work through per-rank ring bytes, conditional sharding costs, activation exchanges and routing imbalance before converting traffic into exposed time.🧮 Napkin Math & Capacity🔒 Premium
Foundational
Multi-Node Serving TopologiesChoose what to distribute: complete replicas, tensor operations, layers, experts or request phases. Compare rank-local fit, communication dependencies, workload concurrency and failure domains. A node boundary is not a universal performance boundary, and multiple layouts can be combined.🧮 Open Weights & Serving Engines
UP NEXT ON YOUR JOURNEY
Next in this trackWhich NCCL environment variables are worth knowing, what does each change, and which should you almost never set?Next in this trackHow do you establish that the link between two GPU nodes is healthy, and what numbers should each test return?Next in this trackWhy do large training clusters provision roughly 400 gigabits per second per GPU rather than more or less?
FEDITOR'S NOTE
List actual TP and DP group members and their devices. Calculate each rank’s traffic before estimating how placement changes complete-step time.
DISCUSSION · 0
No comments yet — be the first to share your approach.