Advanced

Topology-Aware Communication

Map communication groups onto real GPU and network paths. A four-rank example shows how two tensor-parallel streams share one adapter, why per-rank bandwidth can mislead, and when locality is worth waiting for.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

NVLink, NVSwitch and PCIe Rail-Optimized and Fat-Tree Fabrics Tensor Parallelism NCCL and Collective Algorithms Topology-Aware Scheduling Debugging a Slow All-Reduce

LESSONS THAT TEACH THIS

Fleet and ReliabilityPlace communication groups against measured topology14 min→Inference EngineeringSplitting a model across devices buys memory and spends interconnect15 min→Training at ScaleMap communication groups onto measured physical paths24 min→

PRACTICE THIS IN REAL QUESTIONS

Networking, Interconnects & StorageMap a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?→Kubernetes, Slurm & GPU SchedulingTwo identical 8-GPU jobs get 8 GPUs each. One runs at half the speed of the other. What did the scheduler do, and how do you stop it?→GPU & Accelerator ArchitectureNVLink versus PCIe: when does the interconnect actually matter for an LLM workload, and when can you get away with PCIe?→Networking, Interconnects & StorageExplain a rail-optimized fabric. What does it buy over a plain fat tree, and how many switches does one need?→GPU & Accelerator ArchitectureExplain GPUDirect RDMA. What does it remove from the data path, and when does it change performance?→Open-Weights Models & Serving EnginesA multi-node vLLM deployment starts, forms a cluster, and hangs on the first request. Debug it.→