NVLink Domains and the NVL72 Rack

TL;DR: An HGX node and an NVL72 rack offer different-sized NVLink domains. Keeping frequent collective communication inside a domain can reduce time spent on the scale-out network. Choose placement from the model’s actual communication pattern, and distinguish aggregate rack specifications from the bandwidth available to one GPU.

What a domain is, physically

An NVLink domain is the set of GPUs that can reach each other directly over NVLink, the fast GPU-to-GPU interconnect, without going through the network. Inside an HGX node (NVIDIA's eight-GPU server baseboard design), NVSwitch chips on the baseboard connect eight GPUs so that any GPU can access other GPUs through the switched fabric. Effective bandwidth depends on the access pattern and implementation. NVLink, NVSwitch and PCIe covers the switch itself. What NVL72 does is extend the same fabric past the chassis: the switch trays and the compute trays sit in one rack and are joined by an integrated backplane of passive copper cable cartridges, often called the NVLink spine, rather than by optical links between servers. NVIDIA's DGX GB hardware guide describes this "NVLink passive copper cable backplane". Copper is the choice because at these rates optics would add both cost and a large amount of power, and the distances inside one rack are short enough that copper carries the signal.

The bandwidth cliff, as arithmetic

Compare one-way send capacity with one-way send capacity. The advertised 1.8 TB/s per GPU for fifth-generation NVLink is a bidirectional aggregate; a nominal 800 Gb/s network path is 100 GB/s in one direction before overhead. Those figures do not imply an eighteenfold collective speedup.

For an explicitly hypothetical collective, suppose measured effective send rates are 600 GB/s inside a domain and 80 GB/s across the selected network path:

ring all-reduce: 140 GB buffer, 72 ranks
bytes sent per rank = 2 x (71/72) x 140 GB = 276.1 GB
bandwidth-only time at 600 GB/s = 0.460 s
bandwidth-only time at 80 GB/s  = 3.451 s
sanity: the 7.5x time ratio comes from the assumed effective rates, not the domain name.

This model omits startup and synchronization, and a real collective may use another algorithm. See Collective Communication for the byte-count convention and NVLink, NVSwitch and PCIe for the physical paths.

The rule that falls out

Place frequent, latency-sensitive communication on fast paths when the full workload benefits. Tensor parallelism commonly exchanges activations within every layer; expert parallelism dispatches tokens to selected experts and combines their results. Both can benefit from a large scale-up domain.

Pipeline parallelism transfers activations across stage boundaries. Data parallelism synchronizes gradients according to the accumulation schedule. They still need bandwidth, but their communication pattern may be easier to place across domains. The useful boundary follows the model and schedule rather than a universal maximum TP or EP degree.

For a model with 256 routed experts, an EP group of 64 could place four experts per rank inside a 72-GPU domain. That establishes a possible placement, not that it is the fastest or cheapest one. Measure local expert work, exchange time and imbalance before making the choice.

What this does to scheduling

A domain is also a scheduling unit, and that is easy to miss. Topology-Aware Scheduling covers the general problem; the NVL72-specific version is that a job wanting 72 GPUs either gets a whole rack or it does not, and a job wanting 80 is in two domains with the boundary somewhere inside its tensor-parallel group unless the scheduler knows better. Three consequences:

• Allocation granularity effectively becomes the rack for large jobs, which raises the cost of fragmentation.
• A single failed GPU can make an entire 72-GPU domain unusable for a job that needs all of it, so spare capacity is planned per domain rather than per GPU.
• Gang scheduling has to understand the domain, not just the node count, or it will place a gang that fits numerically and performs badly.

What interviewers are listening for

Describe the parallel groups your workload needs and where they sit in the topology. Explain the consequence of crossing a domain using measured or clearly scoped bandwidth and latency figures. Then discuss scheduling: a failed GPU may prevent a job that requires the entire domain from running, while smaller jobs may still use healthy capacity.

Key takeaways

• An NVLink domain is a connected scale-up group, such as an eight-GPU HGX node or a 72-GPU NVL72 system.
• Aggregate memory bandwidth across a rack is not the remote-memory bandwidth available to one GPU.
• Compare interconnect specifications using the same direction and aggregation conventions.
• Tensor and expert parallelism often benefit from staying within a domain; pipeline and data parallelism still communicate and need a suitable scale-out fabric.
• Plan placement and spare capacity around the size of the jobs that must remain runnable.