NVLink Domains and the NVL72 Rack
An NVLink domain is the set of GPUs that can address each other's memory at full fabric speed, and its size is the single most consequential number in a cluster design. Eight on an HGX node, 72 on a GB300 NVL72 rack. Inside the domain a collective moves at terabytes per second over a copper backplane; outside it, the same collective drops to the scale-out fabric at 800 Gb/s per GPU, a gap of roughly twenty times that decides how models are sharded.
TL;DR: The NVLink domain is the boundary between two bandwidth regimes and everything about sharding follows from where that boundary sits. On an HGX node the domain is eight GPUs at 1.8 TB/s each on Blackwell. On GB300 NVL72, NVIDIA's published figures give 72 Blackwell Ultra GPUs and 36 Grace CPUs in one domain with 130 TB/s of aggregate NVLink bandwidth, 20 TB of GPU memory reachable at up to 576 TB/s, and 800 Gb/s of scale-out networking per GPU from two ConnectX-8 devices. Crossing the domain drops a collective from 1.8 TB/s to 0.1 TB/s per GPU, about eighteen times, so the rule that falls out is simple: put the parallelism with the heaviest per-token traffic inside the domain. That means tensor parallelism and expert parallelism inside, pipeline and data parallelism across.
What a domain is, physically
Inside an HGX node, NVSwitch chips on the baseboard connect eight GPUs so that any GPU can read any other's memory at the link rate. NVLink, NVSwitch and PCIe covers the switch itself. What NVL72 does is extend the same fabric past the chassis: the switch trays and the compute trays sit in one rack and are joined by a copper backplane, often called the NVLink spine, rather than by cables running between servers. Copper is the choice because at these rates optics would add both cost and a large amount of power, and the distances inside one rack are short enough that copper carries the signal.
The bandwidth cliff, as arithmetic
per-GPU bandwidth on each side of the boundary
inside the domain, Blackwell NVLink: 1.8 TB/s = 1,800 GB/s
leaving the domain, 2x ConnectX-8: 800 Gb/s = 100 GB/s
ratio 1,800 / 100 = 18x
what that costs a collective
an all-reduce of S bytes over N ranks moves 2(N-1)/N x S bytes per rank
a 70B model's gradient all-reduce, bf16, S = 140 GB, N = 72:
per-rank bytes = 2 x (71/72) x 140e9 = 276 GB
inside one domain at 1,800 GB/s: 276 / 1800 = 0.15 s
across the fabric at 100 GB/s: 276 / 100 = 2.76 s
sanity: the same collective takes 18 times longer the moment it crosses the boundary, which
is why domain-aware placement is worth more than almost any kernel optimization
The rule that falls out
rank the parallelism dimensions by bytes moved per token, heaviest first
tensor parallelism two all-reduces of one hidden vector per layer per token
70B, hidden 8,192, 80 layers: about 4.6 MB per token per GPU at TP=8
expert parallelism an all-to-all of the routed tokens, twice per MoE layer
heavy and latency-sensitive, and it grows with top-k
pipeline parallelism one activation tensor per micro-batch boundary, not per token
data parallelism one gradient all-reduce per step, not per token
so the placement rule
tensor parallel and expert parallel INSIDE the NVLink domain
pipeline and data parallel ACROSS the scale-out fabric
and the domain size sets the maximum useful TP or EP degree
HGX node: TP up to 8
NVL72: TP or EP up to 72, which is what makes it interesting for large MoE models
sanity: a design that puts tensor parallelism across nodes on an HGX fleet has moved the
heaviest traffic onto the slowest link, and it is the most common serious mistake in
this area
Expert Parallelism for MoE is where the NVL72 domain earns its price. A mixture-of-experts model with 256 experts and 8 active per token wants its experts spread across many GPUs, and the routing all-to-all is exactly the traffic that suffers most when it crosses a fabric. A domain of 72 lets an EP degree of 64 sit entirely inside copper.
What this does to scheduling
A domain is also a scheduling unit, and that is easy to miss. Topology-Aware Scheduling covers the general problem; the NVL72-specific version is that a job wanting 72 GPUs either gets a whole rack or it does not, and a job wanting 80 is in two domains with the boundary somewhere inside its tensor-parallel group unless the scheduler knows better. Three consequences:
- Allocation granularity effectively becomes the rack for large jobs, which raises the cost of fragmentation.
- A single failed GPU can make an entire 72-GPU domain unusable for a job that needs all of it, so spare capacity is planned per domain rather than per GPU.
- Gang scheduling has to understand the domain, not just the node count, or it will place a gang that fits numerically and performs badly.
What interviewers are listening for
The number and its consequence, in that order. Saying "NVL72 has 72 GPUs" is a fact anyone can read; saying "the domain is 72, so we put TP and EP inside it and pipeline across, and the crossing penalty is about eighteen times per GPU" is a design position. Interviewers also listen for the scheduling implication, because that is where people who have run these systems differ from people who have specified them. If you can add that a partial rack failure removes a whole domain from the pool of things that can run the largest job, you have said the thing that matters operationally.
Key takeaways
- A domain is the set of GPUs reachable at full NVLink speed: 8 on an HGX node, 72 on GB300 NVL72.
- NVIDIA publishes 130 TB/s of NVLink and 20 TB of GPU memory at up to 576 TB/s for one NVL72, with 800 Gb/s of scale-out per GPU.
- Leaving the domain drops per-GPU bandwidth from 1,800 GB/s to about 100 GB/s, roughly eighteen times.
- Place tensor and expert parallelism inside the domain and pipeline and data parallelism across it, because the first two move bytes per token and the last two do not.
- The domain becomes the scheduling and spares unit, so fragmentation and partial failures are priced per rack rather than per GPU.