SXM, PCIe and Rack-Scale Form Factors
GPU form factors determine how accelerators connect, receive power and get serviced. Compare PCIe cards, SXM modules and rack-scale systems against the workload and the facility.
TL;DR: A PCIe GPU is an add-in card. An SXM GPU is a module mounted on a server baseboard. A rack-scale system connects GPUs across multiple compute trays. To choose between them, identify where GPU-to-GPU traffic travels, then check whether the facility and software can support that layout.
See where the GPUs connect
Suppose you want eight GPUs to work on one model. Eight available devices tell you the capacity count; the connections between them determine how they exchange data. Form factor describes how a device is physically packaged and installed. It influences the available connections, power delivery and cooling.
Read these examples from a card, to a server, to a rack. Blue lines show the connection being discussed. The drawings simplify wiring and omit storage, power supplies and most CPU connections.
PCIe: cards in a server
Illustrative two-card PCIe server
Each GPU is a separate card installed in a slot. This example connects the cards and CPU through a PCIe switch.
Peer-to-peer transfers depend on the GPU, switch layout and software. Some PCIe GPU products also support NVLink bridges; a card shape alone does not tell you the topology.
SXM: modules on a baseboard
Example: an eight-GPU HGX H100 system
Eight GPU modules share a purpose-built baseboard. NVLink connections through NVSwitch give the GPUs a fast path to one another inside the server.
A job spread across several such servers must also use the cluster network. Count the transfers that stay inside a server and those that leave it.
NVL72: the domain spans a rack
Example: a GB300 NVL72 system
Eighteen compute trays contain four GPUs each. Rack-level NVLink switches connect the GPUs across trays into one 72-GPU domain.
The domain is larger than one server tray. The cluster network still connects this rack to other systems; cooling, power and service access must be planned for the whole rack.
An NVLink domain is the group of GPUs connected through that NVLink fabric. It describes a communication boundary, not a promise that any job will use the whole group efficiently. NVLink Domains and the NVL72 Rack explains how that boundary affects placement.
The eight-GPU example follows NVIDIA’s DGX H100 system description. The rack example follows NVIDIA’s rack-system hardware guide. Check the exact product configuration before applying these examples to a purchase.
What changes when you choose a form factor?
| Decision | PCIe card server | Eight-GPU SXM server | GB300 NVL72 rack |
|---|---|---|---|
| Physical integration | Cards, slots and their host server | Modules, baseboard and complete server | Compute trays, switch trays and rack services |
| Communication to inspect | PCIe topology; bridges if supported | Local NVLink fabric, then the cluster network | Rack NVLink fabric, then the cluster network |
| Facility check | Chassis power, airflow and space | The vendor’s server power and cooling requirements | Liquid cooling, rack power and service access |
| Repair scope | Identify the failed card or host component | Identify the failed module, baseboard, switch or host component | Identify the failed tray, switch or shared rack service |
Avoid treating a GPU’s power rating as the rack’s power budget. CPUs, switches, memory and cooling also consume power, and service procedures depend on the system vendor. A “swap in minutes” promise needs an actual maintenance procedure and spare parts.
Follow one exchange between GPUs
Tensor Parallelism divides work within a model layer. The GPUs then combine partial results. Consider an illustrative 70B model with 80 layers and a hidden size of 8,192, running one token step with eight tensor-parallel GPUs (TP=8). Assume two ring all-reduces per layer and two bytes per value:
Payload for one hidden vector = 8,192 × 2 = 16,384 bytes
Ring all-reduce sent bytes per GPU = payload × 2 × (8 - 1) / 8
Whole model step = 80 layers × 2 collectives × 16,384 × 1.75
= 4,587,520 bytes sent per GPU, about 4.59 MB
Sanity: one collective sends 28,672 bytes; 160 of them send about 4.6 MB.
On the eight-GPU SXM example, those exchanges can stay inside one NVLink domain. Spread the group across two such servers and some traffic must cross the cluster network. That is why GPU count alone cannot describe the deployment.
This simplified byte count excludes protocol overhead and does not predict elapsed time. Small sequential collectives pay launch and synchronization costs. Batch size, algorithm and placement can change the result. Collective Communication explains the operations behind the calculation.
Compare bandwidth figures on the same basis. PCIe Gen5 x16 carries about 64 GB/s in each direction. NVIDIA lists H100 SXM NVLink bandwidth as 900 GB/s bidirectional aggregate. These figures describe different links and direction conventions. Measure collective latency for the actual message sizes and placement before choosing a system.
A 64-GPU job makes the boundary visible
On eight-GPU HGX servers, a 64-GPU job occupies 64 ÷ 8 = 8 servers. Communication between those servers uses the cluster network. A rack has enough GPUs for the same device count within one rack-level NVLink domain. That does not establish equal memory, equal speed or an efficient parallelism layout; the GPU models and workload still matter.
Check your understanding: If you spread an eight-GPU job over two HGX servers, four GPUs in each, does it still have one local NVLink domain? No. You have crossed the server boundary, so the communication plan must account for the cluster network.
First check whether the facility can support the candidate system. Then check model memory and communication. A model that fits on one card may run well as independent PCIe replicas. When the model must be sharded, compare its measured communication cost on the proposed layouts. Plan repair around the components and services a job actually depends on; buying a rack does not mean replacing the rack for every fault.
What interviewers are listening for
Explain where data travels. Name the local GPU connection, the point where traffic leaves that domain, and the power or cooling constraint that could rule out the hardware. Describe the effect of a component failure on the running job before discussing replacement hardware.
Key takeaways
- A PCIe card, an SXM module and an NVL72 rack describe different physical boundaries.
- Eight GPUs in one HGX H100 server share a local NVLink domain; eight GPUs split across servers also need the cluster network.
- GB300 NVL72 extends the NVLink domain across trays to 72 GPUs.
- Compare per-direction and aggregate bandwidth separately, then measure the workload’s actual communication.
- Check the complete system’s power, cooling and repair requirements.