SXM, PCIe and Rack-Scale Form Factors

TL;DR: Three form factors, and the choice is made early because everything downstream depends on it. SXM is a module soldered onto an eight-GPU baseboard, fully meshed over NVLink, drawing 700 W on H100 and about 1,400 W on B300, which puts a node past 10 kW and pushes most deployments to liquid cooling. PCIe is an add-in card in an ordinary server, limited by what the chassis and the slot can deliver and cool, with no NVLink on the current professional parts, so cards talk over PCIe Gen5 at roughly 64 GB/s each direction. Rack-scale, meaning GB300 NVL72 and its relatives, replaces the node boundary: 72 GPUs and 36 CPUs share one 130 TB/s NVLink domain, cooling is liquid and mandatory, and the unit you buy, power, schedule and repair is a rack rather than a server. Each step up buys interconnect bandwidth and costs flexibility.

What actually differs

The row that decides most arguments is the interconnect. Tensor Parallelism explains why a tensor-parallel step communicates twice per layer; the practical consequence is that the same model split the same way runs at a completely different speed depending on which of these three rows it lands in.

The interconnect arithmetic, which is the whole argument

tensor-parallel decode traffic for a 70B model, hidden 8,192, 80 layers, bf16
  each layer does two all-reduces of one hidden vector per token
  bytes per token per GPU (ring all-reduce, TP=8, factor 2(N-1)/N = 1.75):
    80 layers x 2 all-reduces x 8,192 x 2 B x 1.75 = 4.6 MB per token

time for that traffic, per token
  PCIe Gen5 x16 at 64 GB/s:        4.6e6 / 64e9   = 72 microseconds
  NVLink on H100 at 900 GB/s:      4.6e6 / 900e9  = 5.1 microseconds
  NVLink on Blackwell at 1.8 TB/s: 4.6e6 / 1.8e12 = 2.6 microseconds

put that against a decode budget
  a 20 ms per-token budget spends 0.36% on NVLink Blackwell, 0.026 s per 1,000 tokens
  and 72 microseconds per token on PCIe is 0.36% too at TP=8... except PCIe is shared,
    contended, and not a full mesh, so the realized number is several times worse
sanity: the arithmetic says PCIe is survivable and the topology says it is not, because
        PCIe Gen5 bandwidth is per-link and a card's link is shared with storage, the NIC
        and the host, while NVLink is a dedicated all-to-all fabric

Rack-scale changes the unit of everything

Everything about operations changes at the third box. A job that wants 64 GPUs on an HGX fleet is eight nodes and a fabric problem. On NVL72 it is part of one rack and a scheduling problem, because the 72 GPUs share an NVLink domain that a job either sits inside or does not. NVLink Domains and the NVL72 Rack covers what that domain buys; the form-factor point is narrower: the rack is now the thing you order, power, cool, cable and replace, and a half-broken rack is a much worse operational state than a half-broken node.

Choosing, in the order the decision is actually made

1. does the workload need more than one GPU's worth of memory in one model?
   no  -> PCIe is viable and often cheaper per unit of served throughput
   yes -> you need a real scale-up link, so SXM or rack-scale

2. how many GPUs must be in one NVLink domain?
   up to 8   -> HGX node
   up to 72  -> NVL72-class rack
   more      -> the domain boundary is crossed by the scale-out fabric regardless, so
                design for it rather than trying to buy your way past it

3. what can the facility actually take?
   an air-cooled hall at 10 to 15 kW per rack cannot host an NVL72 at 120 kW+
   this constraint kills more configurations than any technical argument, and it is the
   one to check first rather than last

4. what is the repair story?
   a card is a 10-minute swap by anyone; a rack needs coolant handling and trained staff
   spare strategy differs: spare cards are cheap to hold, spare racks are not
sanity: steps 3 and 4 are where deployments actually fail, and they are the two that get
        discussed last in most design conversations

What interviewers are listening for

Whether you connect the form factor to consequences rather than reciting definitions. The strong version names the interconnect difference with a number, then says what it does to tensor parallelism, then says what the facility has to provide. A candidate who mentions that an NVL72 rack cannot go into a hall provisioned for 15 kW has demonstrated something the specification sheet does not teach. Interviewers also probe the failure domain, because people who have operated these systems think about it and people who have only read about them do not.

Key takeaways

• PCIe gives about 64 GB/s each way per card and no NVLink on current professional parts; NVLink gives 900 GB/s on H100 and 1.8 TB/s per GPU on Blackwell.
• Tensor-parallel decode on a 70B model moves roughly 4.6 MB per token per GPU at TP=8, which is 2.6 microseconds on Blackwell NVLink and 72 microseconds on a shared PCIe link.
• SXM at 700 W to about 1,400 W per GPU puts a node past 10 kW, which is where liquid cooling stops being optional.
• GB300 NVL72 makes the rack the unit: 72 GPUs, 36 Grace CPUs, 130 TB/s of NVLink, 20 TB of GPU memory, and roughly 120 to 140 kW.
• Check what the facility can power and cool before choosing a form factor, because that constraint eliminates more options than any performance argument.