Scale-Out Fabric Choice: InfiniBand XDR vs Spectrum-X

TL;DR: A scale-out fabric must carry the workload’s collectives under load, not just reach its advertised link rate. InfiniBand and Spectrum-X use different management and congestion-control mechanisms. Choose a topology and operating model the team can validate, then count endpoints, uplinks, switches and cables.

Scale-up links connect GPUs inside one server or rack (NVLink); the scale-out fabric is the network that connects those servers to each other, one or more network ports per GPU.

The two options, compared on what differs

InfiniBand offers a purpose-built fabric operating model. Spectrum-X may fit a team’s existing Ethernet experience, but still requires AI-specific configuration and validation. Congestion Control for AI Fabrics covers why Ethernet needs the extra work at all, and RDMA, InfiniBand and RoCEv2 covers the transport underneath both.

Radix decides the shape of the whole build

a two-tier fat tree from switches with R ports
  each leaf spends half its ports downward to endpoints and half upward to spines
  endpoints per leaf = R/2
  number of leaves that can be fully connected = R (each leaf uses R/2 uplinks, one per spine
    group), so
  max endpoints = R x R / 2

for the Quantum-X800 Q3400-RA at R = 144 (144 ports of 800 Gb/s in 4U)
  max endpoints = 144 x 144 / 2 = 10,368
  at 8 NICs per node that is 1,296 nodes, or 10,368 GPUs on a rail-optimized design where
    each GPU owns one port
sanity: a 8,192-GPU cluster fits in two tiers on 144-port switches and would need three tiers
        on 64-port switches, which is one extra hop of latency and roughly a third more
        optics, so radix is a budget decision and not a detail

How rail optimization groups traffic

A node has eight GPUs and eight NICs. Rail-optimized means NIC i on every node connects to leaf switch i, so all the rank-i GPUs in the cluster share one switch layer. The payoff is that a collective which communicates between the same rank on different nodes, which is what data-parallel all-reduce does, stays within one rail. It avoids the spine only when the participating endpoints share a leaf; traffic across larger rail groups can still use spine links.

what rail-optimized buys, in numbers
  a data-parallel all-reduce among rank-0 GPUs on 64 nodes:
    if all 64 endpoints share one leaf, traffic stays on that leaf;
    if they span leaves, the rail also uses spine paths
  the same collective on a non-rail-optimized design:
    traffic spreads across leaves and crosses the spine, adding a hop and competing with
    every other rail's traffic
  a hop of switch latency is small, on the order of hundreds of nanoseconds, and the real
    gain is contention: keeping eight independent rails from sharing one oversubscribed path
sanity: rail-optimized does not make any single link faster, it stops eight collectives from
        colliding, which is why its benefit shows up as tail latency rather than as bandwidth

Sizing from NVIDIA's reference architecture

NVIDIA's DGX SuperPOD B300 reference architecture, published in their documentation, uses a scalable unit of 64 DGX B300 nodes and 512 GPUs served by 8 leaf switches, with four nodes to a rack and rack power above 50 kW. Four scalable units make 256 nodes and 2,048 GPUs. Using that as the unit rather than inventing one has a practical benefit beyond correctness: the cable counts, rack layouts and switch placements are already worked out, and deviating from them means redoing that work.

one scalable unit, from the reference architecture
  nodes                64 DGX B300
  GPUs                 512
  leaf switches        8, which is one per rail
  node-to-leaf links   512, one per GPU
  racks                16, at 4 nodes per rack
  compute power        16 racks x >50 kW = over 800 kW before storage, management or cooling
sanity: 512 GPUs and 8 leaves means 64 ports used per leaf downward, which fits comfortably
        inside a 144-port switch and leaves the rest for uplinks, so one SU does not even
        fill a leaf

What interviewers are listening for

Start with the traffic pattern, required bandwidth and latency, and the team’s operating experience. Explain how the proposed fabric handles congestion and failures. When using a reference architecture, identify which assumptions you retain, including endpoint count, oversubscription and rail layout.

Key takeaways

• As of September 2026 both fabrics run 800 Gb/s per port, so the decision is congestion behavior, operational familiarity and multi-tenancy rather than speed.
• Two-tier fat tree capacity is R squared over 2 endpoints: 10,368 on a 144-port Quantum-X800 switch, against 2,048 on a 64-port switch.
• Rail-optimized connects NIC i of every node to leaf i, keeping same-rank collectives on one rail; whether they cross a spine depends on the topology.
• NVIDIA's DGX SuperPOD B300 scalable unit is 64 nodes, 512 GPUs, 8 leaf switches, 4 nodes per rack, over 50 kW per rack.
• Rail-optimization improves tail latency by removing contention rather than raising any single link's bandwidth.