TL;DR: Count communication bytes, collective frequency and exposed waiting time before choosing an interconnect. SXM's 900 GB/s bidirectional NVLink aggregate is roughly 450 GB/s per direction; PCIe 5.0 x16 is roughly 64 GB/s per direction before protocol losses. That is a roughly sevenfold directional ceiling ratio, not a universal application slowdown. Tensor, pipeline, data and fully sharded parallelism have different dependencies and overlap opportunities.
How to approach it
First establish the actual GPU pair topology and link direction convention. Derive one message size for the chosen parallel layout. Compare measured collective time with the useful work that can run after that message becomes ready. Then explain whether a different sharding strategy or more local work changes the purchasing decision.
A strong answer
An eight-GPU server can look attractive on price per FLOP while a sharded model spends its token latency waiting between layers. Another workload on the same machine may barely communicate. Count the time the workload must wait for communication before it can continue. This exposed communication can differ greatly between workloads on the same number of GPUs.
Normalize the link numbers
NVIDIA's H100 page lists the SXM interconnect figures. NVLink, NVSwitch and PCIe explains the topology distinction. Compare approximately 450 GB/s in one direction across an H100 SXM GPU's NVLink links with approximately 64 GB/s in one direction on a PCIe 5.0 x16 link. Do not compare 900 with 64 as though both counted the same directions.
Those are endpoint ceilings. An NVSwitch fabric does not give every simultaneous pair a separate 450 GB/s beyond each GPU's total injection capacity. PCIe peer traffic may stay within a switch or cross bridges and a root complex (the PCIe controller on the CPU side); contention with NIC and host traffic depends on placement. A bridged NVL product and an SXM system are different configurations.
The nvidia-smi documentation describes topology and peer-access reporting. nvidia-smi topo -m reports connectivity classes; it does not prove which transport a particular NCCL operation selected. Check runtime transport logs and a trace as well. The hardware commands are diagnostic examples, not measurements executed on this review machine.
Count a representative tensor-parallel reduction
For an illustrative dense transformer with 80 layers, hidden width 8,192, BF16 activations, batch-64 decode and a conventional two-all-reduces-per-layer TP8 layout:
activation payload S = 64 × 8,192 × 2 = 1,048,576 bytes
ring transferred bytes per rank = 2 × (8-1)/8 × S = 1,835,008 bytes
160 collectives per decode step → 293,601,280 bytes per rank
bandwidth-only floor at 450 GB/s ≈ 0.652 ms
bandwidth-only floor at 64 GB/s ≈ 4.588 ms
sanity: about seven times the transfer floor, not seven times total step time.
The NCCL tests performance guide defines algorithm bandwidth and the collective-specific bus-bandwidth correction. Use the ring volume as a model, then measure the selected algorithm at these message sizes. SHARP, NVLink-specific algorithms, topology and protocol choices can change the implementation. Per-collective launch and latency costs matter especially for small messages; no fixed eight- or twenty-five-microsecond product guarantee is assumed here.
A conventional TP dependency needs a reduction result before downstream work consumes it. That makes some waiting unavoidable for that dependency. It does not mean all implementations have zero overlap: scheduling independent work, chunking and specialized communication kernels can hide portions. A profile must separate total communication duration from its exposed contribution.
Data parallelism needs a real overlap window
For a toy replicated 7B training job with BF16 gradients across eight ranks, a full gradient payload is 14 GB and the ring volume is 2 × 7/8 × 14 = 24.5 GB per rank. At 64 GB/s, the ideal transfer floor is about 383 ms. Gradient bucketing can start reductions during backward, but each bucket has a release time and the optimizer has a deadline.
The diagram deliberately uses a simple hypothetical window: a bucket is ready at 100 ms, compute ends at 300 ms, and reduction lasts 383 ms. At most 200 ms overlap, leaving at least 183 ms exposed. A whole training step estimated at 435 ms includes forward and other work; it is not automatically a 435 ms backward overlap window. Later buckets often determine the final tail.
Larger local batches can increase useful compute relative to gradient bytes. Test the batch's memory use and optimization behavior too. A feasible fully replicated 7B Adam training state may exceed one 80 GB GPU depending on its recipe; the 14 GB gradient example isolates communication and is not a complete fit claim.
Pipeline and fully sharded layouts have other costs
Pipeline parallelism sends activations at stage boundaries rather than reducing every layer over all ranks. At hidden8,192 and BF16, one token's activation is 16,384 bytes. A boundary tensor with 1,000 token positions is 16.384 MB, giving a 0.256 ms transfer floor at 64 GB/s. Training also returns activation gradients. Stage imbalance, pipeline bubbles and link contention can dominate; PCIe is not automatically harmless.
FSDP adds parameter all-gathers and gradient reduce-scatters. It is not simply replicated data parallelism's one gradient reduction. Parameter prefetch consumes memory and must finish before a layer needs its weights. Communication Volume Estimates provides the payload models; profile their release times, deadlines and memory budget on the intended topology.
Decision and verification
Choose a fast scale-up domain when repeated inter-rank dependencies materially consume the token or step budget. PCIe can be economical for independent replicas, suitable pipeline layouts or training with enough proven overlap. Cross-node TP is a valid option when capacity and performance measurements justify it; it is not categorically forbidden. Rank count alone also does not establish a factor-of-two collective latency change.
Reverse the fast-link purchase when measured application goodput and reliability meet the objective on the cheaper topology after accounting for operational cost. Run representative collective sizes and rank placements, then compare with the application trace under concurrent traffic. Record p99 token gaps for serving and the final exposed reduction tail for training. A high bus-bandwidth number is useful evidence, but it is not the application's latency result.
What interviewers probe next
- "Does TP8 on PCIe always fail?" No. Model size, kernels, batch, topology and the target determine whether its exposed communication is acceptable.
- "Why does a good all-reduce benchmark not prove good overlap?" The benchmark lacks the application's bucket release times and compute dependencies.
- "Can a pipeline hide everything?" Only if the schedule, available micro-batches and stage balance support it; the activation must still arrive before consumption.
Common mistakes
- Comparing a bidirectional aggregate with a one-direction link number.
- Using whole-step compute time as an available backward overlap window.
- Treating FSDP like one gradient all-reduce or topology output like a transport trace.
Key takeaways
- Approximately 450 versus 64 GB/s is a sevenfold directional ceiling comparison.
- The toy TP8 layout moves about 294 MB per rank across 160 collectives; latency and exposure still need measurement.
- A 383 ms reduction released with 200 ms of compute left has at least a 183 ms tail.