TL;DR: NVLink on an H100 SXM is 900 GB/s bidirectional per GPU, about 450 GB/s each way; a PCIe 5.0 x16 slot is about 64 GB/s each way, and it is shared with the host and the NIC. Tensor parallelism sends two all-reduces per layer on every step and cannot overlap them with compute, so on PCIe a 70B decode step spends longer communicating than reading weights; it needs NVLink. Data parallelism sends one gradient all-reduce per step that overlaps with the backward pass, so PCIe is tolerable when the compute per step is long enough to hide it. Pipeline parallelism sits in between and is the usual answer for PCIe boxes.
How to approach it
Ask what the workload is and how it is parallelized, because the bytes on the wire come from the parallelism scheme, not from the model alone. Write the two link bandwidths down. For the scheme in question, count the collectives per step and the bytes per collective, convert to time on each link, and compare that time with the compute or memory time of the same step. Say whether the communication can overlap. Close with a rule per scheme and the condition that would move a workload across the line.
A strong answer
A typical situation: a team buys PCIe cards because the price per FLOP looked better, then runs tensor parallelism across them and measures a step time four times what they planned. The gap is one division, and it depends entirely on which parallelism axis they chose.
The numbers come from the concept page on NVLink, NVSwitch and PCIe:
H100 SXM NVLink: 900 GB/s bidirectional per GPU = 18 links × 50 GB/s; about 450 GB/s each direction,
and on an HGX board every GPU reaches every other at full rate through NVSwitch
PCIe 5.0 x16: about 64 GB/s each direction, per card, shared with H2D copies and the NIC
behind the same root complex; peer-to-peer through a PCIe switch is at best this
ratio per direction: 450 ÷ 64 ≈ 7x; and PCIe adds tens of microseconds per transfer
Now count bytes for each scheme, using Llama 3.1 70B (hidden 8,192, 80 layers) as the model.
Tensor parallelism, decode, TP8, batch 64. Each layer does two all-reduces (after attention, after the MLP) on an activation of shape [batch × hidden]:
activation = 64 × 8,192 × 2 B = 1.05 MB
ring all-reduce bytes per rank = 2 × (n-1)/n × 1.05 MB = 2 × 7/8 × 1.05 = 1.84 MB
per step = 80 layers × 2 = 160 all-reduces × 1.84 MB = 294 MB per rank
time on NVLink = 294e6 ÷ 450e9 = 0.65 ms + 160 × ~8 µs latency = 1.3 ms ≈ 2.0 ms
time on PCIe = 294e6 ÷ 64e9 = 4.6 ms + 160 × ~25 µs latency = 4.0 ms ≈ 8.6 ms
the step it competes with: weights per GPU = 141 GB ÷ 8 = 17.6 GB; 17.6e9 ÷ 3.35e12 = 5.3 ms
KV at 4k, 64 sequences, sharded: 64 × 1.31 GB ÷ 8 ÷ 3.35e12 = 3.1 ms; memory time ≈ 8.4 ms
NVLink: 8.4 + 2.0 = 10.4 ms per step, communication is 19%
PCIe: 8.4 + 8.6 = 17.0 ms per step, communication is 51%, and it cannot be hidden
sanity: the all-reduce output feeds the next layer, so there is nothing to overlap it with;
a TP step really is the sum, and PCIe cuts throughput by about 40%
The latency term matters as much as the bandwidth term at this batch: 160 collectives per step means a fixed cost per step of 160 × (link latency). That is why TP over PCIe hurts even at batch 1, where the bytes are tiny.
Data parallelism, training a 7B model in bf16 with one all-reduce of the gradients per step:
gradients = 7e9 × 2 B = 14 GB
ring all-reduce bytes per rank over 8 GPUs = 2 × 7/8 × 14 GB = 24.5 GB
time on NVLink = 24.5e9 ÷ 450e9 = 54 ms
time on PCIe = 24.5e9 ÷ 64e9 = 383 ms
compute per step at 4,096 tokens per GPU: 6 × 7e9 × 4,096 = 172 TFLOP
at 40% MFU on 989 TFLOPS: 172e12 ÷ (0.4 × 989e12) = 435 ms
NVLink: 54 ms hides behind 435 ms of backward pass easily
PCIe: 383 ms almost fills the 435 ms window; it hides only if bucketing starts the
reduction early and nothing else touches the bus; double the tokens per GPU
to 8,192 and the window becomes 870 ms, and PCIe is comfortable
sanity: DP communication is fixed per step while compute grows with tokens per step,
so the ratio is a tunable; TP communication grows with the same batch that
grows the compute, so the ratio is fixed by the model
That contrast is the whole answer. DP bytes are proportional to parameters and independent of batch, and they overlap with backward, so a slow link is paid for by a larger per-GPU batch. TP bytes are on the critical path, sent 160 times per step, and scale with the batch, so a slow link is paid on every token. FSDP sits with DP: its all-gathers per layer are sequential with compute unless prefetched, and on PCIe you see 20% to 40% step-time overhead where NVLink shows a few percent.
Pipeline parallelism is what people run on PCIe boxes when the model does not fit on one card. A stage boundary sends one activation tensor per micro-batch, [micro-batch × sequence × hidden], once, forward and once backward. For 4 tokens-in-flight of 8,192 hidden in bf16 that is 64 KB per token, and a 1,000-token micro-batch is 16 MB, 0.25 ms on PCIe against tens of milliseconds of stage compute. The cost of PP is the bubble, not the link.
The decision, with the reversal conditions:
| scheme | on PCIe | reverses when |
|---|---|---|
| TP within a node | no; step time up 40% at batch 64, worse at batch 1 | model is small enough that TP is unnecessary |
| PP within a node | yes | micro-batches too small to hide the bubble, then TP would have been better anyway |
| DP, single-node | yes, with bucketed overlap | tokens per GPU per step so small that compute < 400 ms for a 7B |
| FSDP | tolerable, 20% to 40% overhead | prefetch cannot keep up; then fall back to DP with more memory |
The reversal condition: data parallelism on a model small enough that one all-reduce per step overlaps inside the backward pass. There PCIe is fine, the 14x gap costs nothing measurable, and paying for NVLink buys a link the job never saturates. nvidia-smi topo -m prints which one a given pair of GPUs is actually using, which is worth checking before any of this arithmetic. Communication Volume Estimates gives the bytes for every other parallelism axis, because the answer is sometimes not the one the purchase order implies.
What interviewers probe next
- "Your PCIe box has 8 GPUs; how would you serve a 70B on it?" PP8 or PP4 with two replicas, never TP8; accept the pipeline bubble, and batch to fill it.
- "What about a two-node TP16 over InfiniBand?" 400 Gb/s is 50 GB/s per NIC, below PCIe; the 160-per-step latency cost doubles on top. Nobody does TP across nodes on purpose; the NVL72 exists to make 72 GPUs one NVLink domain so they do not have to.
- "How would you measure whether the link is the bottleneck?" nccl-tests all_reduce_perf at the message size your step uses, compared with the busbw ceiling of the link; and a profile showing the fraction of step time inside ncclAllReduce with no overlapping kernel.
- "Does NVLink help inference at batch 1?" Yes, through latency: 160 collectives at 8 µs is 1.3 ms; at PCIe's 25 µs it is 4 ms on a step whose memory time is 5.3 ms.
Common mistakes
- Comparing 900 GB/s with 64 GB/s and stopping there, without counting how many bytes the workload sends.
- Assuming all communication overlaps with compute; TP all-reduces do not, and that is why they need the fast link.
- Forgetting the latency term at small batch, where TP over PCIe is slow even though the bytes are small.
- Sizing DP per-GPU batch without checking that the compute window is longer than the gradient reduction.
Key takeaways
- NVLink 450 GB/s per direction vs PCIe 64 GB/s: 7x in bandwidth and 3x in latency, and PCIe is shared with the host and NIC.
- TP sends 2 × layers all-reduces per step on the critical path: 70B TP8 at batch 64 is 2 ms on NVLink, 8.6 ms on PCIe against 8.4 ms of memory time.
- DP sends 2 × (n-1)/n × gradient bytes once per step, overlapped: 24.5 GB for a 7B is 54 ms NVLink, 383 ms PCIe; hide it with a longer step.
- PCIe boxes run PP and DP; TP is the scheme that needs NVLink, and cross-node TP is worse than PCIe.