TL;DR: Tensor parallelism all-reduces a tokens × hidden activation four times per transformer block (twice in forward, twice in backward), and each one blocks the next layer, so it cannot hide behind compute the way a gradient all-reduce can. For a 70B at 8k tokens the four collectives cost about 84 ms per micro-batch over NVLink at 900 GB/s against about 1.1 s of compute, roughly 8%. The same bytes over a 50 GB/s NIC would cost 1.5 s, more than the compute itself. Eight is the size of the NVLink domain on an HGX node, so eight is where TP stops. A second limit arrives at the same time: each GEMM's per-GPU dimension shrinks by t, and below a few thousand the tensor cores are underfed.
How to approach it
Name what TP communicates and how often before doing any arithmetic, because the four-per-block count and the "on the critical path" property are the whole argument. Then size one all-reduce from the activation shape, multiply by the count, and compare against NVLink and against the NIC for the same layer's compute time. Add the GEMM-shape argument as the second limit. Close with what to do instead when a model needs more than 8 ways, and the one case where TP16 is used.
A strong answer
A typical situation: a team has a 405B model, computes that TP8 leaves 50 GB of parameters per GPU in bf16 with no room for optimizer state, and asks why they cannot go to TP16 across two nodes.
Tensor Parallelism splits each weight matrix across t GPUs. In Megatron's arrangement the attention block's output projection and the MLP's second linear layer each produce a partial sum on every GPU, and those partial sums must be added: one all-reduce after attention, one after the MLP, per block, in forward. In backward each of those has a mirror all-reduce for the gradient with respect to the block's input. Four per block. The tensor being reduced is the activation, of shape tokens × hidden, in bf16.
inputs: Llama 3.1 70B: hidden h = 8,192, layers L = 80
micro-batch of 8,192 tokens, bf16 activations (2 B)
t = 8, NVLink 900 GB/s, NIC 50 GB/s
one all-reduce buffer = tokens × h × 2 B = 8,192 × 8,192 × 2 = 134 MB
per-GPU bytes (ring) = 2 (t−1)/t × 134 MB = 2 × 7/8 × 134 = 235 MB
count per micro-batch = 4 per block × 80 blocks = 320
total per GPU = 320 × 235 MB = 75 GB
time over NVLink = 75 GB ÷ 900 GB/s ≈ 84 ms
time over NIC = 75 GB ÷ 50 GB/s ≈ 1.5 s
compute per GPU per micro-batch (TP8 share):
6 × 70.6e9 × 8,192 ÷ 8 = 4.3e14 FLOPs ÷ (989e12 × 0.4) ≈ 1.1 s
NVLink: 84 ms ÷ 1.1 s ≈ 8% overhead
NIC: 1.5 s ÷ 1.1 s ≈ 140% overhead
sanity: the NIC case spends more time waiting on the network than computing, and unlike
a gradient all-reduce it cannot overlap, since layer l+1 needs the reduced output
of layer l before it can start. TP across nodes would halve throughput or worse.
The overlap point deserves emphasis. A data-parallel gradient all-reduce can start while backward is still producing earlier layers' gradients, so most of it hides. A TP all-reduce sits between two dependent layers; there is nothing to overlap it with except the next micro-batch's independent work, which pipeline-style scheduling can partly arrange but which costs memory and complexity. Sequence parallelism and the async-TP work in PyTorch decompose each all-reduce into a reduce-scatter and all-gather and overlap them with chunks of the adjacent GEMM, which recovers part of the 8% on NVLink but does nothing for the 140% on the NIC.
Eight is not a magic number; it is the size of the domain in which every GPU talks to every other at NVLink speed. An HGX H100 node has 8 GPUs on 4 NVSwitches. GB200 NVL72 puts 72 GPUs in one NVLink domain, and on that hardware TP of 16 or 32 is used, because the bandwidth argument now allows it. The rule is "TP stops at the NVLink domain", and on 2026 fleets of HGX nodes the domain is 8.
The second limit is GEMM shape. A 70B's MLP projects 8,192 to 28,672. At TP8 each GPU holds 28,672/8 = 3,584 output columns; at TP16 it would hold 1,792, and at TP64 only 448. Tensor cores reach peak on large square-ish tiles; a GEMM with a dimension of a few hundred runs at a fraction of peak because the tile grid does not fill the SMs and the memory traffic per FLOP rises. So even inside a large NVLink domain, TP beyond 16 on a 70B loses efficiency in the kernels themselves, and the fix for "more ways" is a different axis: Pipeline Parallelism and the Bubble for more layers per replica, ZeRO and FSDP for sharded state, Expert Parallelism for MoE.
Back to the 405B team. TP8 leaves 405e9 × 2 B ÷ 8 = 101 GB of bf16 weights per GPU, which already exceeds 80 GB before gradients and optimizer state. The answer is not TP16; it is TP8 × PP16, so each GPU holds 1/128 of the model (6.3 GB of weights, 51 GB of full training state before FSDP over the DP axis brings it down further). That is the Llama 3 layout, and it exists because TP stopped at 8.
Decision: TP equal to the NVLink domain size, or smaller when the model is small enough that the per-GPU GEMMs would be underfed. The condition that reverses it is hardware with a larger domain (NVL72) or a model with a hidden size large enough that TP16 GEMMs stay efficient, and even then only within the domain.
The reversal condition: an NVLink domain of 72 rather than 8, where tensor parallel degrees that were unreachable become a config flag and the whole placement argument reopens. nvidia-smi topo -m is how you confirm the domain you actually have.
What interviewers probe next
- "Why four all-reduces and not two per block?" Forward needs one after attention's output projection and one after the second MLP matrix; backward needs the gradient with respect to the input of each of those, which is another all-reduce each.
- "What does sequence parallelism do to this number?" It replaces each all-reduce with a reduce-scatter plus an all-gather of the same total bytes but lets the layernorm and dropout run on sequence shards, cutting activation memory; it does not reduce traffic.
- "Could you use TP2 across two nodes with TP8 inside?" The cross-node pair would carry the full 134 MB per collective at 50 GB/s, 2.7 ms each, 320 times per micro-batch, 0.86 s; the answer is still no, and pipeline parallelism across the pair costs 5 ms instead.
- "How would TP look on NVL72?" TP16 or TP32 within the rack at 1,800 GB/s per GPU; the 405B's 101 GB per GPU at TP8 becomes 25 GB at TP32, and PP can drop to 4.
Common mistakes
- Saying TP stops at 8 "because of NVLink" without sizing the traffic or noting it is the domain size, not a property of the link.
- Treating the TP all-reduce as overlappable like a DDP all-reduce.
- Forgetting the backward-pass collectives and computing half the traffic.
- Proposing TP16 across nodes for a 405B rather than adding a pipeline axis.
Key takeaways
- Four all-reduces per block on a tokens × hidden activation, on the critical path.
- For a 70B at 8k tokens: about 75 GB per GPU per micro-batch, 84 ms on NVLink (8% of compute), 1.5 s on a NIC (more than compute).
- TP stops at the NVLink domain: 8 on HGX, larger on NVL72.
- The second limit is GEMM shape; below a few thousand columns per GPU the tensor cores idle, so beyond 16 use another axis.