TL;DR: Tensor parallelism does not have to stop at eight. The useful degree depends on topology, matrix shapes, memory and communication overlap. In one conventional training schedule, 320 ring all-reduces of a 128-MiB buffer send about 75.16 GB per rank at TP8. Sending that payload takes 0.251 or 1.879 seconds at assumed achieved send rates of 300 or 40 GB/s. Those estimates compare two paths; actual step time also depends on overlap and other work.
How to approach it
Name the specific parallel algorithm and communication group. Count buffers, dtypes and calls over one micro-batch, then compare compatible achieved rates. Recompute both work and communication for a wider candidate before deciding whether it helps.
A strong answer
An eight-GPU node is full, so someone suggests making the tensor group span two nodes. That can be a valid option. Tensor Parallelism explains the layer partition; the question is whether less work per rank outweighs smaller GEMMs (general matrix multiplies) and a different communication path.
Use the conventional two-forward/two-backward all-reduce pattern in the Megatron study, without sequence-parallel substitutions or asynchronous chunk overlap. Modern implementations can change that schedule, so inspect the actual trace rather than requiring four calls everywhere.
s = 8,192 tokens; h = 8,192; BF16 activation = 2 bytes
t = 8 ranks; L = 80 blocks
one buffer S = s × h × 2 = 134,217,728 bytes = 128 MiB
ring sends/rank/call = 2(t−1)/t × S = 234,881,024 bytes
calls/micro-batch = 4 × 80 = 320
sends/rank/micro-batch = 75,161,927,680 bytes ≈ 75.162 GB
sanity: count sent bytes once; receive traffic is a separate quantity.
The rates are hypothetical achieved rates for this send model. A product's bidirectional aggregate NVLink figure is not automatically a unidirectional payload rate. Startup, contention, topology and synchronization add costs; the model does not say either bar is the exposed delay.
For a 70.6B dense model, the simplified training work per TP8 rank is 6×70.6e9×8,192/8=4.3377e14 FLOPs. At an assumed 989e12×0.4 FLOP/s, that is about 1.096 seconds of modeled compute. This comparison tells you that network service can be substantial, but it does not establish how much overlaps or which candidate wins.
At TP16 the ring factor becomes 2×15/16=1.875 instead of 1.75, and the ideal compute share halves. Reusing TP8's compute time would make the wider group look artificially attractive. Its matrix shapes change too: an output dimension of 28,672 splits into 3,584 columns at TP8 and 1,792 at TP16. There is no universal column threshold at which tensor cores stop working; benchmark the complete shape.
| Candidate change | What must be recomputed |
|---|---|
| Wider tensor group | Group topology, latency and local GEMM shape |
| Sequence parallelism | Gather/scatter schedule and saved activations |
| Async tensor parallelism | Chunk dependencies and real overlap |
| Pipeline or state sharding | Different ownership and critical path |
Many TP collectives return data that the next operation needs. A late result can therefore stall computation, but “no overlap is possible” is too strong. Splitting adjacent computation and communication or scheduling independent work can expose overlap opportunities. Whether they help across a slower path is a measurement question.
Start with a degree that fits the fast communication domain and the model dimensions. Expand it when the measured complete update or serving SLO improves, or when a required memory fit justifies the tradeoff. Reverse the expansion if communication or small local kernels erase the benefit. Pipeline Parallelism and the Bubble and ZeRO and FSDP provide other ways to distribute work and state.
The node topology report and p99 collective/update timing are useful signals, but neither a domain size nor a topology label guarantees the optimal degree. A larger switched domain expands the search space; it does not prescribe TP equal to its GPU count.
What interviewers probe next
- “Does sequence parallelism always move the same bytes?” A specific reduce-scatter/all-gather decomposition can match ring all-reduce volume, but actual frequency and dtype still matter.
- “What about a 405B BF16 weight payload at TP8?” 405e9×2/8=101.25 GB per rank before other state; eight 80-GB devices cannot hold that all-resident layout.
- “Should TP16 be rejected without testing?” No. Verify support, fit and a matched performance result.
Common mistakes
- Treating eight as a mathematical maximum.
- Dividing sent bytes by a bidirectional aggregate specification without checking conventions.
- Changing TP degree while leaving its compute share unchanged in the comparison.
Key takeaways
- The declared TP8 schedule sends about 75.16 GB per rank per micro-batch.
- TP16 changes both the communication factor and local matrix work.
- Pick the degree from a supported, measured layout rather than a universal rule.