TL;DR: Data parallelism splits the batch and communicates gradients once per step. Tensor parallelism splits individual weight matrices and communicates activations several times per layer. Pipeline parallelism splits the layers into stages and communicates activations once per stage boundary per micro-batch. Frequency and volume decide placement: TP's per-layer activation all-reduces need NVLink at 900 GB/s and stay inside a node; PP's point-to-point sends and DP's once-per-step all-reduce tolerate the 50 GB/s NIC and go across nodes.
How to approach it
Give the three in one sentence each, then draw the table with three columns: what is sharded, what is communicated, and how often. That table is most of the answer. Then say the placement rule and justify it with one number per method: bytes moved per layer or per step against the bandwidth of the link it would use. Finish by naming what each method fixes and what it costs, because the interviewer wants the trade, not the definitions.
A strong answer
A typical situation: a team has a 70B model and 64 H100s and is deciding how to cut the work. Each parallelism answers a different question. Data parallelism answers "how do I use more GPUs to process more tokens per step". Tensor parallelism answers "how do I make one layer fit and run faster when it is too large for one GPU". Pipeline parallelism answers "how do I hold more layers than one GPU can hold without paying tensor parallelism's communication price across nodes".
| Data parallel | Tensor parallel | Pipeline parallel | |
|---|---|---|---|
| What is sharded | the batch | each weight matrix, by rows or columns | the layers, into contiguous stages |
| What each GPU holds | the whole model | 1/t of every layer | all of p⁻¹ of the layers |
| What moves | gradients | activations (partial sums) | activations at stage boundaries |
| How often | once per optimizer step | 4 all-reduces per transformer block (2 forward, 2 backward) | 1 send per boundary per micro-batch, forward and backward |
| Collective | all-reduce | all-reduce (or reduce-scatter + all-gather) | point-to-point send/recv |
| What it fixes | throughput | per-layer memory and compute | total parameter memory |
| What it costs | full model per GPU | bandwidth, and it stops at 8 | the bubble |
The placement rule follows from the volumes. Take Llama 3.1 70B (hidden size 8,192, 80 layers) with an 8k-token micro-batch.
Tensor parallelism, activation all-reduce per block:
bytes = tokens × hidden × 2 B = 8,192 × 8,192 × 2 = 134 MB
per GPU at t = 8: 2 × 7/8 × 134 MB = 235 MB per all-reduce
four per block × 80 blocks = 320 all-reduces per micro-batch
total per GPU per micro-batch = 320 × 235 MB ≈ 75 GB
on NVLink at 900 GB/s: 75 GB ÷ 900 GB/s ≈ 84 ms
on a NIC at 50 GB/s: 75 GB ÷ 50 GB/s ≈ 1.5 s
Pipeline parallelism, one stage boundary per micro-batch:
bytes = tokens × hidden × 2 B = 134 MB, forward and again in backward
at 50 GB/s: 2 × 134 MB ÷ 50 GB/s ≈ 5 ms per micro-batch
Data parallelism, once per step:
gradient of a 70B model in bf16 = 141 GB; with TP8 × PP-something each DP rank owns a slice,
and the all-reduce is once per optimizer step, overlapped with backward.
compute per GPU per 8k micro-batch (TP8 only, no PP):
6 × 70.6e9 × 8,192 ÷ 8 GPUs ≈ 4.3e14 FLOPs ÷ (989e12 × 0.4) ≈ 1.1 s
sanity: TP's 84 ms on NVLink is 8% of the 1.1 s compute, tolerable; the same traffic on the
NIC would exceed the compute itself, so TP across nodes is off the table.
That is the whole argument for the standard layout: Tensor Parallelism inside the node where NVLink lives, Pipeline Parallelism and the Bubble across nodes where its rare point-to-point transfer costs milliseconds, and data parallelism across the remaining GPUs with its once-per-step all-reduce hidden behind the backward pass. Megatron's original paper made the same placement in 2019 and the Llama 3 report made it at 16,384 GPUs.
Each method also has a cost that is not bandwidth. Data parallelism replicates the model, so it does nothing for memory on its own; that is why ZeRO and FSDP exist. Tensor parallelism divides each GEMM by t, and below a certain per-GPU matrix size the tensor cores are underfed, which is a second reason it stops at 8. Pipeline parallelism idles stages while the pipeline fills and drains, the bubble, and shrinking that bubble means more micro-batches in flight, which means activation memory.
The decision for the 64-GPU 70B example: TP8 within each node, and across the eight nodes either PP8 (each node holds ten layers, DP1) or FSDP across all eight nodes with no pipeline, depending on whether the 8k-token micro-batches give enough overlap. The condition that reverses TP8 is a small model: below about 10B parameters a layer fits comfortably on one GPU, TP buys nothing, and FSDP alone is simpler and faster.
The reversal condition: a fabric fast enough that the pipeline's bubble costs more than tensor parallelism's activations would. That is rare across nodes today and normal inside an NVL72 domain, so the layout that is right on HGX is not automatically right on a rack-scale part. Communication Volume Estimates carries the bytes for both. NCCL_DEBUG=INFO prints which algorithm and topology the library chose, which is the first thing to read on any layout that underperforms.
What interviewers probe next
- "Why does tensor parallelism need four all-reduces per block and not two?" Two in forward (after attention's output projection and after the MLP's second matrix), and each has a matching all-reduce in backward for the input gradient.
- "Where does sequence parallelism fit?" It shards the layernorm and dropout activations along the sequence within the TP group, replacing each all-reduce with a reduce-scatter and an all-gather of the same total bytes but lower peak memory.
- "Can PP ever go inside the node instead of TP?" Yes when NVLink is not present (PCIe boxes) or when the model's layers are small; PP's traffic is so light it works anywhere, and its bubble is the only reason not to prefer it.
- "Which one do you add first when the model does not fit?" Sharded data parallelism (FSDP) first, because it changes no math and needs no model surgery; TP when a single layer's GEMMs are too slow or large; PP when the network cannot carry FSDP's all-gathers.
Common mistakes
- Describing pipeline parallelism as "splitting the model across GPUs" without saying that tensor parallelism also does, and that the difference is which axis is cut.
- Placing tensor parallelism across nodes because "the fabric is fast now". A 400 Gbps NIC is 50 GB/s, one eighteenth of NVLink, and TP traffic is per layer, not per step.
- Forgetting that data parallelism needs a full copy of the training state per rank, and then wondering why a 70B model does not train on eight GPUs with DP8.
- Quoting the bubble as a fixed percentage without saying it depends on micro-batch count.
Key takeaways
- DP shards the batch and moves gradients once per step; TP shards matrices and moves activations four times per block; PP shards layers and moves activations once per boundary per micro-batch.
- For a 70B at 8k tokens, TP8's per-micro-batch traffic is about 75 GB per GPU: 84 ms on NVLink, 1.5 s on a NIC. That number places TP inside the node.
- PP moves about 134 MB per boundary and tolerates any link; its cost is the bubble.
- Standard order: TP inside the node, PP across nodes, DP across the rest.