TL;DR: Ordinary data parallelism splits examples across model replicas; tensor parallelism splits work inside a layer; pipeline parallelism splits layers into stages. Their communication differs in payload, frequency and dependency. DP2×TP2×PP2 uses eight ranks. Place those groups from the actual topology and workload rather than a rule that TP must stop at eight GPUs.
How to approach it
Describe ownership first, then follow a forward activation and a backward gradient. Count the independent groups before counting bytes. Check persistent and peak memory, communication timing and batch constraints before choosing a layout.
A strong answer
Eight GPUs do not automatically mean eight copies of the model or one model split eight ways. The parallelism coordinates determine which state each rank owns. Data Parallelism and DDP, Tensor Parallelism, and Pipeline Parallelism and the Bubble describe three different axes.
Ordinary data-parallel ranks process different examples and synchronize the corresponding gradients, usually in several buckets as backward progresses. Initialization and some runtime operations may also synchronize parameters or buffers. Tensor-parallel ranks instead cooperate on work inside a layer, exchanging shards or partial results. Pipeline stages pass activations forward and their gradients backward across stage boundaries for each micro-batch. If data-parallel state is sharded too, count the additional movement that sharding requires.
For the illustrated layout, label a rank by (dp, pp, tp), each coordinate in {0,1}:
world size = DP × PP × TP = 2 × 2 × 2 = 8
one complete model replica = PP × TP = 4 ranks
DP group: fix pp,tp; vary dp → 2 ranks, 4 such groups
TP group: fix dp,pp; vary tp → 2 ranks, 4 such groups
PP chain: fix dp,tp; vary pp → 2 ranks, 4 such chains
sanity: 2 complete replicas × 4 ranks per replica = 8 ranks.
Corresponding DP members hold matching model partitions; they need not each hold the full model once TP/PP is added. A global batch counts unique examples across DP replicas, not once per TP rank. The Megatron-LM parallelism paper explains how these axes combine and why scheduling and placement matter.
Consider an illustrative unsharded BF16 boundary activation with 4,096 token rows and width 8,192. Its payload is 4096×8192×2 = 67,108,864 bytes = 64 MiB. If the backward boundary uses the same shape and dtype, the two directions total 128 MiB per micro-batch across that boundary. Actual sequence/tensor sharding changes what each rank sends; do not multiply a full-tensor count by TP degree without checking ownership.
For a two-rank ring all-reduce of a 64 MiB buffer, each rank sends 2×(2−1)/2×64 = 64 MiB and receives the same amount. How many such operations occur depends on the model implementation and parallelism scheme. Four all-reduces per transformer block is one traditional formulation, not a universal contract.
| Axis | Cost to check |
|---|---|
| DP | Replicated state or sharding traffic, gradient readiness and global batch |
| TP | Frequent layer dependencies, shrinking local matrices and collective latency |
| PP | Stage imbalance, boundary transfer and fill/drain idle time |
| Combined | Exact groups, per-rank peak memory and competing paths |
TP often benefits from a fast local interconnect because its dependencies occur within layers. That is a useful starting hypothesis, not a prohibition on crossing hosts. Rack-scale fabrics, model shape, supported algorithms and the alternative pipeline schedule can reverse the placement. Similarly, PP can suffer on a slow link; its communication is not free merely because it is point-to-point.
Build the simplest feasible candidate, measure per-rank peak memory and complete update time, then compare alternatives at the same useful global batch. Use a device trace and collective logs to identify exposed delay. If additional TP makes the local matrix too small or communication too costly, try another supported split. If PP creates stage imbalance, redistribute measured work or reconsider the number of stages. There is no fixed “small model below 10B needs no TP” rule.
What interviewers probe next
- “What if one layer does not fit?” More PP stages alone cannot split that layer. Consider tensor/state sharding, checkpointing or a supported tiling/offload strategy.
- “Do more micro-batches change the global batch?” They can. At fixed global batch, they reduce micro-batch size and may change kernel efficiency.
- “Does DP communicate once?” An optimizer update can include many bucket collectives; distinguish one logical synchronization from one API call.
Common mistakes
- Counting total GPUs as every collective's group size.
- Using bidirectional aggregate bandwidth as one-direction payload rate.
- Assigning exact fractions of every tensor to TP/PP without checking replicated components.
Key takeaways
- DP2×TP2×PP2 creates two four-rank model replicas.
- A stated 4,096×8,192 BF16 boundary contains 64 MiB before any sharding.
- Ownership, readiness and measured topology determine the layout.