TL;DR: A declared 16-byte training-state recipe for 100B parameters totals 1.6 TB. A memory packing minimum does not determine the deadline. On 1,024 GPUs, TP8×full-shard128 and TP8×PP8×DP16 have very different per-rank state and communication; choose after measuring feasible updates, not by assuming every parallelism axis divides memory.
How to approach it
Inventory state and peak buffers first. Size a work/deadline scenario separately, then assign each parallel dimension to an actual group of devices. Compare the two layouts at the same effective batch and numerical requirements, including failure and save-resume behavior.
A strong answer
A lab has a 100B model, an illustrative 2T-token target and 1,024 SXM GPUs. Layout names multiply group sizes: TP8×full-shard128 means 8-way tensor parallelism inside each node and 128-way fully sharded data parallelism across nodes, 8 × 128 = 1,024 GPUs. TP8×PP8×DP16 adds 8 pipeline stages and keeps 16 data-parallel copies: 8 × 8 × 16 = 1,024. A 32-GPU layout might hold its state, but that says little about finishing the training run on time.
static recipe = 100e9 × 16 = 1,600 GB
at a declared 64 GB static allowance/card: ceil(1,600/64) = 25 cards
whole 8-card nodes: 4 nodes = 32 GPUs
work approximation = 6 × 100e9 × 2e12 = 1.2e24 FLOPs
at 1,024 × 989e12 × assumed 0.40 FLOP/s:
time ≈ 34.285 days
sanity: with the same per-GPU rate, 32 GPUs take 32× longer.
Training FLOPs: 6ND is a projection-work approximation. This duration excludes additional work and any availability loss not included in the assumed rate. A six-week deadline needs validation under the actual sequence lengths and training schedule.
In layout A, TP8 first splits each layer's work across eight ranks. Full sharding then distributes those tensor-parallel shards across 128 data-parallel members. With ideal balance, persistent state averages 1,600/1,024 = 1.5625 GB per rank. That small number describes the shards at rest; execution also needs memory for the parameter groups gathered before use.
In layout B, eight pipeline stages divide layers and TP8 divides each stage. Ordinary DP16 replicates the resulting 64-GPU model; it does not shard optimizer state further. Static state is 1,600/(8×8)=25 GB per rank. Unequal layers and tied parameters can make the maximum higher.
For layout A, an illustrative BF16 materialization schedule with two weight gathers and one gradient reduce-scatter sends 3×(127/128)×(200/8)=74.414 GB per rank per micro-batch. At an assumed achieved 50 GB/s path, payload service takes 1.488 seconds. Gradient accumulation and resharding policy can change calls per optimizer update, so label the unit carefully.
For layout B, state the pipeline boundary tensor and whether TP leaves it sharded or replicated before counting network bytes. Pipeline Parallelism and the Bubble explains the scheduling cost: in a balanced ideal p=8, m=32 schedule, idle-to-busy is 7/32=21.875%, while idle-to-elapsed is 7/39=17.949%. Do not compare those two percentages as if they had the same denominator.
| Candidate | Main question for the trace |
|---|---|
| TP plus full sharding | Are parameter gathers ready before use? |
| TP plus pipeline and replicas | Are stages balanced and activation buffers bounded? |
| Wider TP | Does saved compute exceed extra communication? |
| Activation replay | Does the feasible complete update improve? |
The Megatron parallelism study motivates these tradeoffs, but it does not prescribe one layout for every fabric. Start with the simpler supported sharded candidate; switch to pipeline stages when measured gather stalls or memory peaks make that candidate miss the requirement and the pipeline passes it. A communication/compute ratio below one is only an opportunity for overlap.
Use allocation snapshots and p99 complete-update duration. Context and Sequence Parallelism may be needed for long inputs, but no universal activation coefficient can certify the fit. Checkpoint ownership and failure restart must match the chosen group structure; a lost node does not automatically reshard a running job.
What interviewers probe next
- “Can I use TP16 across nodes?” Yes if supported and measured to help; eight is not a mathematical maximum.
- “Why is layout B's static state larger?” Its data-parallel dimension replicates rather than shards.
- “What if the token target becomes 10T?” Modeled work grows fivefold; linear fleet scaling still needs measurement.
Common mistakes
- Dividing state by replicated data-parallel ranks.
- Treating a total-time ratio as proof of hidden communication.
- Promising a memory fit from a generic activation formula.
Key takeaways
- The recipe has 1.6 TB static state; its packing minimum is not a training plan.
- The two illustrated layouts have 1.5625 versus 25 GB static state per rank.
- Compare complete updates under the same work and recovery requirements.