Practice tests · 30 questions

Distributed Training & Parallelism: the practice test

DDP, ZeRO and FSDP, tensor, pipeline, context and expert parallelism, collectives and their cost, MFU, activation checkpointing, elastic and fault-tolerant training, checkpoint economics and the RL post-training stack. Owning the training run at cluster scale. This test drills exactly that: 10 easy, 12 medium and 8 hard questions, every one explained, every explanation linking into the worked material.

Set up your test

Topic

How confident are you feeling?

Questions

10 in this pool · about 7 min

Reveal answers

Sample questions, answered

easy · sample

In plain data-parallel training with DDP, what crosses the network each step?

The activations of every layer, exchanged so that each rank can run the backward pass for the others

✓The gradients, all-reduced so every replica applies the same averaged update

The optimizer states, broadcast from rank 0 after it applies the update to its own copy

The training data, shuffled between ranks so that every replica eventually sees every sample

Each rank holds a full model copy and its own slice of the batch. After the backward pass the gradients, one value per parameter, are all-reduced so every rank ends with the same average and takes the same optimizer step. For a 7B model in bf16 that is about 14 GB of gradients per step, moved as 2 × (N − 1)/N of that per GPU in a ring, and overlapped with the backward pass by bucketing. Nothing else has to cross the wire in DDP; the data is sharded by rank up front.

easy · sample

Data, tensor and pipeline parallelism split what, respectively?

The layers across devices, the batch across replicas, and the optimizer across ranks

The dataset files, the tokenizer vocabulary, and the checkpoint shards

✓The batch, the weight matrices within a layer, and the layers across devices

The GPUs within a node, the nodes within a pod, and the pods within a cluster

Data parallelism gives each replica the whole model and a slice of the batch, communicating gradients once per step. Tensor parallelism splits individual matrices across devices and communicates activations several times per layer, which is why it stays inside an NVLink domain. Pipeline parallelism assigns contiguous layers to stages and passes activations point to point between them, trading communication for pipeline bubbles. A frontier run combines all three, plus expert and context parallelism where the model calls for them.

Go deeper than the quiz

A practice test measures recall. The material it draws from teaches the reasoning: