← 🕸️ Distributed Training
Advanced
Pipeline Parallelism and the Bubble
Pipeline parallelism puts consecutive groups of layers on different GPUs and streams micro-batches through them, which is the only parallelism whose traffic is small enough to cross a slow fabric comfortably. Its cost is the bubble, the idle time while the pipeline fills and drains, and the schedule you pick (GPipe, 1F1B, interleaved, zero-bubble) decides how much of each step is wasted.
Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew
RELATED CONCEPTS
LESSONS THAT TEACH THIS
PRACTICE THIS IN REAL QUESTIONS
Distributed Training & ParallelismDo the pipeline bubble math for me. How do micro-batches shrink it, and what does 1F1B change?→Distributed Training & ParallelismCompare data, tensor and pipeline parallelism. What does each one shard, what does each one communicate, and where does each one live?→Distributed Training & ParallelismYou have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.→Distributed Training & ParallelismIn data-parallel training, what actually gets communicated between GPUs, and how much is it per step?→Distributed Training & ParallelismWhat is MFU, how do you compute it from a running job, and what counts as a good number?→Distributed Training & ParallelismExplain ZeRO stages 1, 2 and 3. How much memory does each stage leave per GPU for a 70B model?→