← Distributed Training
Advanced
Pipeline Parallelism and the Bubble
Pipeline parallelism assigns groups of layers to different devices. Microbatches keep stages busy, while the schedule controls idle periods and activation memory.
This concept is part of Premium, which includes all concepts, course lessons and answers.
Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew
RELATED CONCEPTS
LESSONS THAT TEACH THIS
PRACTICE THIS IN REAL QUESTIONS
Distributed Training & ParallelismDo the pipeline bubble math for me. How do micro-batches shrink it, and what does 1F1B change?→Distributed Training & ParallelismCompare data, tensor and pipeline parallelism. What does each one shard, what does each one communicate, and where does each one live?→Distributed Training & ParallelismYou have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.→Distributed Training & ParallelismIn data-parallel training, what actually gets communicated between GPUs, and how much is it per step?→Distributed Training & ParallelismWhat is MFU, how do you compute it from a running job, and what counts as a good number?→Distributed Training & ParallelismExplain ZeRO stages 1, 2 and 3. How much memory does each stage leave per GPU for a 70B model?→