TL;DR: A pipeline of p stages needs p−1 steps to fill and p−1 to drain, during which most stages idle; against m micro-batches of useful work the idle time is (p−1)/m of the compute time, or (p−1)/(m+p−1) of the whole step. PP16 with 64 micro-batches idles 23% of compute, 19% of the step. Raising m shrinks it but GPipe holds all m micro-batches' activations until backward starts; 1F1B interleaves one forward with one backward so a stage holds at most p micro-batches of activations, which is what makes large m affordable. Interleaved 1F1B and zero-bubble schedules push the bubble down further at the cost of more communication and more scheduling complexity.
How to approach it
Draw the timeline for a small p (three or four stages) so the fill and drain are visible, then count idle slots and turn the count into a fraction. Say both denominators, compute time and step time, and which one a paper is quoting. Then present the memory problem with GPipe and the fix with 1F1B, with a number for each. Close with the schedules beyond 1F1B and what each trades.
A strong answer
A typical situation: a 405B run uses PP16 across nodes, and a new engineer proposes going to PP32 to fit longer contexts. The bubble arithmetic is what says how much that costs.
Set up: p stages, each holding a contiguous slice of the layers, and a global batch split into m micro-batches. The forward pass of micro-batch j on stage s can start only when stage s−1 has finished it. In the simplest schedule (GPipe) all m forwards run, then all m backwards.
timeline for p = 4, m = 8, in units of one micro-batch forward on one stage (F) and
one backward (B, roughly 2F):
stage 1: F1 F2 F3 F4 F5 F6 F7 F8 . . . B8 B7 ... B1
stage 2: . F1 F2 F3 F4 F5 F6 F7 F8 . . . B8 ...
stage 3: . . F1 F2 F3 F4 F5 F6 F7 F8 . . . B8 ...
stage 4: . . . F1 F2 F3 F4 F5 F6 F7 F8 B8 B7 ...
stage 1 idles p−1 = 3 slots at the start of the forward fill,
and p−1 slots again while the backward drains back up to it;
every stage idles p−1 slots in each phase, just at different times.
inputs: p stages, m micro-batches, t_f forward time per micro-batch per stage,
t_b ≈ 2 t_f backward
ideal compute per stage = m (t_f + t_b)
bubble per stage = (p−1)(t_f + t_b)
bubble ÷ compute = (p−1)/m
bubble ÷ total step = (p−1)/(m + p − 1)
PP16, m = 64: (p−1)/m = 15/64 = 23.4% of compute; 15/79 = 19% of the step
PP16, m = 128: 15/128 = 11.7%; 15/143 = 10.5%
PP32, m = 64: 31/64 = 48%; 31/95 = 33%
PP8, m = 64: 7/64 = 10.9%; 7/71 = 9.9%
sanity: the proposal to go from PP16 to PP32 at fixed m roughly doubles the bubble,
from a fifth of the step to a third, and the run would need 128 micro-batches
to get back to where it was.
The two denominators matter when reading a paper: "bubble fraction" in the Megatron papers is (p−1)/m, and an efficiency of 1 − (p−1)/(m+p−1) is the same fact stated as utilization.
Why not just raise m. In GPipe's schedule every micro-batch's forward activations must be kept until its backward pass, and the backward passes start only after all m forwards complete. Stage 1 therefore holds m micro-batches of activations at the peak. Activation memory for one micro-batch of one stage is roughly the per-layer activation bytes times layers per stage; for a 405B (hidden 16,384, 126 layers, about 8 layers per stage at PP16) at 8k tokens with TP8 and selective checkpointing, call it 4 GB per micro-batch per stage. At m = 64 that is 256 GB, more than three cards, so GPipe cannot run this configuration at all.
1F1B (one forward, one backward, from PipeDream) changes the order, not the bubble. After the warm-up of p−1 forwards, each stage alternates one forward with one backward, so a micro-batch's backward runs as soon as it can, and its activations are freed. The number of micro-batches whose activations a stage holds at once is at most p (the first stage holds p, the last stage holds 1). Memory becomes independent of m:
activations in flight per stage:
GPipe: m micro-batches → 64 × 4 GB = 256 GB at m = 64 (does not fit)
1F1B: ≤ p micro-batches → 16 × 4 GB = 64 GB at stage 1 (fits, barely, on 80 GB)
bubble: unchanged at (p−1)/m for both schedules
That bound is what lets a run raise m to 128 or 256 to shrink the bubble without paying memory for it, and it is the default schedule in Megatron-LM and PyTorch's torch.distributed.pipelining. Pipeline Parallelism and the Bubble has the diagrams.
Beyond 1F1B, two families reduce the bubble itself. Interleaved 1F1B gives each GPU v non-contiguous chunks of layers (virtual stages), so the pipeline has p × v stages of 1/v the size each; the bubble becomes (p−1)/(m × v) at the cost of v times more point-to-point sends and a more tangled schedule. With v = 2 the PP16, m = 64 bubble drops from 23% to 12%. Zero-bubble schedules (ZB-H1, ZB-H2, and DeepSeek's DualPipe) split the backward pass into its input-gradient half and its weight-gradient half and use the weight-gradient work, which has no downstream dependency, to fill the bubble slots; they approach zero idle at the cost of holding weight gradients longer and, in DualPipe's case, two copies of the parameters.
Decision: 1F1B with the largest m the global batch allows and interleaving at v = 2 when the bubble is still above about 10%. The condition that reverses toward fewer stages is when the model fits with fewer: every stage removed takes its (t_f + t_b) out of the bubble for free, which is why PP is used at the smallest p that fits the layers and the rest of the scale goes to data parallelism.
The reversal condition: activation memory that will not allow more micro-batches. Then the bubble is the price of fitting at all, and Activation Checkpointing is the lever that buys the micro-batches back at a compute cost you can compute. p99 step time against the median across stages is where an unbalanced pipeline shows up before the bubble formula does.
What interviewers probe next
- "The stages are unbalanced; what happens?" The slowest stage sets the clock and every other stage idles the difference on every micro-batch, on top of the bubble; the embedding and output layers are the usual culprits and get their own stage or are split.
- "Does the bubble change the number of tokens per step?" No; it is idle time, so it lowers MFU at fixed batch rather than changing what is computed.
- "How does 1F1B interact with activation checkpointing?" Checkpointing shrinks the 4 GB per micro-batch figure, so the same p-micro-batch bound costs less memory; the two are complementary, not alternatives.
- "Why does the Llama 3 report use PP16 rather than fewer stages?" Because TP8 × PP16 is 128 GPUs holding one model replica of 405B, which leaves enough memory per GPU for the activations at 8k context; PP8 would need TP16, which crosses the node.
Common mistakes
- Quoting the bubble as a constant "about 10%" without p and m.
- Confusing the two denominators and reporting (p−1)/m as a fraction of the step, which overstates the loss.
- Saying 1F1B reduces the bubble; it reduces activation memory, which is what allows the m that reduces the bubble.
- Proposing more stages to fit a longer context without recomputing the bubble at the new p.
Key takeaways
- Bubble = (p−1)/m of compute time, (p−1)/(m+p−1) of the step; PP16 with 64 micro-batches is 23% and 19%.
- GPipe holds m micro-batches of activations at once; 1F1B holds at most p, which decouples memory from m.
- Interleaving divides the bubble by v at the cost of v times the sends; zero-bubble schedules fill it with weight-gradient work.
- Use the smallest p that fits the layers; every stage removed is free bubble reduction.