TL;DR: With balanced stages and idealized communication, GPipe takes (m+p−1)(t_f+t_b) for m micro-batches over p stages. Idle time divided by busy time is (p−1)/m; divided by elapsed time it is (p−1)/(m+p−1). A 1F1B schedule, which alternates one forward with one backward and finishes the batch before updating weights, can shorten activation lifetimes, but it does not automatically remove the fill/drain cost or guarantee memory fit.
How to approach it
Draw a small schedule in which each operation waits for its required inputs before quoting the formula. State stage balance, transfer and forward/backward timing assumptions. Count both denominators, then discuss activation lifetime separately from total batch size and throughput.
A strong answer
Pipeline parallelism puts consecutive layers on different GPUs (stages) and feeds them small slices of the batch, called micro-batches. GPipe runs every micro-batch's forward pass before any backward pass; 1F1B (one forward, one backward) lets each stage alternate between them once the pipeline is full. The bubble is the time stages sit idle waiting for work. A proposal doubles pipeline stages to fit more layers. The model fits more easily, but that does not mean the job finishes sooner. Pipeline Parallelism and the Bubble explains the dependency chain; Activation Checkpointing addresses a different memory term.
Start with two balanced stages and two micro-batches, with forward and backward each taking one slot and transfers treated as free. GPipe runs all forwards before backwards. The timeline below is one valid flush schedule, with no optimizer update between its micro-batches.
Two stages, two microbatches; forward and backward each take one slot. Transfers are free in this toy.
Microbatch 1 dependency path
Slot boundaries; full schedule ends at 6
Stage 0: all six slots
- F1 → F2 → idle → idle → B2 → B1
Stage 1: all six slots
- idle → F1 → F2 → B2 → B1 → idle
Micro-batch 1 runs forward on stage 0, then stage 1; its backward returns in the opposite order. A stage can idle at a different point from its neighbor. In particular, stage 0 starts the first forward immediately; it does not idle at the beginning merely because the pipeline is filling.
general balanced GPipe model:
busy time per stage = m×(t_f+t_b)
elapsed time = (m+p−1)×(t_f+t_b)
idle time per stage = (p−1)×(t_f+t_b)
idle / busy = (p−1)/m
idle / elapsed = (p−1)/(m+p−1)
p=2, m=2, t_f=t_b=1: busy=4, elapsed=6, idle=2
sanity: busy + idle = elapsed; 4+2=6.
For p=16,m=64, idle/busy is 15/64=23.44%, while idle/elapsed is 15/79=18.99%. At p=32,m=64 the latter becomes 31/95=32.63%. These are ideal schedule fractions, not measured utilization. Unequal stage times, communication, optimizer work and overlap policy alter actual timing.
GPipe may retain forward activations for all m micro-batches on an early stage before backward frees them. In a standard non-interleaved flush 1F1B schedule, warm-up is followed by alternating forward and backward work, bounding live micro-batches by the pipeline depth rather than m when m is large. Exact peaks differ by stage, schedule and implementation. Megatron's pipeline study discusses the memory and scheduling trade.
| Change | Tradeoff to test |
|---|---|
| More micro-batches | Smaller ideal bubble, but smaller kernels at fixed global batch |
| 1F1B | Earlier release, with scheduling and communication constraints |
| Checkpoint activations | Fewer saved tensors for recomputation work |
| Fewer stages | Less fill/drain but more state and work per stage |
For a synthetic four-GB activation payload per live micro-batch, GPipe with m=64 could require 256 GB on a stage holding all of them. A bound of p=16 live micro-batches would instead be 64 GB. That is not proof of fit on an 80 GB card: parameters, gradients, optimizer and buffers remain. The four-GB input is an assumed payload, not a measured 405B recipe.
At a fixed useful global batch, increasing m reduces micro-batch size. More kernel launches and worse matrix shapes may outweigh the smaller bubble. Changing m by increasing global batch can change optimization behavior. Keep these experiments distinct.
Interleaved and zero-bubble families can use additional scheduling freedom, including separation of input-gradient and weight-gradient work. Their gains depend on balance, dependencies, memory and communication; there is no universal “use interleaving above 16 stages” threshold. Inspect the selected framework's actual schedule rather than treating every 1F1B variant as identical.
Check a schedule instead of trusting the picture
Extend the toy to three microbatches. Here is an executable CPU check for the two legal eight-slot schedules shown in Pipeline Parallelism and the Bubble. Every operation lasts one slot and becomes available to the next stage in the following slot. The check rejects duplicate work, backward before local forward and cross-stage dependencies that finish too late.
def check(rows, microbatches):
done, live = set(), [set() for _ in rows]
peak = [0 for _ in rows]
assert len({len(row.split()) for row in rows}) == 1
for slot in zip(*(row.split() for row in rows)):
newly_done = set()
for stage, op in enumerate(slot):
if op == ".":
continue
kind, micro = op[0], int(op[1:])
assert kind in ("F", "B") and 1 <= micro <= microbatches
key = (stage, kind, micro)
assert key not in done
if kind == "F":
assert stage == 0 or (stage-1, "F", micro) in done
live[stage].add(micro)
peak[stage] = max(peak[stage], len(live[stage]))
else:
assert (stage, "F", micro) in done
assert stage == len(rows)-1 or (stage+1, "B", micro) in done
live[stage].remove(micro)
newly_done.add(key)
done.update(newly_done)
assert len(done) == 2 * len(rows) * microbatches
assert not any(live)
return peak
all_forward = ["F1 F2 F3 . . B3 B2 B1", ". F1 F2 F3 B3 B2 B1 ."]
one_f_one_b = ["F1 F2 . B1 F3 B2 . B3", ". F1 B1 F2 B2 F3 B3 ."]
assert check(all_forward, 3) == [3, 3]
assert check(one_f_one_b, 3) == [2, 1]
print("peak live activation sets: [3, 3] versus [2, 1]; both 8 slots")
Move stage 0's B1 from slot 3 to slot 2 and the dependency check fails: stage 1's B1 has not finished. The model deliberately excludes transfer time, backward workspace and optimizer work. It verifies a legal dependency order and saved-set count, not a production memory peak.
I would compare complete update time and per-stage memory using a trace. Balance measured work, not just layer counts. Reverse an extra-stage or micro-batch change if its communication, kernel efficiency or memory costs outweigh the scheduling benefit. A single median/p99 step-time ratio cannot identify stage imbalance; inspect stage-local work and dependencies.
What interviewers probe next
- “Does 1F1B change the optimizer update?” A flush schedule can preserve one update after the full batch. Asynchronous schedules with weight versions have different semantics and must be named.
- “Does less saved activation mean less total memory?” It lowers that term; peak memory still includes live state, workspaces and communication buffers.
- “Can the observed bubble differ?” Yes. Define what the trace counts as idle and which interval it covers before comparing it with the model.
Common mistakes
- Reporting idle/busy as the fraction of elapsed time.
- Drawing a backward before its activation/gradient dependencies exist.
- Calling more stages or more micro-batches a free improvement.
Key takeaways
- The two-stage toy is idle 2/6 of elapsed time, but 2/4 of busy time.
- PP16 with 64 micro-batches has an ideal 18.99% elapsed-time bubble.
- Activation lifetime, global batch and actual throughput need separate checks.