Gradient accumulation versus a bigger per-GPU batch: same result or not, and what changes underneath?

TL;DR: Accumulation can reproduce a larger batch's gradient when the loss decomposes into independent contributions, weights stay fixed across micro-batches, and normalization uses the full batch denominator. You accumulate gradients and step once. Activation peaks may fall, but runtime, randomness and batch-dependent operations can differ. In DDP, deferring synchronization requires no_sync() around both forward and backward on non-final passes.

How to approach it

Keep the global batch fixed while comparing the two execution plans. Derive the gradient weighting, then separate activation memory, communication and optimizer timing. Close with an example that breaks equivalence, such as BatchNorm or in-batch negatives.

A strong answer

Suppose one rank must process eight examples for an optimizer update, but only two at a time fit comfortably. It can run four micro-batches without changing the weights between them, accumulate gradients, and update once. A micro-batch is one forward/backward unit; the effective batch includes all examples contributing to the update across accumulation passes and data-parallel ranks.

For four equal-sized micro-batches with mean losses L1 through L4:

L = (L1 + L2 + L3 + L4) / 4
gradient(L) = (gradient(L1) + ... + gradient(L4)) / 4

Dividing each already-mean micro-batch loss by four is correct here. It is not accidental double averaging: the loss function averages within a micro-batch, and the additional division weights that micro-batch within the full batch. Summing four unscaled mean gradients would instead multiply the intended gradient by four.

Unequal valid-token counts need different weights. Let one micro-batch contain two valid tokens with mean gradient 1, and another contain six with mean gradient 3. The correct token-mean gradient is (2×1 + 6×3)/8 = 2.5. Averaging the two means gives 2 and overweights the shorter micro-batch. Padding and masked targets make this case common enough to test explicitly.

Here is a runnable scalar example. It computes gradients of mean((w*x-y)**2) with respect to w, keeping w fixed across the micro-batches:

from math import isclose

def mean_gradient(w, examples):
    return sum(2 * x * (w * x - y) for x, y in examples) / len(examples)

micro_batches = [[(1, 0)], [(2, 0), (3, 0), (4, 0)]]
w = 1.0
all_examples = [xy for batch in micro_batches for xy in batch]
large = mean_gradient(w, all_examples)
weighted = sum(
    len(batch) * mean_gradient(w, batch) for batch in micro_batches
) / len(all_examples)
wrong = sum(mean_gradient(w, batch) for batch in micro_batches) / 2
assert isclose(large, weighted)
assert not isclose(large, wrong)
print(f"large={large:.2f}, accumulated={weighted:.2f}, wrong={wrong:.2f}")
# large=15.00, accumulated=15.00, wrong=10.67

The example assumes nonempty micro-batches. It demonstrates normalization, not a GPU training loop. With DDP's default rank averaging, extend the denominator across all ranks too: for a total valid count T over n ranks, scaling each local summed loss by n/T makes the subsequent rank average produce the global token mean.

For a hypothetical memory budget, suppose persistent and fixed working allocations take 12 GB, while activations for eight examples take 8 GB. If the activation term scales linearly, two-example micro-batches reduce it to 2 GB: peak falls from 20 GB to 14 GB, not to 5 GB. Actual kernels, sequence lengths and temporary buffers can break linear scaling. Record torch.cuda.max_memory_allocated() over the complete update and distinguish it from reserved allocator memory and device-wide usage.

With Data Parallelism and DDP, wrap the first three forward/backward pairs in no_sync() and let the fourth synchronize. Clear gradients before the accumulation window, not between its micro-batches. All ranks must agree on when that final synchronized pass happens.

If one set of gradient reductions sends 7 GB per rank, four synchronized micro-batches send 28 GB. Deferring them sends 7 GB. One eight-example batch also sends 7 GB under the same reduction layout. This is a 4× reduction relative to naive accumulation, not relative to the larger batch. Remaining buffer synchronization or layer-specific communication is outside this gradient count. FSDP has different parameter-gather and gradient-storage behavior; DDP's savings cannot simply be transferred to it.

Where equivalence breaks

BatchNorm computes statistics over the samples it sees together. Four small forwards are different from one large forward; cross-rank synchronization does not combine statistics across time. Contrastive losses using in-batch negatives also change when the batch is split. LayerNorm and RMSNorm do not mix examples that way. Other parts of the loss may still couple examples, so check the complete objective.

Dropout and other random operations can draw different values after changing execution shape. Floating-point summation order also changes. Expect the same intended objective under the stated assumptions, not guaranteed bitwise-identical updates or training trajectories. Clipping each micro-batch separately, stepping the optimizer between them, or changing a loss-scaling factor halfway through the window breaks the simple derivation.

At fixed global batch, benchmark feasible micro-batch sizes and accumulation counts. Larger micro-batches may use kernels more efficiently, while smaller ones leave memory for longer sequences or a larger model. If shrinking the micro-batch makes step time worse, compare activation checkpointing or state sharding before assuming more accumulation is the answer.

What interviewers probe next

• What about the learning-rate schedule? Its clock is whatever the recipe specifies, often optimizer steps or consumed tokens. Advance it consistently with that clock, not automatically on every micro-batch.
• What about a short final window? Normalize by its actual examples or tokens. Dividing by a planned four passes when only two occur halves the gradient incorrectly.
• Can accumulation fix an optimizer-state OOM? No. Model Memory Footprint distinguishes persistent state from activations; sharding or changing the optimizer may be needed.
• Does one GPU become as fast as four? No. Accumulation serializes micro-batches on that device; equivalence of the objective does not imply equivalence of wall time.

Common mistakes

• Clearing gradients between micro-batches instead of once per update.
• Placing only backward inside no_sync() and still triggering synchronization.
• Dividing a short final window by the planned accumulation count.
• Comparing communication at different effective batch sizes without saying so.

Key takeaways

• Weight micro-batch gradients by their actual loss denominator; the unequal-size example gives 15.00, not 10.67.
• Accumulation can shrink the activation term without shrinking the whole memory footprint by k.
• At a fixed effective batch, deferred DDP accumulation restores one set of gradient reductions per update.
• Verify a tiny combined-batch comparison before timing the larger run.