Google DeepMind Distributed Training & Parallelism interview questions

A global-array program lets the compiler derive communication from sharding choices. Work through a small matrix example, then separate that programming model from TPU topology, compilation and Pathways recovery.

Three routes with different costs: keep PyTorch through XLA, bet on the native backend, or adopt the JAX reference trainer. Choose from the model's custom code and the timeline, then prove parity before scaling.

Count gradient traffic from a named dtype, separate initialization and buffer broadcasts, and measure the reduction tail instead of assuming it is hidden.

Compute MFU from a declared global token count and time interval. Check the accounting before judging the percentage or choosing an optimization.

Compare FSDP2 and ZeRO-3 through one parameter group’s lifetime, a scoped traffic model and a save-resume experiment, rather than a universal speed ranking.

Count tensor-parallel messages and compare their payload service time on two explicit paths. Explain why TP8 is a common placement choice, not a fixed ceiling.

Show what each parallelism axis owns, then derive the groups and traffic for an eight-rank layout. Placement follows measured communication and memory constraints.

Track which state remains replicated at each ZeRO stage. Compute a declared 70B memory recipe, then add live gathers, activations and buffers before deciding fit.

Separate the 100B memory floor from the training deadline, then compare two 1,024-GPU layouts with their actual state ownership and communication schedules.

Trace a legal two-stage schedule and count its idle slots. Separate bubble denominators and explain how 1F1B changes activation lifetimes under explicit assumptions.

Derive the weighted gradient, test a short micro-batch, and separate activation savings from total memory and communication. Matching the optimizer step is only part of matching a larger batch.

Trace a chunk, derive both traffic phases, and state the endpoint model behind bandwidth optimality. A worked example shows why nearly constant bytes do not imply constant completion time.

Follow one token to two expert ranks and back, then count logical assignments separately from physical network transfers.

Treat recovery as a new worker generation, then compare spare replacement with a legal shrink and its batch consequences.

Follow saved and recomputed values through backward. Compare complete update time and peak memory instead of promising a fixed memory saving or slowdown.

Trace a causal query across local and remote KV blocks. Separate sequence and context sharding, count one concrete tensor, and avoid dividing memory twice.

Use per-rank arrival times and controlled substitutions to explain a slow step without mistaking collective wait for network transfer.