Distributed Training Interview Questions

Count gradient traffic from a named dtype, separate initialization and buffer broadcasts, and measure the reduction tail instead of assuming it is hidden.

Show what each parallelism axis owns, then derive the groups and traffic for an eight-rank layout. Placement follows measured communication and memory constraints.

Compute MFU from a declared global token count and time interval. Check the accounting before judging the percentage or choosing an optimization.

Track which state remains replicated at each ZeRO stage. Compute a declared 70B memory recipe, then add live gathers, activations and buffers before deciding fit.

Compare FSDP2 and ZeRO-3 through one parameter group’s lifetime, a scoped traffic model and a save-resume experiment, rather than a universal speed ranking.

Separate the 100B memory floor from the training deadline, then compare two 1,024-GPU layouts with their actual state ownership and communication schedules.

Trace a chunk, derive both traffic phases, and state the endpoint model behind bandwidth optimality. A worked example shows why nearly constant bytes do not imply constant completion time.

Trace a legal two-stage schedule and count its idle slots. Separate bubble denominators and explain how 1F1B changes activation lifetimes under explicit assumptions.

Count tensor-parallel messages and compare their payload service time on two explicit paths. Explain why TP8 is a common placement choice, not a fixed ceiling.

Derive the weighted gradient, test a short micro-batch, and separate activation savings from total memory and communication. Matching the optimizer step is only part of matching a larger batch.

Follow saved and recomputed values through backward. Compare complete update time and peak memory instead of promising a fixed memory saving or slowdown.

Trace a causal query across local and remote KV blocks. Separate sequence and context sharding, count one concrete tensor, and avoid dividing memory twice.

Follow one token to two expert ranks and back, then count logical assignments separately from physical network transfers.

Use a ten-expert example to separate assignments, rank load and dropped routes, then compare balancing controls against quality.

Construct a 16384-GPU candidate layout from ownership and batch constraints, with explicit sharding recipes and pipeline idle time.

Separate gradient range from update precision using a small numerical example, and make the training-state recipe explicit.

Use per-rank arrival times and controlled substitutions to explain a slow step without mistaking collective wait for network transfer.

Distinguish stale telemetry, skipped updates and a real learning plateau using counters, parameter changes and a fixed evaluation batch.

Balance checkpoint pauses against lost work using job-level interruption data, while accounting for asynchronous durability lag.

Treat recovery as a new worker generation, then compare spare replacement with a legal shrink and its batch consequences.

Separate a longer step from lower throughput, trace the dependency that changed, and distinguish input, workload, device and path causes. Then calculate whether a repair recovers more time than it costs.

Read a watchdog timeout without blaming the reporting rank. Compare group-scoped traces, separate missing evidence from mismatched calls, and distinguish CPU submission from GPU completion.

Overlap starts with a dependency: communication must finish before its consumer needs the result. Estimate the available window, then measure readiness, contention and complete step time for DDP, FSDP or tensor parallelism.

A global-array program lets the compiler derive communication from sharding choices. Work through a small matrix example, then separate that programming model from TPU topology, compilation and Pathways recovery.

Separate packed base weights, quantization metadata and trainable adapter state. Count the real projection shapes, then leave room for activations and temporary work before calling a one-card configuration feasible.

Separate generation, scoring, learning and policy publication. Size the pools from matched workload measurements, and make policy versions explicit before overlapping rollout generation with updates.

Offline DPO trains on existing preference pairs, so online rollout and reward services leave the update loop. Account for both responses, decide whether reference scores can be cached, and retain the data-generation and evaluation costs outside training.

GRPO replaces a learned value baseline with comparisons within response groups. Size generation and verification from the actual group workload, then distinguish continuous batching from changes to the training algorithm.

Separate FP8 input quantization from accumulation error. A small representable-value example shows when a shared scale loses small values, while the DeepSeek-V3 report supplies one carefully scoped training recipe.

Describe logical tensors independently of writer ranks, publish only a complete checkpoint, and reconstruct the target layout from stored chunks. Asynchronous saving reduces blocking but still needs a consistent snapshot and a memory budget.

Turn the daily loss into recovery and replay costs, then identify which part is measured and which is assumed. Compare checkpoint cadence, recovery work and failure prevention without promising one universal floor.

Build a training schedule from explicit model, token, state and recovery assumptions. Keep active compute time separate from availability and contingency.

Version teacher outputs, separate learning from evaluation, and catch a release whose domain score improves while a protected capability regresses.

Three routes with different costs: keep PyTorch through XLA, bet on the native backend, or adopt the JAX reference trainer. Choose from the model's custom code and the timeline, then prove parity before scaling.