How do you overlap communication with compute in training, and where does the overlap stop working?

Overlap starts with a dependency: communication must finish before its consumer needs the result. Estimate the available window, then measure readiness, contention and complete step time for DDP, FSDP or tensor parallelism.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

GPUs and time-to-train calculator

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational

🧭 Ownership & Judgment

Talking About Cost and Capacity with LeadershipA capacity discussion should connect technical measurements to cost, delivery dates and risk. Present the assumptions, feasible options and a recommendation, and distinguish committed capacity from actual consumption under the relevant contract.

Foundational

🧭 Ownership & Judgment

Escalation That WorksEscalation helps a responsible owner make a decision that cannot be resolved at the current level. State the decision needed, the options and their costs, your recommendation, and what happens if the deadline passes.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPReplicas combine gradients from different data. Follow a two-worker update, fix unequal-token weighting, and see why queued gradient buckets can leave more than one reduction exposed.

Core

🕸️ Distributed TrainingSign in

ZeRO and FSDPZeRO and FSDP reduce replicated training state by distributing it across workers. The memory saving comes with communication and temporary-buffer costs.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow is training on TPUs with JAX different from training on GPUs with PyTorch? What do you stop doing by hand?Next in this trackFine-tune a 70B on one 80 GB card. What do NF4 and double quantization actually buy, and where does DoRA change the arithmetic?Next in this trackDesign the infrastructure for RLHF on a 70B: where do rollouts and the learner run, and how do weights move between them every step?

FEDITOR'S NOTE

Explain the dependency that permits overlap, derive a scoped service-time ratio and distinguish partial overlap from full hiding. Include readiness, live memory and resource contention in the decision.

DISCUSSION · 0

No comments yet — be the first to share your approach.