How do you overlap host-to-device transfers with compute, and what has to be true before the overlap actually happens?

An asynchronous call returns early, but the timeline still shows copies and kernels taking turns. Derive the pipeline's fill and drain costs, then check dependencies, buffers and hardware before expecting overlap.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

Roofline calculator

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

⚡ Kernels & Compilers🔒 Premium

Shared Memory and Bank ConflictsShared memory lets threads in a block exchange and reuse data. Bank conflicts occur when an access needs multiple distinct words from the same bank.

Foundational

⚡ Kernels & Compilers

CUDA Programming ModelCUDA programs launch GPU kernels from host code. Thread indexing, memory access and stream ordering determine whether the result is correct.

Advanced

🔌 Networking & Storage🔒 Premium

Data Loading Pipelines for TrainingTrace batches from storage to GPU readiness, budget workers and prefetch memory, and resume from the input position saved with training state. An executable example shows how a read-ahead cursor can silently skip samples.

Advanced

💻 Coding for Infra🔒 Premium

Asynchronous GPU Buffer OwnershipA host function can return while a GPU still uses its pointers. Tie buffer reuse to device completion, including cancellation, cross-stream consumers and foreign-function boundaries.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhy do tensor cores accumulate in fp32 when the inputs are bf16, and why do optimizers keep fp32 master weights?Next in this trackIn an int8 or fp8 GEMM, what exactly gets scaled, where does the dequantization happen, and what breaks if you get it wrong?Next in this trackA mixture-of-experts layer is 256 small matmuls. Why is a loop over cuBLAS calls slow, and what does a grouped GEMM do instead?

FEDITOR'S NOTE

Derive finite-pipeline time, state who owns each buffer and verify concurrency in a trace. Distinguish asynchronous submission from completed overlap and stream priority from preemption.

DISCUSSION · 0

No comments yet — be the first to share your approach.