Measure batch readiness and producer capacity, then size prefetch for bursts without hiding a sustained throughput deficit.
This answer is included with a free account. Sign in to read it and save your progress.
free account · no card needed
← Networking, Interconnects & Storage / 11
medium★ EssentialMetaAnthropicDatabricks
The GPUs are idle between steps and the profiler says the data loader. Find the actual constraint and fix it.
Measure batch readiness and producer capacity, then size prefetch for bursts without hiding a sustained throughput deficit.
Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.
The concepts behind this question
Use these concept explanations to review the ideas behind this question.
Advanced
Data Loading Pipelines for TrainingTrace batches from storage to GPU readiness, budget workers and prefetch memory, and resume from the input position saved with training state. An executable example shows how a read-ahead cursor can silently skip samples.🔌 Networking & Storage🔒 Premium
Foundational
Dataset Lifecycle: Ingest, Shard and RetainFollow a dataset from source records to verified shards, a pinned version and safe retention. Calculate token storage, distinguish caches from authoritative data, and learn why deleting a version does not necessarily free its shards.🔌 Networking & Storage
Advanced
Producer-Consumer PipelinesProducer-consumer pipelines connect stages that run at different rates. Bounded buffers absorb temporary variation, while the slowest sustained stage limits throughput.💻 Coding for Infra🔒 Premium
Advanced
Evaluation and Data Pipeline InfrastructureData pipelines prepare traceable training inputs; evaluation pipelines compare model behavior. Both need versioned inputs, reproducible execution and useful failure reporting.📐 AI Systems Design🔒 Premium
UP NEXT ON YOUR JOURNEY
Next in this trackYou have 60 terabytes of filtered text and need 15 trillion training tokens. Design the tokenization and sharding stage.Next in this trackDesign the checkpoint write path for a 405B model on 16,384 GPUs. What has to be true for the pause to stay under a minute?Next in this trackMap a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?
FEDITOR'S NOTE
Count workers and prefetched batches across all ranks on a host. A modest per-loader setting can become a large node-wide allocation.
DISCUSSION · 0
No comments yet — be the first to share your approach.