The GPUs are idle between steps and the profiler says the data loader. Find the actual constraint and fix it.

For text training the bytes are trivial and the bottleneck is never bandwidth, so the usual advice about faster storage misses. What the loader actually has to deliver per second, the four things that consume the time instead, and the order to fix them with the measurement that proves each.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

Data Loading Pipelines for TrainingThe dataloader is the only part of a training job that runs on the CPU, the disk and the network at once, and it is the part most often found starving the GPUs. A pipeline that keeps 1,024 accelerators fed has to read sharded files sequentially, decode and tokenize in parallel workers, prefetch several batches ahead, pin memory for the PCIe copy, and do it deterministically enough to resume mid-epoch. The symptom of failure is a GPU at 30% utilization with nothing wrong on the GPU.

Foundational

🔌 Networking & Storage

Dataset Lifecycle: Ingest, Shard and RetainA training dataset is not a file, it is a pipeline with four stages and a retention policy, and each stage has a different bottleneck. Ingest is metadata-bound rather than bandwidth-bound. Tokenization is CPU work that should happen once offline rather than every epoch. Sharding decides whether the training read is a stream or a storm of small files. And retention decides how much of the bill is paid for bytes nobody reads.

Advanced

💻 Coding for Infra🔒 Premium

Producer-Consumer PipelinesA data loader, a log shipper, a batch inference job and a checkpoint writer are the same program: stages connected by bounded buffers, each running at its own pace, the slowest setting the throughput and the buffers absorbing the jitter between them. The coding screen asks you to build one (read, decode, batch, feed a consumer) and then pushes on the production questions: buffer sizes, clean stops, failure propagation, and why it runs at a third of the expected speed. This page derives throughput from stage times, implements the pipeline in threads and asyncio, and works the stop and failure semantics.

Advanced

📐 AI Systems Design🔒 Premium

Evaluation and Data Pipeline InfrastructureBehind every model release is a pipeline that turns raw text into training shards and a harness that runs thousands of evaluation prompts against every checkpoint, and both are infrastructure problems with GPU-sized budgets. The data side is a batch system: dedup, filter, tokenize and shard petabytes with lineage. The eval side is a serving system in disguise: run a benchmark suite against a checkpoint in minutes, on shared GPUs, reproducibly, with results a researcher can trust. This page designs both, derives the compute and storage they need, and gives the reproducibility rules that separate a real harness from a script.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYou have 60 terabytes of filtered text and need 15 trillion training tokens. Design the tokenization and sharding stage.Next in this trackDesign the checkpoint write path for a 405B model on 16,384 GPUs. What has to be true for the pause to stay under a minute?Next in this trackMap a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?

FEDITOR'S NOTE

Scored on computing the required byte rate and finding it trivial, on identifying CPU work and per-file overhead as the real constraints, and on the fix order with a measurement for each step.

DISCUSSION · 0

No comments yet — be the first to share your approach.