Advanced

Data Loading Pipelines for Training

The dataloader is the only part of a training job that runs on the CPU, the disk and the network at once, and it is the part most often found starving the GPUs. A pipeline that keeps 1,024 accelerators fed has to read sharded files sequentially, decode and tokenize in parallel workers, prefetch several batches ahead, pin memory for the PCIe copy, and do it deterministically enough to resume mid-epoch. The symptom of failure is a GPU at 30% utilization with nothing wrong on the GPU.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Parallel Filesystems vs Object Storage GPUDirect RDMA and GPUDirect Storage MFU and HFU NVLink, NVSwitch and PCIe Evaluation and Data Pipeline Infrastructure Checkpoint I/O

LESSONS THAT TEACH THIS

Training at ScaleA training step is four costs and only one of them is the model13 min→Fleet and ReliabilityStorage is a throughput problem until the files get small14 min→

PRACTICE THIS IN REAL QUESTIONS

Networking, Interconnects & StorageThe GPUs are idle between steps and the profiler says the data loader. Find the actual constraint and fix it.→Networking, Interconnects & StorageYou have 60 terabytes of filtered text and need 15 trillion training tokens. Design the tokenization and sharding stage.→Networking, Interconnects & StorageWhy does object storage behave badly for random reads, and what does a training pipeline do about it?→AI Infrastructure System DesignDesign the pipeline that produces 15 trillion training tokens: ingest, dedup, tokenize, shard, serve. Throughput per stage.→Napkin Math, Cost & CapacityWhat storage throughput does a 16k-GPU training cluster need?→CUDA, Triton & Kernel EngineeringA training step runs at 20 percent model FLOPs utilization. Profile it and find where the missing time goes.→