Core

Parallel Filesystems vs Object Storage

A training cluster's storage has two very different jobs: stream terabytes of training data to thousands of GPUs at a steady rate, and absorb a multi-terabyte checkpoint burst every few minutes. Parallel filesystems (Lustre, GPFS, WEKA, VAST, FSx) give POSIX semantics and hundreds of GB/s of aggregate throughput; object storage (S3 and its equivalents) gives durability and cost at a fraction of the price with high first-byte latency. Almost every real cluster uses both, and the interview question is which job goes where and how big each tier has to be.

a free account unlocks the core curriculum tier · no card

RELATED CONCEPTS

Data Loading Pipelines for Training Checkpoint I/O GPUDirect RDMA and GPUDirect Storage Training Cluster Design at 10k GPUs Capacity Planning and Utilization TCO: Buy vs Rent

LESSONS THAT TEACH THIS

Fleet and ReliabilityStorage is a throughput problem until the files get small14 min→Fleet and ReliabilityIsolation is what you sell, and the levels guarantee different things15 min→

PRACTICE THIS IN REAL QUESTIONS

AI Infrastructure System DesignDesign the checkpoint store for a lab running several large training jobs: write bursts, retention, resharding and lineage.→Networking, Interconnects & StorageParallel filesystem or object storage for training data? Make the call and explain the two-tier pattern most clusters end up with.→Networking, Interconnects & StorageWhy does object storage behave badly for random reads, and what does a training pipeline do about it?→Napkin Math, Cost & CapacityEstimate how long it takes to write a checkpoint for a 405B training run→AI Infrastructure System DesignDesign the eval pipeline for a frontier model: thousands of evals per checkpoint, sharded inference, caching, reproducible results.→Napkin Math, Cost & CapacityWhat storage throughput does a 16k-GPU training cluster need?→