object storage

One gives you POSIX semantics and hundreds of gigabytes per second at a price per terabyte that hurts at petabyte scale. The other is cheap and durable and answers a first byte in tens of milliseconds. The cost arithmetic that makes most clusters run both, and the metadata limit that decides how you shard.

The bandwidth is effectively unlimited and the first byte takes fifty milliseconds, so throughput is set by how many requests you keep in flight rather than by the network. Little's law applied to a data loader, the request size where the latency stops mattering, and the two design changes that follow.

Every 30 minutes a thousand GPUs write a terabyte in a burst that must finish in a minute, then nothing until the next. The burst arithmetic that sizes the write tier, the two-stage path to durable storage, retention that keeps the right checkpoints, resharding so a 512-GPU checkpoint resumes on 256, and lineage.