storage

Capacity is the easy half and almost never the constraint. The number that sizes the system is the checkpoint write burst, which is thousands of times the steady dataset read, and the tier that serves each is different. The full derivation, and the tiering that avoids paying burst prices for archive bytes.

Most architecture diagrams show one network and a cluster has four. Each has a different traffic pattern, a different failure consequence and a different port count, and the one that is under-counted every time is the one you need when everything else is broken.

Sixteen bytes per parameter is 6.5 TB per checkpoint; over a 100 GB/s filesystem that is about a minute, which at a 30-minute cadence is 3% of the run. The chain, the storage tiers that meet it, and why the write is made asynchronous.

The token stream is tens of megabytes a second and the checkpoints are tens of gigabytes a second; the two workloads differ by a thousand and want different tiers. The chain from tokens per second to bytes, the checkpoint burst, and the restore that sets the real spec.

A 141 GB artifact is not a Docker image, and shipping it like one costs an hour per node and a registry that melts under a rollout. Content-addressed chunks, a manifest per version, a fan-out that turns one origin into a thousand nodes in minutes, integrity that fails closed, and promotion tied to evaluation.

Every 30 minutes a thousand GPUs write a terabyte in a burst that must finish in a minute, then nothing until the next. The burst arithmetic that sizes the write tier, the two-stage path to durable storage, retention that keeps the right checkpoints, resharding so a 512-GPU checkpoint resumes on 256, and lineage.

Metrics arrive in time order and are queried by time window, which is the pair of facts the whole design turns on. Sealed segments with a sparse index, pruning that skipped 82 of 84 segments in the executed test, and TTL compaction that drops whole files without rewriting a byte.