20Design a training system that survives losing a node without a human in the loop. What does elasticity cost you?▼hardNewAnyscaleMetaCrusoe4 replies○ sign inA dead node at 4 a.m. costs twenty minutes if a human restarts it and under five if the system does. The four stages of an automatic recovery, the reason a lost node takes a whole pipeline replica with it, why hot spares beat resharding, and the minutes that each stage still costs even when everything works.Open full answer →
30Design a checkpoint format for thousands of GPUs: no gather on write, resumable at a different world size, no stall.▼hardNewMetaMicrosoftDatabricks4 replies◆ premiumEvery rank writes its own shard, a metadata file maps every chunk to a global tensor, and a loader intersects the old layout with the new. The bytes per rank for a 405B, the manifest that makes the write atomic, the metadata that makes resharding possible, and the host-memory staging that keeps the GPUs running.Open full answer →
18Design the checkpoint store for a lab running several large training jobs: write bursts, retention, resharding and lineage.▼hardNewMetaAnthropic4 replies○ sign inEvery 30 minutes a thousand GPUs write a terabyte in a burst that must finish in a minute, then nothing until the next. The burst arithmetic that sizes the write tier, the two-stage path to durable storage, retention that keeps the right checkpoints, resharding so a 512-GPU checkpoint resumes on 256, and lineage.Open full answer →
