13Design the checkpoint write path for a 405B model on 16,384 GPUs. What has to be true for the pause to stay under a minute?▼hardNewMetaMicrosoft4 replies○ sign inSix and a half terabytes leaving sixteen thousand GPUs at once is a burst no shared filesystem absorbs, and the fix is to stop trying. What the barrier actually has to wait for, the per-node arithmetic that makes it seconds, and the manifest rule that decides whether a checkpoint is usable at all.Open full answer →
25Checkpoints and datasets both live on storage. Why does one system sized for both usually serve neither well?▼mediumNewCrusoeWEKA4 replies◆ premiumOne is a write burst of terabytes in a minute followed by half an hour of silence; the other is a steady read that never stops and never spikes. Provisioning for the peak of the first wastes most of its capacity, and provisioning for the average of the second fails the moment a checkpoint lands.Open full answer →