12You have 60 terabytes of filtered text and need 15 trillion training tokens. Design the tokenization and sharding stage.▼hardNewMetaDatabricksAnthropic4 replies○ sign inA tokenizer moves about a megabyte of text per second per core, which makes this a seven-hundred-core-day batch job rather than something to run during training. The throughput arithmetic per stage, the shard format the loader needs, and the determinism requirements that let you resume without corrupting a run.Open full answer →
13Design the checkpoint write path for a 405B model on 16,384 GPUs. What has to be true for the pause to stay under a minute?▼hardNewMetaMicrosoft4 replies○ sign inSix and a half terabytes leaving sixteen thousand GPUs at once is a burst no shared filesystem absorbs, and the fix is to stop trying. What the barrier actually has to wait for, the per-node arithmetic that makes it seconds, and the manifest rule that decides whether a checkpoint is usable at all.Open full answer →
