llama 3

Tensor parallel 8, pipeline 16, data parallel 128: the layout Meta trained Llama 3 405B on, rebuilt from the memory floor, the compute budget and the bandwidth of each link. Why the axes go in that order, what each one costs per step, and the bubble that the schedule has to hide.

Meta's Llama 3 report counted 419 unexpected interruptions in 54 days on 16,384 GPUs, one every three hours. The failure-rate arithmetic, the loss as a function of the interval, the square-root formula that minimizes it, and why the write must be sharded and asynchronous first.