fsdp

Track which state remains replicated at each ZeRO stage. Compute a declared 70B memory recipe, then add live gathers, activations and buffers before deciding fit.

Compare FSDP2 and ZeRO-3 through one parameter group’s lifetime, a scoped traffic model and a save-resume experiment, rather than a universal speed ranking.

Separate the 100B memory floor from the training deadline, then compare two 1,024-GPU layouts with their actual state ownership and communication schedules.

Construct a 16384-GPU candidate layout from ownership and batch constraints, with explicit sharding recipes and pipeline idle time.

Three routes with different costs: keep PyTorch through XLA, bet on the native backend, or adopt the JAX reference trainer. Choose from the model's custom code and the timeline, then prove parity before scaling.