Core

ZeRO and FSDP

ZeRO and FSDP keep data parallelism's simple programming model but shard the optimizer state, gradients and parameters across ranks, cutting per-GPU memory from 16 bytes per parameter toward 16/N. The price is 1.5x DDP's communication and a dependence on tokens per GPU that decides when sharding stops paying and tensor parallelism takes over.

a free account unlocks the core curriculum tier · no card

RELATED CONCEPTS

Data Parallelism and DDP Tensor Parallelism Collective Communication Primitives Activation Checkpointing Model Memory Footprint Communication Volume Estimates

LESSONS THAT TEACH THIS

Training at ScaleMemory decides your parallel layout before you get an opinion14 min→Training at ScalePrice a parallel dimension by the collective it buys, not by its name15 min→Training at ScaleSharding is what makes a large model fit, and it is not free15 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismExplain ZeRO stages 1, 2 and 3. How much memory does each stage leave per GPU for a 70B model?→Distributed Training & ParallelismFSDP or DeepSpeed ZeRO-3: which would you pick for a new training codebase today, and why?→Distributed Training & ParallelismYou need to train a 100B dense model and it does not fit on one node. Walk me through how you would lay it out.→Napkin Math, Cost & CapacityHow much memory does it take to fine-tune a 70B model, full fine-tuning versus LoRA?→Distributed Training & ParallelismYou have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.→Distributed Training & ParallelismHow is training on TPUs with JAX different from training on GPUs with PyTorch? What do you stop doing by hand?→