Core

ZeRO and FSDP

ZeRO and FSDP reduce replicated training state by distributing it across workers. The memory saving comes with communication and temporary-buffer costs.

This concept is included with a free account. Sign in to continue reading.

free account · no card needed

RELATED CONCEPTS

Data Parallelism and DDP Tensor Parallelism Collective Communication Primitives Activation Checkpointing Model Memory Footprint Communication Volume Estimates

LESSONS THAT TEACH THIS

Training at ScaleA per-rank memory budget narrows the feasible training layouts17 min→Training at ScalePrice parallelism by message size, frequency and exposed time26 min→Training at ScaleSharding trades persistent memory for a materialization schedule22 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismExplain ZeRO stages 1, 2 and 3. How much memory does each stage leave per GPU for a 70B model?→Distributed Training & ParallelismFSDP or DeepSpeed ZeRO-3: which would you pick for a new training codebase today, and why?→Distributed Training & ParallelismYou need to train a 100B dense model and it does not fit on one node. Walk me through how you would lay it out.→Napkin Math, Cost & CapacityHow much memory does it take to fine-tune a 70B model, full fine-tuning versus LoRA?→Distributed Training & ParallelismA PyTorch team wants to train on TPUs. Do they rewrite in JAX with MaxText, use PyTorch/XLA, or wait for TorchTPU?→Distributed Training & ParallelismYou have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.→