Advanced

Activation Checkpointing

Activation checkpointing saves memory by recomputing selected forward results during backward. Choose what to retain by comparing storage cost with recomputation cost.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

ZeRO and FSDP MFU and HFU Context and Sequence Parallelism Tensor Parallelism Model Memory Footprint GPU Memory Hierarchy

LESSONS THAT TEACH THIS

Training at ScaleA per-rank memory budget narrows the feasible training layouts17 min→Training at ScaleSharding trades persistent memory for a materialization schedule22 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismWhen does activation checkpointing pay for itself, and what would you checkpoint first?→Distributed Training & ParallelismYou need to train a 100B dense model and it does not fit on one node. Walk me through how you would lay it out.→Distributed Training & ParallelismExplain ZeRO stages 1, 2 and 3. How much memory does each stage leave per GPU for a 70B model?→LLM Inference & ServingWhat is the KV cache, and why does it keep growing while a request is being served?→Napkin Math, Cost & CapacityHow much GPU memory does it take to run Llama 3.1 70B?→Napkin Math, Cost & CapacityHow big is the KV cache for Llama 3.1 70B at a 128k context?→