When does activation checkpointing pay for itself, and what would you checkpoint first?

Follow saved and recomputed values through backward. Compare complete update time and peak memory instead of promising a fixed memory saving or slowdown.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Follow saved and recomputed values through backward. Compare complete update time and peak memory instead of promising a fixed memory saving or slowdown.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

GPUs and time-to-train calculator

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🕸️ Distributed Training🔒 Premium

Activation CheckpointingActivation checkpointing saves memory by recomputing selected forward results during backward. Choose what to retain by comparing storage cost with recomputation cost.

Core

🕸️ Distributed TrainingSign in

MFU and HFUMeasure useful model work, separate recomputation, and align token counts, timing and hardware peaks. Learn what activity counters can and cannot tell you.

Core

🧮 Napkin Math & CapacitySign in

GPU-Hours and Time to TrainConvert a stated work budget into elapsed time and GPU-hours. Vary utilization, invert for a deadline, and separate productive time from calendar delays without counting them twice.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache saves attention keys and values from processed tokens. Follow one decode step, calculate its memory cost, and see why longer conversations reduce the number of requests a GPU can hold.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow do sequence parallelism and context parallelism make 128k-context training of a 405B possible, and what do they cost?Next in this trackExplain expert parallelism for an MoE model: what the all-to-all is, how much it moves, and why load balance is the whole problem.Next in this trackYour MoE router sends 90% of tokens to 10% of the experts. What happens to the step, and how do you fix it without hurting the model?

FEDITOR'S NOTE

Identify the tensors responsible for the memory peak, include replay workspace and compare complete updates. Verify gradients and random or mutable state when recomputing.

DISCUSSION · 0

No comments yet — be the first to share your approach.