← Napkin Math, Cost & Capacity / 27

hardMetaNVIDIA

Estimate the activation memory for training a 70B model at an 8k sequence length

Estimate activation memory from saved tensor shapes and lifetimes. Use a published reference formula carefully, then check checkpointing and parallel ownership against the real peak.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Estimate activation memory from saved tensor shapes and lifetimes. Use a published reference formula carefully, then check checkpointing and parallel ownership against the real peak.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🧮 Napkin Math & CapacitySign in

GPU-Hours and Time to TrainConvert a stated work budget into elapsed time and GPU-hours. Vary utilization, invert for a deadline, and separate productive time from calendar delays without counting them twice.

🔌 Networking & Storage🔒 Premium

Checkpoint I/OTrace checkpoint bytes through staging and storage, calculate when background saves fall behind, and distinguish upload completion from recoverable-state age. Includes an executable queue example and restore budget.

🕸️ Distributed Training🔒 Premium

Context and Sequence ParallelismContext parallelism distributes a long sequence across devices. Attention then exchanges the information each rank needs instead of keeping the whole sequence locally.

🧮 Napkin Math & Capacity

Model Memory FootprintSeparate weight payload, growing request state and training allocations. Work through exact units and per-rank budgets, then check the live peak.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackModel our serving request queue with Little's law. What happens as we approach saturation?Next in this trackRank the H100, MI300X and Trainium2 by cost per token for decode Next in this trackA vendor claims 10,000 tokens per second per GPU. Sanity-check it.

FEDITOR'S NOTE

Distinguish a reference GPT inventory from a deployed Llama allocation and count recomputation workspace with retained boundaries.

DISCUSSION · 0

No comments yet — be the first to share your approach.