hardNewMetaNVIDIA

Estimate the activation memory for training a 70B model at an 8k sequence length

Activations are the term the 16-bytes-per-parameter rule leaves out, and at 8k context they are bigger than the weights: about 34 × sequence × hidden bytes per layer, 2.3 GB, 182 GB per micro-batch across 80 layers. The chain, what FlashAttention removes, and what checkpointing does to the number and the compute.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

🧮 Napkin Math & CapacitySign in

GPU-Hours and Time to TrainThe fleet equation turns a training run's FLOPs into a schedule: time = 6ND divided by (GPUs times peak FLOPS times MFU). Every term is a stated assumption, and the interviewer grades the assumptions rather than the digits: which peak, which MFU, and what happens to the answer when MFU falls from 40% to 30%. This page works three runs end to end (an 8B, a 70B and a 405B), inverts the equation for the GPU count a deadline needs, and shows the sensitivity that separates a considered estimate from a lucky one.

Advanced

🔌 Networking & Storage🔒 Premium

Checkpoint I/OA checkpoint of a 70B model's training state is 1.13 TB, and a 405B model's is 6.5 TB. Written synchronously every half hour, it stalls thousands of GPUs for as long as the slowest rank takes to reach the disk. The design that scales writes each rank's shard in parallel, copies it off the GPU to host memory first so the run resumes in seconds, and drains it to durable storage in the background. The arithmetic is bytes per rank against the bandwidth of each hop, and the goal is a checkpoint that costs the run under 1% of its time.

Advanced

🕸️ Distributed Training🔒 Premium

Context and Sequence ParallelismContext parallelism splits a single long sequence across GPUs so that attention over 128k tokens fits in memory that would otherwise need terabytes of activations per layer. Ring attention rotates key-value blocks around the ranks while queries stay put, and grouped-query attention is what makes that rotation cheap enough to hide behind the attention math.

Foundational

🧮 Napkin Math & Capacity

Model Memory FootprintThe first calculation in almost every AI infra loop: how many bytes does this model occupy, for inference and for training, and does it fit on the card in front of you? Inference is parameters times bytes per parameter (2 in bf16), plus a KV cache that grows with users. Training is 16 bytes per parameter before activations. A 70B model is 141 GB to serve and 1.13 TB to train, and a reader who can produce those two numbers from the parameter count, with the reasoning, has passed the first five minutes of the estimation round.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackModel our serving request queue with Little's law. What happens as we approach saturation?Next in this trackRank the H100, MI300X and Trainium2 by cost per token for decode Next in this trackA vendor claims 10,000 tokens per second per GPU. Sanity-check it.

FEDITOR'S NOTE

The interviewer wants the per-layer formula with its variables named, the 70B number, and the checkpointing trade. A candidate who gives only the 16 B per parameter static figure has answered a different question.

DISCUSSION · 0

No comments yet — be the first to share your approach.