Model Memory Footprint

TL;DR: Model memory is a collection of allocations, not just the checkpoint size. Start with stored weights, add the states required by the workload, and compare the peak allocation on every GPU with its usable capacity. A 70.6B-parameter BF16 payload is 141.2 GB. The familiar 16 bytes per parameter for Adam training describes one state layout; precision, optimizer and sharding choices can change it.

Why a model loads, then runs out of memory

The process starts successfully. One short prompt works. A longer batch fails during generation. Loading proved only that the initial allocations fit; the live request created more state.

For inference, weights hold learned parameters. The KV Cache stores attention state for retained positions. Intermediate activations hold the current computation, while the runtime also needs graph pools, communication buffers and allocator space. Some allocations persist; others peak only during a particular step. Adding every allocation's separate maximum can overestimate a peak if their lifetimes do not overlap, while inspecting only idle memory can miss the real peak entirely.

GPU Memory Hierarchy distinguishes device memory from caches and registers. Here the budget concerns device memory available to the process. Use exact bytes from the device and allocator when checking a tight fit. GB means 10⁹ bytes; GiB means 2³⁰ bytes. A product label is not a substitute for available-memory measurements after initialization.

Start with the stored tensors

A parameter is one learned number in the model. "70.6B" means 70.6 billion of them. Each is stored in a number format, and the format sets its size: BF16 and FP16 use 2 bytes, FP8 and INT8 use 1 byte, and packed 4-bit formats use half a byte (Numerics). So the first estimate is a multiplication: 70.6 billion × 2 bytes = 141.2 billion bytes, or 141.2 GB.

For a uniform format, raw weight payload is parameter count times bytes per parameter. Packed four-bit values occupy half a byte each in the ideal payload. Actual quantized formats also need scales or other metadata and may leave selected tensors at higher precision. The Transformers quantization documentation explicitly allows modules to retain another dtype.

These are payload calculations, not promised model allocations. Tied tensors should be counted once when storage is shared, and replicated tensors must be counted on every rank that holds them.

BF16 payload = 70.6e9 × 2 = 141.2e9 bytes
in GiB = 141.2e9 / 2³⁰ = 131.50 GiB
sanity: changing units changes the numeral, not the memory required.

One 80 GB-class H100 cannot hold that BF16 payload. More GPUs help only if the deployment actually shards the relevant storage. Eight replicas of a full model do not create eight times as much space for one copy.

Add workload state to each rank

KV Cache Sizing derives the Llama-style BF16 cache payload of 320 KiB per retained position. One 8,192-position sequence therefore holds 2.5 GiB across the model. Thirty-two such sequences hold 80 GiB, before block rounding or sharing. That is a different quantity from 80 GB.

Large models are often split across GPUs with tensor parallelism: each layer's weight matrices are cut into slices, one slice per GPU. TP8 means eight GPUs, and each participating GPU process is called a rank. Attention keeps its cache in groups called KV heads; this model has eight. In an illustrative TP8 layout with the eight KV heads sharded one per GPU, each sequence needs 2.5 GiB ÷ 8 = 320 MiB on each rank at that length. If the limiting rank has 38 GiB available for cache after other reservations, it holds 38 × 1,024 MiB ÷ 320 MiB = 121.6, so 121 whole payloads. Reserve growth as well: a current 8k prompt may need a larger reservation if it will generate further tokens. Actual allocation policies may instead permit preemption or recomputation; that choice has latency consequences.

A fixed “reserve 10%” rule cannot account for every shape, graph configuration or runtime. Measure representative prefill and decode peaks and leave deliberate room for the workload you plan to admit.

Training keeps more than weights

Training needs gradients and optimizer state. One mixed-precision Adam layout stores a low-precision parameter copy and gradients, an FP32 master copy, and two FP32 moment tensors. Its bookkeeping is:

For 70.6B parameters, that state payload is 1.1296 TB. Under an illustrative 64 GB state budget per GPU and perfectly even full sharding, dividing gives 17.65, so at least 18 state partitions are needed. This is a capacity lower bound for that budget, not a runnable 18-GPU training configuration. Layer gathers, activation peaks and supported parallel layouts still have to fit.

ZeRO partitions model states across workers. It changes placement and communication; it does not eliminate the aggregate logical state. Also inspect the actual training implementation: automatic mixed precision can choose operation dtypes while parameters remain FP32. PyTorch's AMP documentation is a reminder that “mixed precision” alone does not specify the five tensors in the diagram.

Adaptation still needs activation memory

LoRA freezes the base parameters and trains adapter matrices. It removes base-parameter gradients and optimizer states, but gradients still flow through the computation needed to train the adapters. Activations do not vanish. Adapter count depends on rank and target matrices, so “adapters are 1%” is an assumption to calculate, not a rule.

QLoRA reduces frozen-base storage through quantization. Its feasibility still depends on metadata, adapters, sequence length, microbatch and activation strategy. A claim that a quantized fine-tune fits on one GPU needs those inputs, not just the four-bit weight payload.

Check the live peak

PyTorch's memory notes distinguish memory occupied by tensors from memory managed by the caching allocator. Device-wide usage can additionally include allocations outside that allocator. Compare like scopes and avoid adding reserved memory to allocated memory as if they were disjoint.

For an interview follow-up, halve the weight bytes while holding the 32-sequence BF16 cache fixed. The weight payload falls from 141.2 GB to 70.6 GB; the cache remains 80 GiB. The whole process therefore cannot be assumed to halve. This is the same check to make before accepting a quantization memory claim.

What to remember

• Raw weights are one term: 141.2 GB for the stated BF16 parameter count.
• Thirty-two 8k cache payloads total 80 GiB in the stated BF16 architecture.
• Declare each training tensor's format before using 16 bytes per parameter.
• Fit and peak memory must hold on every participating GPU.
• Quantization, sharding and adapters change different terms in the budget.