TL;DR: Under an explicit 16-byte training-state recipe, 70.6B trainable parameters require 1,129.6 GB before activations. LoRA keeps the frozen base but stores gradients and optimizer state only for adapters. The example below has 207.1M adapter parameters and about 144.5 GB of static payload. Neither figure proves that a particular GPU layout fits.
How to approach it
First name the precision and optimizer buffers actually stored. Separate static payload from peak allocation, then count LoRA targets from their shapes. Close with the peak on the worker that uses the most memory, and the quality test that decides whether adapters are sufficient.
A strong answer
Two cards can hold inference weights, yet the first optimizer update runs out of memory. Training needs more than a readable model. The static payload below means the weights and training state kept between steps; activations and temporary buffers add to the peak. Model Memory Footprint explains the components; the calculation here makes the chosen recipe explicit.
illustrative trainable-state recipe, bytes per parameter:
BF16 weight 2 + BF16 gradient 2 + FP32 master 4 + two FP32 moments 8 = 16
full static payload = 70.6e9 × 16 = 1,129.6 GB
with a declared 64 GB static-state allowance per card:
ceil(1,129.6/64) = 18 cards, rounded to 24 for whole 8-card nodes
sanity: 24 × 64 = 1,536 GB; this allowance is not all device memory.
Some implementations retain FP32 sharded parameters and create low-precision compute buffers temporarily. Others use different gradient or optimizer dtypes. Inventory the actual tensors rather than insisting that every Adam implementation stores 16 bytes continuously. Offload also changes where the bytes live.
LoRA, low-rank adaptation, trains a low-rank change to selected matrices: two thin matrices whose product is added to a frozen weight, so only the thin ones need gradients and optimizer state. Pretraining, Fine-Tuning and Distillation traces the two matrix paths and their sum. Here the question is how that changes the budget. For a matrix with input width a and output width b, rank r adds r(a+b) parameters. Use an illustrative 80-layer, Llama-70B-shaped target list: hidden width 8,192, MLP width 28,672 and K/V projection width 1,024. Target q, k, v, o and all three MLP matrices, with rank 16:
q and o: 2 × 16 × (8,192 + 8,192) = 524,288 per layer
k and v: 2 × 16 × (8,192 + 1,024) = 294,912
MLP: 3 × 16 × (8,192 + 28,672) = 1,769,472
all 80 layers: 80 × 2,588,672 = 207,093,760 adapter parameters
adapter state at 16 B each = 3.3135 GB
frozen base at 2 B each = 141.2 GB
static payload ≈ 144.5135 GB
Changing targets or rank changes that count. It is about 207 million here, not merely “tens of millions.” Three 64-GB allowances cover this payload arithmetically, but supported sharding degrees, replicated tensors and peak buffers determine the actual placement.
| Component | What freezing changes |
|---|---|
| Base weights | Remain resident or must be fetched |
| Base weight gradients and optimizer | Can be omitted for frozen parameters |
| Adapter state | Added for the trainable matrices |
| Saved activations | Follow the backward graph and implementation |
Activations are not necessarily identical to full fine-tuning. Computing a base weight gradient may require saved values that freezing removes, while propagating gradients to earlier adapters still requires work through frozen layers. Activation Checkpointing can shorten saved lifetimes but adds recomputation; its saving is not a fixed factor.
Start with adapters when they meet the task's held-out quality and operational budget. Compare targets, rank and training recipe before expanding to full fine-tuning; switch when a controlled full-tuning experiment earns the extra state and compute. “New knowledge requires full tuning” is too broad. Quantized-base training changes storage and kernels again, so include scales, nonquantized tensors and dequantization workspace before promising a one-card run.
Measure the peak over forward, backward, optimizer update and save. A p99 step-time regression can also make a memory-saving policy too expensive at the required throughput.
What interviewers probe next
- “Does the frozen base receive no backward work?” It receives no weight update, but backward may need it to propagate gradients to trainable components.
- “Why keep higher-precision state?” Small updates can disappear when added directly to a coarse weight representation; validate the chosen numerical recipe.
- “Can I divide all bytes by the GPU count?” Only tensors actually sharded that way. Count the largest rank and transient gathers separately.
Common mistakes
- Calling 16 bytes the optimizer moments alone; the two FP32 moments are 8 bytes.
- Promising that LoRA leaves activation memory exactly unchanged.
- Turning a static-state quotient into a deployable GPU count.
Key takeaways
- This full-tuning recipe uses 1,129.6 GB of static payload.
- The declared rank-16 target list adds 207.1M parameters, giving about 144.5 GB with the frozen base.
- Peak memory and measured quality decide the actual layout and method.