TL;DR: Full fine-tuning with mixed-precision Adam holds 16 bytes per parameter (bf16 weights 2, bf16 gradients 2, fp32 master 4, two fp32 Adam moments 8): 70.6e9 × 16 ≈ 1.13 TB of static state, at least 18 H100s sharded before activations. LoRA freezes the 141 GB of bf16 weights and trains adapters of tens of millions of parameters, so the trainable state drops to a few gigabytes and the job fits on one node, with activations now the largest term.
How to approach it
Ask which optimizer and precision (mixed-precision AdamW is the default), and whether the question is static state or the peak including activations. Start with the per-parameter byte count and decompose it out loud, because that decomposition is what the interviewer is scoring. Multiply, compare to a node, and then redo the sum for LoRA by separating frozen from trainable parameters. Close by saying what LoRA does not remove: the weights and the activations.
A strong answer
A typical situation: a team plans a fine-tune on the two cards that already hold the model for inference, and discovers that training state is eight times the weights. The optimizer, not the model, is what does not fit.
Training state per parameter for mixed-precision Adam:
bf16 weights 2 B (the copy the forward and backward use)
bf16 gradients 2 B
fp32 master weights 4 B (the precise copy the optimizer updates)
fp32 Adam first moment 4 B
fp32 Adam second moment 4 B
total 16 B per trainable parameter
full fine-tune of Llama 3.1 70B:
static state = 70.6e9 × 16 = 1.13e12 B ≈ 1.13 TB
GPUs to hold it when fully sharded (FSDP / ZeRO-3), 80% of 80 GB usable:
1.13e12 ÷ (80e9 × 0.8) = 1.13e12 ÷ 64e9 ≈ 17.7 → 18 H100s minimum, so 3 nodes (24 cards)
in practice 4 nodes (32 cards), because activations come on top
sanity: 1.13 TB is eight times the 141 GB serving footprint, and more than a full 8-card node
holds (640 GB), so full fine-tuning a 70B is never a single-node job in bf16 Adam.
Now LoRA. The base weights are frozen, so they need no gradients, no master copy and no optimizer moments; they sit in memory at 2 bytes each and are read in the forward and backward passes. The trainable parameters are the low-rank adapters, A (d × r) and B (r × d) on each targeted matrix. For rank 16 on the attention and MLP projections of an 80-layer model the adapter count comes to tens of millions:
LoRA, rank r = 16, applied to q, k, v, o and the three MLP matrices in every layer
per matrix: r × (d_in + d_out); for an 8192 × 8192 projection: 16 × 16,384 = 262k
MLP matrices are 8192 × 28,672: 16 × 36,864 = 590k each
k and v project to 1,024 (8 KV heads × 128): 16 × (8,192 + 1,024) = 147k each
per layer ≈ 2 × 262k + 2 × 147k + 3 × 590k ≈ 2.6M
80 layers ≈ 207M → call it 100M to 250M depending on targets and rank
trainable state = adapters × 16 B = 200e6 × 16 = 3.2 GB
frozen weights = 70.6e9 × 2 B = 141 GB
static total ≈ 145 GB
sanity: 145 GB is one eighth of the full fine-tune's 1.13 TB, and fits on a single 8-card node
with 400 GB left for activations, or on 2 × H200 with fp8 base weights.
What LoRA does not change is the activation memory, which for both methods is proportional to batch × sequence × hidden × layers and, at 8k sequences on a 70B, is tens of gigabytes per sequence without checkpointing. With gradient checkpointing (recompute activations in the backward pass) that drops by roughly an order of magnitude, and it is standard for both full and LoRA fine-tunes. So a LoRA job on one node is usually activation-bound: the batch size is what the leftover 400 GB will hold, not the adapters.
| method | static state | minimum cards (80 GB, 80% usable) | what limits batch |
|---|---|---|---|
| full, bf16 Adam | 1.13 TB | 18 (plan 32) | activations after sharding |
| full, 8-bit Adam (moments at 1 B each) | 70.6e9 × 10 = 706 GB | 12 (plan 16) | same |
| LoRA r=16, bf16 base | 145 GB | 3 (plan 8) | activations |
| QLoRA, int4 base | 35 + 3 GB | 1 | activations, dequant overhead |
The reversal condition, and the decision: full fine-tuning when the task needs the whole model to move (a new language, a large domain shift, continued pretraining) and the budget has 32 H100s for the duration; LoRA when the data is thousands to hundreds of thousands of examples and the change is behavioral, which is the common case and runs on one node. The reversal is quality: if a LoRA run plateaus below the target and rank increases do not help, the remaining lever is full fine-tuning, and the memory jumps by 8x. Model Memory Footprint is the static half of this, and nvidia-smi --query-gpu=memory.used after the first step is the check that the estimate held. Activation Checkpointing is the lever that moves the activation half of the bill.
What interviewers probe next
- "Why fp32 master weights?" bf16 has 8 bits of mantissa; a learning rate times a gradient is often smaller than the weight's bf16 resolution and would round to no update. The fp32 copy accumulates the small steps.
- "Can you drop the master copy?" Pure bf16 Adam with stochastic rounding or Kahan summation exists and saves 4 B per parameter; it is a numerics risk to be validated, not a default.
- "Where do activations go in the LoRA case?" Same place as full fine-tuning: through every frozen layer, because the backward pass still needs each layer's input to compute the gradient for the adapter and to propagate to the layer below.
- "What about serving many LoRA adapters?" The frozen base is shared and each adapter is a few hundred megabytes, so one replica can hold hundreds; that is multi-LoRA serving.
Common mistakes
- Forgetting the master weights or one Adam moment and reporting 12 bytes per parameter.
- Claiming LoRA "reduces memory 8x" without saying that the 141 GB of weights stay; the 8x is on the trainable state, and the total drops from 1.13 TB to about 145 GB plus activations.
- Sizing the job on static state alone and running out of memory on the first 8k-sequence batch.
- Dividing 1.13 TB by 80 GB and reporting 15 cards, with no usable-memory headroom.
Key takeaways
- Full fine-tuning: 16 B per parameter (2 + 2 + 4 + 4 + 4), 1.13 TB for a 70B, 18 H100s minimum sharded, plan 32.
- LoRA: base weights stay at 141 GB in bf16; the adapters' 16 B per parameter is a few gigabytes; one node.
- Activations are unchanged by LoRA and set the batch size; gradient checkpointing is standard for both.
- 8-bit Adam trims full fine-tuning to 10 B per parameter; QLoRA puts a 70B on one card.