TL;DR: Using an approximate 70.6B parameter count, raw weights require 141.2 GB in BF16, 70.6 GB at one byte per value, or 35.3 GB at four bits per value. Those numbers exclude quantization metadata, higher-precision tensors, KV cache and runtime peaks. One 80 GB-class cannot hold the BF16 payload; a useful multi-GPU or quantized deployment needs a per-rank allocation check for the intended workload.
How to approach it
First clarify whether the request is for raw weights or a running service. Name the weight format and compute its payload. Then add cache for the retained prompt and output positions, plus measured non-cache allocations. Compare every rank (each GPU process that holds part of the model) with the exact bytes available and check latency separately.
A strong answer
A model loads on two GPUs, but a batch of long prompts causes an out-of-memory error. The weight estimate was correct. The mistake was treating successful loading as evidence that the serving workload would fit.
Model Memory Footprint starts with stored tensor values. For a rough 70.6B-parameter estimate:
BF16: 70.6e9 × 2 bytes = 141.2e9 bytes = 141.2 GB
one-byte values: 70.6e9 × 1 = 70.6 GB
packed four-bit values: 70.6e9 × 0.5 = 35.3 GB
BF16 in GiB: 141.2e9 / 2³⁰ = 131.50 GiB
sanity: 16 bits / 8 bits per byte = 2 bytes per BF16 value.
Use the checkpoint's actual tensor shapes and storage for an exact count. “70B” is a model-family label, and quantization does not necessarily convert every tensor uniformly. Transformers' bitsandbytes guide describes skipped modules and retained dtypes. Scale tensors, padding and protected weights must be counted alongside packed values.
An H100 SXM is an 80 GB-class device, so the BF16 weight payload alone cannot fit on one. Do not infer exact usable bytes from that label when evaluating a close fit. Read the device's reported capacity and the engine's allocations after initialization, keeping GB and GiB conversions explicit.
The KV Cache is the next large term. KV Cache Sizing derives 327,680 bytes per retained position for the Llama-style BF16 cache: 80 layers, eight KV heads and dimension 128. At 8,192 retained positions:
one sequence = 327,680 × 8,192 = 2,684,354,560 bytes
= 2.5 GiB
32 sequences = 32 × 2.5 = 80 GiB
Those are raw payloads with no prefix sharing. Physical blocks, replication and the growth policy determine actual reservation. At the same context, 32 sequences carry a substantial cache even if the weights are compressed. Weight precision and cache precision are separate settings.
For a deliberately hypothetical TP2 deployment, suppose each GPU has 4 GiB left for KV after loaded weights, runtime buffers and headroom. Assume the eight KV heads shard evenly across the two GPUs with no replication. One 8k sequence then requires 1.25 GiB on each rank:
per-rank capacity = floor(4 / 1.25) = 3 sequences
3 × 1.25 = 3.75 GiB fits; 4 × 1.25 = 5 GiB does not
That gives three current-length payloads, not three guaranteed complete generations. Reserving another 2,048 positions raises the per-rank payload to 1.5625 GiB and lowers the count to two. This example is an allocation exercise, not a claim that every TP2 engine leaves 4 GiB free.
The deployment has to pass both loading and growth checks. Follow the same hypothetical rank budget through the two lengths; successful initialization does not reserve future attention state.
| Required reservation | Per-rank result |
|---|---|
| 8,192 positions | 1.25 GiB each; 3 fit |
| 10,240 positions | 1.5625 GiB each; 2 fit |
| Unequal rank budgets | The smallest capacity wins |
| Different cache format | Recompute payload and metadata |
The table shows why the maximum output length belongs in the memory discussion. It also shows why dividing aggregate free memory by one model-wide cache can hide the limiting GPU.
Load the exact checkpoint in the proposed layout and record memory after initialization. Then exercise the largest prefill and generation shapes the service will admit, because those can create higher peaks. PyTorch's CUDA memory documentation distinguishes memory occupied by tensors from memory reserved by its allocator. Do not add those as if they were separate allocations, and check device-wide usage for memory outside that allocator. Until these peaks are known, a fixed overhead percentage is only a planning placeholder.
If BF16 cannot meet the memory budget, evaluate a supported quantized format or more sharding. If the candidate format harms task quality, keep the higher precision and change placement or admission instead. More ranks may reduce per-rank weight bytes while adding communication; the cheapest layout needs a measured service comparison.
What interviewers probe next
- “Do two H100s suffice?” Their nominal aggregate capacity exceeds the raw BF16 weights, but runtime placement, cache and output growth determine whether the intended workload fits. Loading alone is not a service test.
- “Would FP8 fit on one?” Its uniform raw payload is 70.6 GB. Check actual formats, scales, available bytes and cache. Some short workloads may fit; “no useful cache” is too broad a claim.
- “What about 405B parameters?” At two bytes each, raw weights are 810 GB. The declared precision and placement must accommodate that before adding any workload state.
- “Does a mixture of experts use only its active parameter count?” A fully resident model stores all experts. Active parameters describe selected computation; offloading changes residency and introduces transfer costs.
Common mistakes
- Giving a parameter count the unit GB without a storage format.
- Assuming packed value size includes all quantization overhead.
- Mixing allocated, reserved and device-wide memory as separate additive terms.
- Admitting requests using prompt-only cache when generation can grow it.
Key takeaways
- The stated BF16 raw payload is 141.2 GB, or 131.50 GiB.
- One 8k BF16 cache is 2.5 GiB across the model under the stated architecture.
- In the hypothetical TP2 budget, reserving growth lowers the payload count from three to two.
- Check per-rank peak memory, quality and latency before choosing the deployment.