vLLM crashes with out of memory during startup on a model that should fit. Debug it.

TL;DR: Identify the failing phase and GPU worker (rank) before changing flags: loading weights, profiling activations, sizing or allocating the cache, capturing CUDA graphs, or serving traffic. A paged KV pool is allocated from a memory budget; it does not reserve max-num-seqs × max-model-len positions at startup. Compare the worker's measured weights and non-cache memory with its budget, then choose a fix for the failed allocation. A context-capacity error can require a larger cache; a graph-capture OOM can require a smaller one. Moving the utilization fraction blindly can therefore make either problem worse.

How to approach it

Ask for the first allocation error, the preceding memory report, the failing rank, and the resolved engine configuration. Keep the model revision, engine image, GPU topology and workload fixed while testing one change. State which observation would disprove the diagnosis. vLLM Server Arguments That Matter explains the controls; Reading config.json to Size a Model You Have Never Run separates checkpoint evidence from runtime assumptions.

A strong answer

An FP8 checkpoint has roughly 70 GB of tensors, and the deployment has four nominal 80 GB devices. It still fails to start. Aggregate capacity looks generous, yet one worker fails before readiness. That establishes neither that the weights fit on every rank nor that the failure came from KV allocation. Temporary conversion buffers, replicated tensors and graph capture can each change the peak.

Know what each phase does. Profiling runs a dummy batch to measure peak activation memory. The cache pool is then sized from what remains. CUDA graph capture records common decode steps so they can be replayed with one launch, and that capture needs memory of its own. Start with the phase:

The vLLM 0.28.0 worker implementation profiles non-cache consumption and subtracts it from the requested budget; graph-memory estimation also depends on configuration. Record the accounting your version reports. Do not add its activation estimate a second time under a generic “overhead” percentage.

Work through one rank, then check every rank

Use this synthetic planning example, in decimal GB. Assume an evenly sharded 70 GB checkpoint, TP=4, no additional weight replicas, and an assumed 10 GB allowance for all non-weight, non-KV consumption included in the budget. These are teaching inputs, not a measured model deployment.

Per-device capacity                         80.0 GB
Executor budget at utilization 0.92         73.6 GB
Resident weights: 70 / 4                     17.5 GB
Other consumption included in budget        10.0 GB
Available KV pool: 73.6 - 17.5 - 10          46.1 GB
Capacity outside that budget                 6.4 GB

The last 6.4 GB is a cushion, not a guarantee that future allocations fit. Existing GPU users can leave less free memory than the requested budget. An explicit --kv-cache-memory-bytes setting overrides automatic fraction-based cache sizing in the versioned CLI reference; changing the fraction alone then does not resize that explicit pool.

A cache pool is allocated before it fills

Synthetic 80 GB worker: 17.5 GB weights, 10 GB other budgeted use, 46.1 GB cache and 6.4 GB outside the budget.

1. Startup allocates the pool

   The configured memory budget sets the physical pool. A scheduling limit of 256 does not allocate 256 maximum-length contexts.

2. Requests occupy blocks

   Growing requests use blocks within that same pool. Free blocks and allocated pool bytes are different quantities.

3. Find the allocation that failed

   A context-capacity check may need more cache; a later graph allocation may need more space outside it.

For cache arithmetic, assume 80 full-attention layers, eight KV heads, dimension 128 and BF16 KV, evenly sharded over four TP ranks:

Whole-model KV per position = 2 × 80 × 8 × 128 × 2 = 327,680 B
Per-rank KV per position    = 327,680 / 4 = 81,920 B
One 131,072-position request per rank = 10,737,418,240 B
46.1 GB / 10.7374 GB        = 4.29 full-length requests

Before block rounding and additional state, the pool holds four such requests. Setting --max-num-seqs 256 does not allocate 256 of them. It permits a scheduling upper bound; actual running work also faces token, memory and policy limits. If all 256 eventually needed their maximum contexts, they would require about 11.0 TB across the four ranks. That is a workload-demand calculation, not a startup reservation. Queueing and cache-pressure recovery determine what happens before that demand can be resident.

This division by four is an explicit GQA placement assumption. Replicated latent caches and hybrid recurrent state require different accounting. The smallest available pool across ranks can constrain the entire replica.

Choose the fix from the evidence

If the cache cannot hold one permitted full-context request, reducing --max-model-len can resolve a capacity check. It need not reduce the already budgeted physical pool. If profiling fails, reducing --max-num-batched-tokens can reduce activation demand. If graph capture fails, test eager execution or a reduced capture configuration; compare latency before keeping the change. If the weights alone fail, lowering the context limit does not remove those weights.

Cache pressure during serving may cause preemption and recomputation instead of a CUDA OOM, as described in vLLM's tuning guide. Watch recovery counts and tail latency alongside occupied blocks. Separately investigate runtime allocations, unsupported input paths and competing GPU processes. A prompt above the engine's configured context limit should be rejected; a missing gateway check does not normally disable engine validation.

My first change follows the failed phase. I reverse it if the reproduced error does not move, rather than continuing to shrink unrelated limits. A successful startup then earns a workload test: maximum accepted input and output, multimodal limits where relevant, concurrency, cancellation and a sustained run. No GPU deployment was executed for this example.

What interviewers probe next

• “Should utilization go up or down?” Up can enlarge an automatically sized cache when genuine headroom exists. Down can leave room for unaccounted runtime allocations. Neither repairs a weight-loading problem by itself.
• “Does a 2× weight mismatch prove FP8 became BF16?” No. Check conversion, duplicated tensors, uneven placement and units against the loader report; the ratio is a clue.
• “Does FP8 KV halve total memory?” It can roughly halve eligible KV payload compared with BF16. Weights, metadata and other state remain; backend support and quality need validation.
• “Why did lower concurrency help startup?” It may reduce profiled or captured shapes. That does not establish that all maximum-length caches were reserved.

Common mistakes

• Treating allocated pool bytes and occupied KV blocks as the same metric.
• Dividing every cache component by TP without checking placement.
• Counting activation or graph allowances twice.
• Calling startup success proof of stability or of the latency target.

Key takeaways

• The synthetic rank has a 46.1 GB pool after 17.5 GB of weights and 10 GB of other budgeted memory.
• With the stated GQA layout, one 131,072-position request needs 10.737 GB per rank; four fit in that pool before other costs.
• A 256-request limit does not reserve 256 maximum contexts at startup.
• Identify the failed allocation before deciding which direction to move a memory flag.