vLLM Server Arguments That Matter
Separate executor memory, allocated cache pools and logical occupancy. Map context, sequence and token-batch limits to the constraints they control, then tune a pinned deployment against per-rank memory and latency.
TL;DR: Start with a supported model/backend and parallel layout, then budget each rank's memory.
--gpu-memory-utilizationbudgets the model executor, including weights;--kv-cache-memory-bytescan override automatic cache sizing.--max-model-lenlimits prompt plus output length, while sequence and token-batch limits constrain scheduling. None promises that every admitted request can occupy its maximum context simultaneously. Tune against cache pressure, activation peaks and latency under the actual workload.
Start from a reproducible deployment
A typical situation: a copied launch line starts successfully but long prompts stall every active stream. More GPUs or a larger cache might help, but first preserve the resolved arguments, engine image, model/tokenizer revisions and hardware topology. Defaults and supported combinations change. The vLLM 0.28.0 CLI reference is the version used for this page's flag definitions; check the installed version before applying them.
Choose TP, pipeline parallelism and expert placement together with the supported kernels. Head counts constrain some implementations, but padding, replication and specialized attention backends prevent a universal “TP must divide both head counts” rule. Weight bytes divided by TP is an estimate until placement is verified. Expert Parallel and All-to-All Backends covers a different partitioning axis.
Try multiple supported layouts once memory fits. A smaller TP degree can reduce communication; a larger one can reduce work per rank and improve latency. Pipeline boundaries and bubbles matter too. Choose the layout that meets the measured service target at acceptable cost.
A fraction, a pool and occupancy are different quantities
The executor is the engine's per-GPU worker that holds the weights, the KV cache and working buffers on that device.
The executor fraction is applied to GPU capacity, not simply to whatever memory happens to be free at that moment. Existing allocations can prevent that requested budget from being available. Automatic cache sizing accounts for non-cache consumption; graph estimation and exact profiling behavior depend on version and configuration. The resulting pool is physically allocated, while requests acquire logical blocks within it.
Synthetic example: one 80 GB rank, fraction 0.92, 17.5 GB resident weights and 10 GB of all other consumption charged to the budget:
Requested budget = 80 × 0.92 = 73.6 GB
Estimated KV pool = 73.6 - 17.5 - 10 = 46.1 GB
At fraction 0.90: pool = 72 - 17.5 - 10 = 44.5 GB
Difference = 1.6 GB, assuming the other allocations stay fixed
Suppose the per-rank cache cost is 81,920 B per stored position. That extra 1.6 GB buys about 19,531 positions before block rounding. It does not buy 19,531 extra requests. At 8,192 positions each, the two pools hold roughly 68 and 66 full-length requests respectively, before additional state and safety margin. These are memory estimates, not throughput results.
An explicit cache-byte setting replaces the automatic fraction-based sizing decision. Record which mode is active before interpreting an experiment that changes only the fraction. Also record whether the cache is sharded, replicated or hybrid; the same position count can have a different cost on each rank.
Map each flag to the constraint it controls
| Control | What it bounds or selects | What it does not guarantee |
|---|---|---|
--max-model-len | Accepted prompt-plus-output context | A separate maximum-length allocation for every request |
--max-num-seqs | Sequences processed in an iteration | Memory or latency for all sequences at full length |
--max-num-batched-tokens | Token work scheduled in an iteration | A fixed iteration duration across models and phases |
--gpu-memory-utilization | Automatic executor memory budget | Protection from unrelated processes growing later |
--kv-cache-memory-bytes | Explicit per-GPU cache allocation | Enough space for weights, activations and graphs |
--kv-cache-dtype | Supported cache storage representation | Identical quality or proportional total-memory savings |
Lowering a context cap can resolve a single-request cache-capacity check and restrict future demand. It does not generally shrink a pool that is independently sized from the memory budget. Lowering batch limits can reduce activation or graph requirements, with workload-dependent effects on latency and throughput.
The scheduler spends a token budget
This synthetic iteration has a 1,024-token budget. Assume 128 ordinary decode sequences each need one token of work, with chunked prefill enabled and no speculative verification. That leaves 896 token slots for prefill. These are scheduled input positions, not 1,024 emitted output tokens.
Token positions in the step
128 streams advance one token each
896 positions of a long prompt
Chunked Prefill explains why smaller prefill chunks can reduce interruptions to decode. Smaller chunks also add scheduling and kernel overhead and can lengthen a long request's prefill. Test a budget sweep against TTFT, token-gap tails and completed work within the SLO. A single aggregate token/s number hides who waited.
Keep model features explicit
Quantization, attention kernels, expert communication, prefix caching and tool/reasoning parsers must agree with the exact checkpoint. A flag existing in the CLI does not establish support for every model. For hybrid models, check recurrent-state allocation and prefix restoration as well as full-attention KV. Weight Formats: FP8 Blocks, MXFP4 and AWQ explains why checkpoint storage and runtime execution can differ.
Use the model's official engine recipe as a starting point, then preserve your tested configuration. Review custom code before enabling remote-code execution. Test cold and warm prefixes separately; cached prompt work does not eliminate decode cost. Parser validation belongs in the acceptance test because valid generated text can still become an invalid tool response at the API boundary.
Diagnose before increasing a limit
A capacity check that cannot fit one maximum-length request suggests lowering the supported context or increasing available cache. A graph-capture allocation failure suggests examining captured shapes and headroom; Conserving Memory describes eager execution as a way to remove graph overhead. Keep it only if the performance tradeoff is acceptable. Frequent cache-pressure recovery calls for a different experiment from a weight-loading failure.
In an interview, name the failing constraint, calculate its units and propose one reversible change. Reverse that change when the trace disproves the hypothesis.
Key takeaways
- At the stated assumptions, changing 0.90 to 0.92 adds 1.6 GB of pool, about 19,531 per-rank positions.
- A context limit, a sequence limit and a token-batch limit control different dimensions.
- The 1,024-position example leaves 896 for prefill after 128 decode positions; it predicts allocation of work, not wall time.
- Choose parallelism and batching from measured memory and service behavior, with pinned model and engine versions.