vLLM Server Arguments That Matter
A vLLM deployment is mostly decided by a dozen flags, and the ones that matter fall into four groups: how the model is split across GPUs, how memory is divided between weights and cache, how requests are batched, and which specialized backends the model needs. Getting the first two wrong produces an engine that will not start or that runs out of memory under load. Getting the third wrong produces an engine that starts, serves, and misses its latency target by a wide margin.
TL;DR: Four groups of flags, in the order they are decided. Parallelism:
--tensor-parallel-sizeand--pipeline-parallel-sizefix how the model is split, and the tensor degree must divide the model's attention head count. Memory:--gpu-memory-utilization, which defaults to 0.92 in current versions, sets the fraction of each GPU vLLM claims, and--max-model-lenbounds the KV a single sequence can consume, so the two together decide how many sequences fit. Batching:--max-num-seqscaps concurrency and--max-num-batched-tokenscaps the work in one iteration, which is what actually controls the tradeoff between throughput and time to first token. Model-specific backends:--quantization,--kv-cache-dtype,--enable-prefix-caching, and for mixture-of-experts models the expert-parallel and all-to-all flags. Set parallelism from the config, memory from arithmetic, batching from the latency target, and backends from the model card.
Group one: parallelism
--tensor-parallel-size / -tp default 1
--pipeline-parallel-size / -pp default 1
the constraint people hit
TP must divide num_attention_heads, and for grouped-query models it should also divide
num_key_value_heads, or heads are padded and memory is wasted
a model with 64 heads accepts TP of 1, 2, 4, 8, 16, 32, 64
a model with 1 KV head (multi-query or latent) replicates that head across TP ranks
choosing the degree
the smallest TP whose combined memory holds weights plus the KV budget, because every
extra rank adds two all-reduces per layer per token
GLM-5.3 in FP8, about 760 GB of weights: on B300 at 288 GB, TP=4 gives 1,152 GB which
holds the weights with 392 GB for KV; TP=8 gives 2,304 GB and much more headroom
sanity: the temptation is to use every GPU in the node, and the cost is communication on
every token, so the degree is chosen from the memory requirement rather than from
the node size
Pipeline parallelism is the multi-node answer when the model does not fit in one node's NVLink domain. It communicates once per micro-batch boundary rather than twice per layer, so it tolerates the slower fabric between nodes, at the cost of a pipeline bubble.
Group two: memory
--gpu-memory-utilization default 0.92
--max-model-len derived from the model config if unset
--kv-cache-dtype default auto
--block-size cache block size in tokens
what the utilization fraction actually does
vLLM measures free memory, takes that fraction, subtracts the weights and activations, and
gives the remainder to the paged KV cache
so raising it from 0.90 to 0.95 on a 288 GB GPU adds about 14 GB of cache, which at
GLM-5.3's 87.75 KB per token is roughly 160,000 more tokens of cache
the derivation to do before launching
KV pool per GPU = (memory x utilization) - weights per GPU - activation headroom
sequences that fit = KV pool / (max_model_len x KV bytes per token)
worked, GLM-5.3 on 8 x B300 at TP=8, utilization 0.92:
memory per GPU x utilization = 288 x 0.92 = 265 GB
weights per GPU = 760 / 8 = 95 GB
activation and workspace headroom, call it 20 GB
KV pool per GPU = 265 - 95 - 20 = 150 GB, so 1,200 GB across the eight
at 87.75 KB per token: 1,200e9 / 89,856 = 13.4M tokens of cache
at max_model_len 32,768: 13.4M / 32,768 = 407 sequences at full length
sanity: 407 concurrent full-length sequences is the memory ceiling, and --max-num-seqs should
be set below it rather than discovered by an out-of-memory crash under load
Setting --max-model-len lower than the model's maximum is the single most effective memory
lever. GLM-5.3 supports 1,048,576 tokens; a chat product that never exceeds 32,768 and leaves
the default in place has vLLM reserving for a context nothing will use.
Group three: batching, which sets the latency
the tradeoff these two flags control
--max-num-batched-tokens high: more work per iteration, higher throughput, and a long
prefill can delay every decode sharing that iteration
--max-num-batched-tokens low: prefill is chopped smaller, decode latency is smoother,
and total throughput drops
--max-num-seqs high: more concurrency, more KV pressure, longer queue behind
each decode step
sanity: these are the flags to tune against a time-to-first-token and time-per-output-token
target, and they are the ones most often left at defaults while people tune things
that do not move the number
Chunked Prefill covers the mechanism; the operational point is that chunked prefill plus a token budget is how a mixed workload of long prompts and short generations holds a tail latency target.
Group four: what the model requires
| Flag | When you need it |
|---|---|
--quantization / -q | Rarely set by hand; vLLM reads quantization_config from the model first |
--kv-cache-dtype | fp8 halves cache bytes per token and costs a little quality; worth measuring |
--enable-prefix-caching | On by default for most models, and explicitly required for some new architectures where it starts disabled |
--load-format | fastsafetensors and similar shorten the load of a multi-hundred-gigabyte model |
--trust-remote-code | Needed for architectures whose modelling code ships with the weights |
--enable-expert-parallel | Mixture-of-experts models; covered in the expert-parallel concept |
--enable-chunked-prefill | Mixed long-prompt and decode traffic |
The pattern to internalize is that model-specific flags come from the model's own launch
notes rather than from a general guide. The vLLM project's day-zero note for Kimi K3, for
instance, gives a specific launch line including --tensor-parallel-size 8,
--trust-remote-code, --load-format fastsafetensors, --enable-prefix-caching, and
parser flags for its tool and reasoning formats, and states that prefix caching had to be
passed explicitly because it started disabled for that architecture.
What interviewers are listening for
Whether the numbers come from arithmetic. Anyone can list flags. The candidate who says "at
0.92 utilization on 288 GB with 95 GB of weights per rank we have about 150 GB of cache per
GPU, which is 13.4 million tokens, so 407 sequences at 32k, and I set max-num-seqs to 256 for
headroom" has done the job. The second signal is --max-model-len: reducing it to the actual
product requirement is the cheapest capacity win available and is skipped constantly. The
third is knowing that batching flags rather than parallelism flags are what move tail latency.
Key takeaways
- Tensor parallel degree must divide the attention head count, and the smallest degree that fits the memory is usually the fastest.
--gpu-memory-utilizationdefaults to 0.92 and controls the KV pool; the pool is memory times utilization minus weights minus activation headroom.- Worked example: GLM-5.3 at TP=8 on B300 gives about 150 GB of KV per GPU, 13.4M tokens, or 407 sequences at 32k context.
- Set
--max-model-lento the product's real maximum rather than the model's, since it is the cheapest capacity lever there is. - Tail latency is controlled by
--max-num-batched-tokensand--max-num-seqswith chunked prefill, not by the parallelism degree.