Set max-model-len and max-num-seqs for a chat product from first principles.

TL;DR: Define the product's prompt-plus-output contract, then derive a per-rank state budget and measure latency under offered load. --max-model-len bounds an individual context; it does not reserve a maximum context for every sequence. --max-num-seqs limits sequences processed in an iteration, while actual concurrency also depends on cache and token budgets. A worst-case memory bound is useful for guaranteed full-length service; a mixed workload can use a different admission policy. Keep only configurations that meet both the latency and overload contracts.

How to approach it

Ask what the product promises: accepted prompt length, requested output allowance, streaming latency, and treatment of oversized or overloaded requests. Use observed distributions to inform that promise, not to silently remove it. Calculate state at the limiting rank, then run a load sweep with the promised boundary cases. Finish with a trigger for revisiting the settings.

A strong answer

The model supports a million-token context. The chat product usually sends a few thousand tokens and needs room for an answer. The default context is unnecessarily permissive for this product. That can affect capacity checks and worst-case service demand even though paged attention does not allocate every request's full allowance up front.

Make the context cap a product contract

This synthetic distribution counts prompt plus actual output, using the deployed tokenizer:

Suppose the product chooses a 12,288-position maximum, including generated output. A request with 10,500 prompt tokens and a 2,000-token output allowance exceeds it by 212. Reject with a useful error or explicitly negotiate a smaller output allowance; do not silently discard prompt content. Tool schemas, chat-template tokens and supported multimodal expansion count toward the effective model input. Historical actual outputs do not establish that future requested allowances will fit.

The model's 1,048,576-position limit is 85.33 times this cap. That is a ratio of permitted context sizes, not memory preallocated per request. The vLLM model configuration reference defines the limit as prompt plus output. Verify the gateway and engine enforce a consistent contract.

Calculate a conservative state ceiling

Use synthetic TP=4 on nominal 80 GB devices. Assume 70 GB of evenly sharded resident weights, a 0.92 executor fraction and 8 GB per rank for all other memory charged inside that budget. Assume 80 uniform full-attention layers, eight KV heads, head dimension 128 and BF16 cache, with those KV heads sharded evenly across TP.

Pool per rank = 80 × 0.92 - 70/4 - 8 = 48.1 GB
Whole-model KV per position = 2 × 80 × 8 × 128 × 2 = 327,680 B
Per-rank KV per position = 327,680 / 4 = 81,920 B
One 12,288-position context per rank = 1,006,632,960 B
floor(48.1e9 / 1,006,632,960) = 47 full-length sequences
sanity: 48 would need 48.318 GB per rank, exceeding the 48.1 GB pool.

For a planning policy that keeps 10% of the estimated pool outside the full-length reservation budget, the bound becomes floor(43.29e9 / 1,006,632,960) = 43. That reserve is an explicit assumption, not a universal engine requirement. Reconcile block rounding and additional state with actual startup reports. vLLM Server Arguments That Matter explains the distinction between the physical pool and logical occupancy.

The 47-sequence result is a conservative full-length packing bound for this layout, not a universal maximum running batch. Shorter requests, shared blocks and progressive allocation can permit more active sequences. If the policy oversubscribes potential growth, define how it queues, preempts or otherwise recovers when requests grow together. Replicated MLA and hybrid state need a different per-rank calculation.

Measure a service operating point

Use an arrival-rate sweep with the real prompt/output mix, bursts and cache behavior. Record offered, admitted, completed and rejected requests; TTFT, per-request TPOT, individual token-gap tails and end-to-end completion. Latency Metrics: TTFT, TPOT and Goodput explains those measurements. A fixed-concurrency sweep is useful for diagnosis, but waiting for one response before sending another can hide queue growth under independent arrivals.

The following is an invented teaching dataset, not a benchmark. Assume the product requires p99 TPOT ≤40 ms. All other criteria still need separate tests.

Cap 40 is the largest tested passing candidate, not proof that 41 fails or that 40 has adequate margin. Repeat near the boundary, include long outputs and cold prefixes, and require the TTFT and rejection targets too. An average rate of 25 tokens/s corresponds to 40 ms per token, but average rate cannot establish a p99 TPOT result or the longest streaming gap.

Choose and revisit

I would test 12,288 and cap 40 under the stated synthetic assumptions, keeping the reserve policy and admission behavior explicit. I would lower the cap or change the token-batch budget if mixed prefill breaks streaming targets. I would revisit the context contract when legitimate longer workflows appear, rather than treating yesterday's percentiles as permanent limits.

For offline work, optimize completed useful work per unit cost and any batch deadline. Larger concurrency may help, but repeated recovery and wasted work can make it worse. Dropping interactive latency requirements does not remove memory limits. vLLM's optimization guide describes the cache-pressure tradeoff; measure its effect instead of pushing a limit indefinitely.

What interviewers probe next

• "Why not set the cap to p99?" That rejects or changes valid tail workflows unless the product explicitly accepts it. Percentiles inform the contract; they do not define it automatically.
• "Can more than 47 short requests run?" Potentially, under this state model. The 47 bound assumes every request occupies 12,288 positions; verify growth and recovery policy.
• "Low cache usage, but the cap is always hit?" Check latency and other bottlenecks before raising it. Spare memory does not prove spare execution capacity.
• "What would trigger retuning?" Changes in model/backend, prompt or output distribution, prefix hit lengths, multimodal shapes, or the product's SLO.

Common mistakes

• Treating maximum length as an eager per-sequence reservation.
• Substituting average output rate for a tail-latency measurement.
• Dividing replicated state by TP or ignoring the smallest rank's pool.
• Measuring only successful completions while overload rejects many arrivals.

Key takeaways

• A 10,500-token prompt plus 2,000-token output allowance exceeds 12,288 by 212.
• The synthetic pool fits 47 maximum contexts, or 43 with the stated 10% reserve.
• Cap 40 passes only the illustrated TPOT test at 38 ms; real acceptance needs other latency, growth and overload checks.
• Context, scheduling and admission limits must agree with the product promise.