How one server serves many people at once

A serving engine rarely runs one request at a time. People send prompts whenever they like, with prompts and answers of every length, and the engine has to fit all of them onto the same weights in the same GPU memory.

Each request needs two kinds of work: its prompt processed once (prefill), then one pass per output token (decode). Everything on this page is about deciding which requests share each pass, and what that choice costs in time and memory.

The requests below arrive as people really do: independently, at random moments, with a set average rate. Their prompt and answer lengths vary within the ranges shown.

A decode pass reads every weight to produce one token. If eight people are waiting for their next token, the same pass can read the weights once and produce all eight. The extra cost is only the arithmetic for the extra tokens and the reads of each person's own cache.

The chart uses real measurements of the model from the first explainer, timed at each batch size and cache length. Move the sliders: the time per pass grows slowly with the batch at first, so the total tokens per second climbs steeply, while each person's own tokens arrive more slowly. Once every person's cache is long, the cache reads take over and a bigger batch stops paying.

That trade, total throughput against each person's speed, is the decision behind every setting on this page.

Try it. Find the batch size where adding people stops raising total tokens per second, at a short and at a long context.

Further reading: Continuous batching (free account), Bandwidth-bound decode throughput (Premium).

Watch a batch start together and finish together: requests that are done keep their slot as padding until the longest one ends, and newcomers wait outside. The simulator shows how many slots sit idle and what that does to waiting time.

The same requests, scheduled again before every pass: finished requests leave at once and waiting ones take their slot. Compare throughput and time to first token with static batching on the identical workload.

Watch the KV cache fill block by block. Compare reserving room for the longest possible answer with allocating 16-token blocks as each answer grows, and see how many more people fit.

Shrink the cache and watch the scheduler evict a request and recompute it later, as vLLM does by default. See what preemption costs the evicted request and everyone else.

Change the batch limit, the per-pass token budget and the arrival rate. Track time to first token, time per token, throughput and the share of requests that meet a latency target.

Carry the same scheduling arithmetic onto a 70-billion-weight model on H100 and B200 GPUs: how much cache each conversation needs, how many fit, and the throughput a batch can reach at a given latency.