Follow two requests through one loaded model

Two requests can share one loaded model while keeping their own progress and attention state. Follow what stays in memory, what runs again, and why a waiting request needs both space and a turn to execute.

21 MIN

TL;DR: Two requests can use the same loaded model, but each needs memory for its own progress and a turn to run. The scheduler decides which work runs next, and the allocator assigns the memory it needs. A request may therefore have to wait even when the server has room for another request in its schedule.

Where you are. First lesson of the inference course. You have one model loaded and two people waiting for answers. Follow what they share and what each request needs, then work out why the second request can still wait for memory or an execution turn.

First identify what survives between requests

A support assistant answers one person successfully. A second person asks a different question. Does the server load another complete model? In a normal resident deployment, both requests use the same loaded weights. Their progress and attention state must still be tracked separately.

The weights are learned numerical arrays. The architecture specifies how operations use them. A tokenizer turns text into IDs, and a chat template arranges messages in the format the model expects. The engine loads a compatible model implementation and artifacts, prepares its memory and executes requests. The model introduction explains these pieces from the beginning.

For this lesson, follow one resident replica serving a single fixed model revision. A replica is a complete serving copy; it may span multiple devices. Ignore adapters, offload and prefix sharing initially so the ownership is easy to see.

A forward pass evaluates the model on some input and retained state. For ordinary causal text generation, processing the prompt is prefill. Its final-position scores can select the first output token. Later decode steps consume new tokens and extend the sequence. One forward pass is therefore different from the whole answer, which may need many iterations.

The key-value cache, or KV cache, stores attention data from processed tokens for reuse. A generated token's own keys and values appear when that token is processed. The token trace follows this boundary closely. Model weights normally remain unchanged throughout this work.

Return to the two people using the assistant. The weights stay loaded while each person's answer grows separately. When one request finishes or is safely cancelled, the engine can release that request's references to its state. The other request still needs its own state.

Using the same weights does not by itself save prompt processing. If both people send identical prompts, the engine still processes each unless a separate prefix-cache feature lets it reuse compatible attention state. Such a cache may keep reusable state after the original request finishes.

This distinction explains the opening symptom. A free request slot says nothing about whether the next sequence's state fits. You need both a scheduling decision and an allocation decision.

What does the request repeat?

The word inference covers different execution patterns. Before copying a text-serving configuration to another model, identify the work that repeats and the result the caller receives.

For a pooled embedding example, the encoder processes a passage and combines its hidden representations into one vector. That vector can contain hundreds of values without needing hundreds of generation turns. “One evaluation” still includes many layers and device kernels; it does not mean all computation happens simultaneously. Long-document splitting or a multi-stage encoder can add further work.

In autoregressive text generation, a selected output token becomes input to later computation. Prefill can select the first token; the decode loop extends the sequence while reusing compatible attention state. This lesson's remaining scheduler calculations follow that causal text workload.

For a latent-diffusion image example, a latent is an array representing the image in a compressed space. Start from noise, keep the prompt's encoded features available, and repeatedly update that array using model predictions. After the final update, an image decoder maps it to pixels. An update changes the evolving representation; it does not append one finished pixel the way a text loop appends a token.

These are reference patterns, not rules that assign every modality to one architecture. Some image models generate tokens; some pipelines combine encoders and generators. Identify the actual execution graph. In particular, a diffusion library's scheduler computes the numerical update between noise levels. The serving scheduler decides which request gets device time. Those two meanings describe different jobs.

For text, a first token and the following token gaps matter separately. An embedding caller waits for the usable vector. An image caller may see previews, but a progress event is not a completed image. Keep that output contract in mind as we return to the two text requests. The Premium engine reference works through the batch-size and completion-deadline decision.

A useful loop, with its limits

This is conceptual pseudocode, not an engine implementation:

for each scheduling iteration:
    observe completions and cancellations from prior work
    choose eligible prefill chunks and decode work within token/time budgets
    acquire or reuse state blocks; revise the choice if allocation fails
    execute the selected model work, possibly in a mixed batch
    sample/verify outputs and update progress
    release references that are safe to reclaim

A real engine may overlap these stages or schedule several steps ahead. Prefill and decode need not run as separate device calls. Prefill can produce a sequence's first sampled token; later decode advances selected sequences. Speculative decoding can accept multiple tokens after verification, so the number of emitted tokens need not equal the number of scheduled positions.

Several serving techniques change this loop. Continuous Batching lets requests enter between iterations, so one request need not wait for an entire batch to finish. Chunked Prefill limits how much new prompt work is processed in an iteration. PagedAttention stores attention state in blocks that do not need to sit next to one another in memory.

Prefix Caching and KV Reuse lets a new request use compatible state that already exists. That affects both halves of the engine: the allocator can reuse stored blocks, and the scheduler has less prompt processing to schedule.

The scheduler spends a finite iteration budget

The scheduler chooses eligible work under token limits, active-sequence limits, memory constraints and latency policy. A sequence is one evolving token history; a request can create several sequences when it asks for multiple candidates. Adding a request can increase step duration, but the effect is workload-dependent: unused compute or memory capacity may absorb some work with little latency change.

Consider an illustrative scheduler with a budget of 512 token positions. It prioritizes one decode position for each of 32 selected sequences, then fills remaining capacity with prompt work:

prefill budget this iteration = 512 - 32 = 480 token positions
prompt still unprocessed = 1,500 positions
minimum iterations at this fixed split = ceil(1,500 / 480) = 4
sanity: 3 × 480 = 1,440, leaving 60 prompt positions for the fourth iteration

This is a scheduling count, not a latency estimate. Prefill positions and decode positions have different execution costs; state growth, new arrivals and completions can change the split. Measure iteration duration and each request's wait. The Orca study establishes iteration-level scheduling as a way to admit work without waiting for an entire fixed cohort.

The allocator tracks physical state and ownership

The allocator keeps track of where request state is stored and which requests use it. With full attention, the KV cache grows as the request retains more tokens; other attention architectures can store state differently. The engine may reserve a large memory pool at startup and assign blocks from it as requests arrive. This explains why a GPU can show substantial reserved memory even when little of that pool belongs to active requests.

In a paged KV cache, the engine maps positions in a sequence to blocks in device memory. Compatible requests may point to the same unchanged prefix blocks. A reference count records how many users still need a block, helping the engine decide when it can reclaim it.

Identical-looking text is not enough to establish compatibility. The token IDs, model revision, adapter, treatment of token positions and other inputs affecting state must match the engine's cache rules. Its isolation policy must also permit the sharing.

When space is tight, the choices include evicting unreferenced cached prefixes, deferring admission, preempting active work for later recomputation, or moving state to another tier if supported. These actions have different costs. Dropping an unused cached prefix does not immediately trigger a prefill; a future request may miss it. Dropping an active sequence's needed state creates recovery work when it resumes.

Distinguish a capacity problem from a recovery loop

Suppose a controlled trace shows ten active sequences per second being preempted, each losing 2,000 tokens of state, and each later recomputing that span:

extra prompt work = 10 preemptions/s × 2,000 tokens = 20,000 tokens/s
ordinary prompt demand = 20,000 tokens/s
total prompt work with this recovery = 40,000 tokens/s
sanity: recomputation doubles prompt work in this example, not necessarily total GPU time

If goodput drops while preemptions and recomputed tokens rise, reduce admission or active reservations and repeat the test. That can stop repeated recovery from competing with useful work. Reverse an admission-only fix when the rejected demand must be served and a supported memory increase, shorter retained context or state representation meets the cost and quality constraints. More memory can resolve a real shortage; it is not inherently the wrong intervention.

A throughput cliff alone does not prove preemption. Check kernel changes, thermal or power limiting, host bottlenecks and communication. Likewise, a long prompt followed by worse token gaps suggests prefill interference, but the trace must show that work occupying the shared execution path.

Do this before moving on

First point to what is fed back in each repeated-work diagram: a chosen token with cached state, or an updated latent. An embedding dimension is not a generation-turn count.

Then annotate the two-request diagram: what can remain after A finishes, and what must stay available for B? Loaded weights remain; A's references can be released safely, while B's state must remain valid. Reusable cached blocks may also remain by policy.

Change the illustrative token budget to 256 while keeping 32 decode positions. Prefill receives 224 positions, so the 1,500-position prompt needs at least seven iterations at that fixed split. Predict which measurements could improve and worsen: shorter individual steps may help decode gaps, while the prompt may wait through more iterations. Then identify what extra state a speculative verification batch must reserve before it can run.

Go deeper

Serving Engines adds a Premium batch-size exercise where more images per second still misses the completion deadline.
Continuous Batching is the admit and evict steps in detail, and the changing membership of each iteration.
PagedAttention is the allocator's data structure, and the next module opens it up.
Capacity and Backpressure explains what to do when allocation cannot support the proposed work.
How PagedAttention works is the interview version of the allocator half.
GPU utilisation is 30% in serving: why? is a case that only resolves once you can see both components.

Key takeaways

Requests can share resident weights while needing separate state and progress. Scheduler, allocator and executor determine what runs and what fits.
A 512-position budget minus 32 decode positions leaves 480 for prompt work.
Ten recomputations/s of 2,000 tokens add 20,000 prompt tokens/s; measure recovery work explicitly.
Prefix eviction, active preemption and offload have different costs and should have distinct counters.
Count the repeated operation and define usable output before transferring a serving policy between model types.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1Two requests use the same loaded model revision. Does this automatically let the second request skip processing its prompt?
2A token budget is 512 and 32 selected sequences need one decode position each. How much budget remains for prefill?
3Goodput drops while ten preemptions/s each cause 2,000 tokens of recomputation. What evidence supports a recovery loop?
4Does evicting an unreferenced cached prefix immediately add a prefill?
5A pooled encoder returns a vector with 768 values. Does that imply 768 autoregressive generation turns?