Read the client clock before tuning the server

TL;DR: A server can send response headers while a request is still waiting for the model. Time the first token at the client, count rejected requests alongside completed ones, and choose queue limits from the caller's deadline and fallback options.

Where you are. This free lesson bridges the local model benchmark and the Premium serving workshop. You can work through the example without running the earlier lab.

A fast model can have a slow front door

Imagine a model that needs 200 ms to produce its first token and 600 ms to finish a response. Those are hypothetical values for this exercise. Four readers ask at nearly the same time, but the worker can execute only one request, so the other three wait. Giving each waiting reader a quick HTTP acknowledgement does not make their answer arrive sooner.

HTTP carries requests and responses between the client and server. A response starts with a status and headers describing the body to follow. Streaming sends that body in pieces over time. In this exercise, a token event contains one predicted token ID; headers and keepalive messages contain no prediction and must not stop the first-token clock.

Time to first token (TTFT) measures the interval from when this client begins sending the request until it parses the first token event. It includes the local connection, request transfer, waiting, server work and delivery back to the client. It stops before text is drawn on a screen. State that boundary beside the result so someone comparing another measurement can check whether the clocks cover the same work.

Follow four arrivals through one worker

Assume the worker processes one complete response at a time, each takes exactly 600 ms, no request is cancelled and transport overhead is zero. The requests are numbered by execution order, because arriving simultaneously does not guarantee which reader gets which position.

Completion times are 600, 1,200, 1,800 and 2,400 ms. With a 1,500 ms completion deadline, two of four requests finish in time. Sanity check: four serial responses require 4 × 600 = 2,400 ms even though all arrived together. More waiting slots do not make the worker faster.

Now admit only one request and promptly reject the other three. The accepted request finishes in 600 ms, making accepted-request latency look better, but timely completions fall from two of four offered requests to one of four. Rejection may still help if the caller can use a fallback. Deciding whether it does requires both the latency of accepted work and the outcomes for callers who were refused.

Choose the denominator before the dashboard

Finishing a stream means the client received the expected output; it does not establish that the answer is correct or supported. The earlier model lab checks those outcomes separately. Its fixed-token diagnostic is useful for comparing execution, but predictions made after the model's stop signal cannot be counted as useful answers.

Decide what happens when there is no room

When a service admits a request, it reserves capacity for that work. A bounded queue lets some admitted requests wait, while requests beyond the limit are rejected. Signaling that limit upstream is a form of backpressure. Choose a waiting budget that leaves room for execution before the caller's deadline, using measured service times, and then test whether the policy achieves the intended result.

A rejected client may try again, creating another attempt at the same logical request. If every client retries immediately, those attempts can produce another overload burst. A retry delay hint helps, but clients also need a retry limit, varied delays called jitter, and an overall deadline. The workshop performs no automatic retries, so the original four requests stay easy to follow.

A short queue can absorb a brief burst when the waiting requests still have time to finish. Permit that wait if measurements show it recovers useful completions that immediate rejection would lose. Shrink the queue when requests spend their entire budget waiting and then execute too late. The deadline and caller's fallback determine which outcome is preferable.

Do this before moving on

Repeat the example with a 900 ms completion deadline. How many of the four complete in time with waiting, and how many with a one-request admission limit? Both policies deliver one timely completion, but the waiting policy still spends work finishing three late responses. Write the fallback or retry behavior that would make rejection useful to the caller. There is no universal winner without that product contract.

The following Premium workshops give you a real local API and a repeatable experiment. They keep model work serialized so you can see the queue directly; continuous batching and production fleet scheduling are later steps.

Go deeper

• Latency Metrics: TTFT, TPOT and Goodput defines the clock boundaries used in a serving report.
• Capacity and Backpressure connects waiting limits to overload and caller behavior.
• Designing for Latency SLOs turns a deadline into budgets across the request path.
• Failure modes of token streaming explains cancellation and incomplete delivery beyond the successful response.

Key takeaways

• Acknowledgement time and first-token time measure different events.
• Four serial 600 ms responses finish at 600, 1,200, 1,800 and 2,400 ms.
• Count rejected callers when reporting timely completions per offered request.
• Queue limits protect a budget; they do not create execution capacity.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1Four serial responses take 600 ms each. With a 1,500 ms completion deadline, how many finish in time?

   OneFourTwo

2. 2Why can accepted-request latency improve while fewer callers succeed?

   Answer it, then reveal