How a language model serves your prompt

A language model does not answer a question in one go. It repeats one small job, predicting the next token, until it stops. Serving infrastructure exists to make that repetition fast and affordable for many people at once.

This page follows that repetition through a real model, SmolLM2-135M, with 135 million weights. It is tiny next to production models, but it is built the same way as most current ones: 30 layers, and attention that shares 3 sets of keys and values across 9 query heads (grouped-query attention). Every number below comes from that model processing the prompt you choose.

Recorded or live. By default you see a recorded run of the same model file, so the page loads instantly. Choose Run it live to download the model once and run it on your own device, with your own prompt.

The model never sees letters. A tokenizer splits the text into pieces from a fixed vocabulary, here 49,152 entries, and replaces each piece with its number. Common words are usually one token; rare words split into several. A leading space is part of the token, which is why " tea" and "tea" are different entries.

Token counts are what everything else is priced in. API bills, context limits, latency targets and GPU memory are all measured per token, so the first question about any workload is how many tokens go in and how many come out.

Try it. The presets use common words, so each word is one token and only punctuation stands alone. The examples under the prompt show words this tokenizer splits: a product name, a long word, and a number that costs one token per digit. In live mode, type your own.

Further reading: Tokens, embeddings and logits (Premium).

Before it can write anything, the model must process every prompt token. It does this in a single forward pass called prefill: all prompt tokens go through all 30 layers together.

Most of the work is multiplying each token's vector by the weight matrices. Each weight takes part in one multiply and one add for every token, so the work is about 2 floating-point operations (FLOPs) per weight per token. For 135 million weights that is about 269 million FLOPs per prompt token.

Doing the tokens together is cheaper per token, because the weights are read from memory once and used for every token in the pass. Prefill still takes longer as the prompt grows, and it is most of the time to first token (TTFT): the wait before the first word appears.

Try it. Compare the prefill time with one decode step in the timeline. Prefill processes several tokens; a decode step processes one. Divide prefill time by the number of prompt tokens and compare.

Further reading: Prefill vs decode, Latency metrics: TTFT, TPOT and goodput.

Attention lets each new token look back at every earlier token. To do that it needs a key vector and a value vector for each earlier token, in every layer. Recomputing them for the whole conversation at every step would repeat all the earlier work, so the engine stores them: this store is the KV cache.

The size per token follows from the model's shape: 2 vectors (key and value) × 30 layers × 3 KV heads × 64 numbers = 11,520 numbers per token. This runtime stores each number in 4 bytes (FP32), so every token adds 46,080 bytes; a server storing BF16 would use 23,040.

A row, once written, never changes: an earlier token's keys and values do not depend on what comes after it. That is what makes caching correct, and it is why the cache only ever grows by appending.

Try it. Step through decode and watch the cache gain one column per token. Switch layers in the heatmap: each layer stores different keys for the same tokens.

Further reading: The KV cache, KV cache sizing, PagedAttention (Premium).

After prefill, generation runs one token at a time. Each decode step feeds in only the newest token, reads the whole cache for attention, and produces the scores for the next token. Its own keys and values are appended to the cache.

Here is the cost that shapes serving: to produce one token, the step multiplies by every weight matrix, so it reads every weight from memory. In this 8-bit model that is 134.6 MB per token, almost all of the work, for a single token's worth of arithmetic. Memory bandwidth, not arithmetic speed, sets the pace. Servers answer this by batching: one pass over the weights serves many users' next tokens at once.

The cache is read every step too. It starts tiny next to the weights, but it grows with every token, and at a few thousand tokens it overtakes them. Long conversations are expensive because of that.

Try it. Drag the context length and find where the cache bytes pass the weight bytes.

Further reading: Bandwidth-bound decode throughput (Premium), Roofline model, Continuous batching (free account).

The last layer produces a score, called a logit, for each of the 49,152 tokens. Softmax turns the scores into probabilities. The model does not pick a word; the serving code does, using sampling settings sent with the request.

Temperature divides every logit before softmax: below 1 the likeliest tokens gain probability, above 1 the distribution flattens. Top-k keeps only the k likeliest tokens; top-p keeps the smallest set whose probabilities add up to p. Greedy decoding always takes the likeliest token, which is what the recorded runs do.

These settings change the text, not the cost: every candidate's score was already computed by the time sampling starts.

Try it. On the capital preset, notice that " Paris" is not the likeliest first token. Raise the temperature and watch probability move from the top choices to the long tail.

Further reading: Tokens, embeddings and logits (Premium), Structured outputs and constrained decoding (Premium).

Every step above happens in production too, with the same shape and much larger numbers. A 70-billion-weight model in BF16 reads about 141 GB of weights per decode step instead of 134.6 MB, and its cache costs 320 KiB per token instead of 45 KiB.

This section carries your prompt and output length onto production models and GPUs: prefill time, the decode floor per token, cache memory per conversation and how many conversations fit.