LLM Inference from Scratch: What to Test Before You Optimize
Learn how to validate an LLM inference implementation with numerical references, cache-state checks and deliberate bugs, then connect correctness evidence to serving benchmarks.
BY MAYA CASTILLO · AIINFRAINTERVIEWS EDITORIAL · UPDATED NOVEMBER 18, 2026 · 4 MIN READ
PRACTICE THIS:Run the Premium cache-correctness lab ·Study inference engineering ·Practice serving questions
Before optimizing an LLM inference implementation, prove that its execution paths compute the intended result on the same inputs. Compare full recomputation with cached execution, vary chunk boundaries, and check that page placement does not change sequence meaning. Only then use performance measurements to choose between correct implementations.
Tiny LLM is useful reading for engineers who want to connect model equations to a working serving system. Its MLX-based course builds from model execution toward kernels, caching and request scheduling on . Our addition takes a narrower task: an original Premium CPU correctness lab that makes several inference bugs visible without downloading weights or renting a GPU.
Give each learning step an observable result
Reading a serving architecture explains its components. Implementing a small part forces you to specify which values those components exchange. Both belong in a useful learning path, but they need different exit checks.
Start with prefill and decode: identify which input tokens a call processes and which output it predicts. Then study The KV Cache and explain which state survives the call. The project follows that introduction with executable comparisons. It uses one synthetic attention layer, so success means understanding and checking that layer's state; it does not produce a trained assistant.
A larger engine project introduces loading, tokenization and generation as additional boundaries. Keep a trusted implementation available at each boundary. A decoded sentence can look plausible even when an intermediate operation is wrong, and replacing everything at once makes the first mismatch difficult to locate.
Tiny LLM's own repository roadmap distinguishes implementation, tests, documentation and review. Some checkpoints are still marked under review. Follow the prerequisites for the exact revision you use; a complete library baseline is a useful control, but it does not establish that your own cache or scheduler has been implemented.
A cache changes the work, so test the preserved meaning
During full causal attention, a query may use its current position and earlier positions. Cached execution retains past keys and values instead of producing them again. Hugging Face's cache documentation describes that reuse and the combined past-plus-current state used by attention.
The useful comparison keeps input tokens fixed. If one execution generates a different token and you feed that token back in, later calls no longer receive the same sequence. Their differences can be consequences of the first mismatch. Record the earliest differing numerical output before investigating the rest of the generated text.
Three numbers often become confused: the position of a token in the sequence, the amount of valid cache history, and the capacity allocated for that history. A buffer can have spare capacity. A chunk can begin at a nonzero position. Neither fact changes which earlier tokens a query is allowed to read.
- Hold the input fixed
Use the same token positions, model settings and numerical reference.
- Change one execution choice
Split the prompt, reuse past state or permute physical pages.
- Locate the first disagreement
Compare outputs before sampling, then inspect the state at the failed boundary.
- Reject a deliberate bug
Show that the check fails when a required dependency or mapping is broken.
Test boundaries that ordinary examples skip
One request with a convenient length is a weak test of an allocator or mask. Choose a length immediately below a storage boundary, exactly on it, and one position beyond it. Vary how the same input is divided into chunks. A final short chunk exercises a different case from a sequence of equally sized calls.
Also change the relationship between query heads and KV heads. Grouped-query attention shares stored state across query heads; a mapping error may remain invisible when every head receives identical synthetic data. Use nonuniform values for indexing tests and reserve uniform cases for arithmetic checks with a known answer.
The new debugging question asks you to isolate a failure that appears after chunking is enabled. It complements our existing scheduler question, which concerns interference between requests. One tests whether the execution is correct; the other asks how to schedule work that is already correct.
Paging needs an execution check as well as an allocation check
A block table translates a sequence's logical position into physical storage. Correct attention must preserve logical order even when those pages are scattered. Test with a deliberately permuted page table; an identity mapping can conceal an implementation that ignores the table entirely.
The vLLM kernel design provides a concrete reference for page-based access. A separate concern is whether a candidate first gathers all the pages into a dense history before computing attention. Such a candidate can return correct outputs while paying an extra data-reorganization cost.
Our lab exposes that distinction with a modeled gather-element count alongside numerical comparisons. It does not infer hardware traffic from Python lists. The paged kernel question supplies the deeper reduction and addressing discussion, while actual device profiling determines whether an implementation is faster.
Keep the claims narrower than the evidence
The lab runs on Python's CPU floating-point arithmetic. Its tolerance belongs to that synthetic experiment. Production acceptance must account for the model, dtype, backend and permitted numerical differences. Small tensor differences can be acceptable, while a large indexing mistake should never be excused by increasing a threshold until the test passes.
After numerical checks, evaluate the tasks the service must perform. After task acceptance, benchmark the intended request workload. Preserve failed candidates and use the same measurement boundaries for each comparison. Our inference engineering course and benchmark project develop those later decisions.
Start by predicting one failing case in the cache lab. Run it, find the first mismatching position, and explain the state transition that caused it. That explanation is the useful artifact to carry into a larger implementation or an inference interview.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
Yes. Small CPU experiments can test attention dependencies, cache indices, head mapping and numerical comparisons. GPU throughput, kernel scheduling and device memory behavior require separate hardware measurements.
No. Different score vectors can select the same token. Compare numerical outputs on identical input tokens and include tests that deliberately corrupt a position, mask or storage mapping.
No. Tiny LLM provides a larger MLX-based implementation course. Our independent CPU project focuses on validating one synthetic attention layer and connects that exercise to inference interview preparation.
This article and the linked KV-cache introduction are public. The complete cache-correctness project, its source downloads and the new debugging answer require Premium. Course lessons retain their individual access labels.
Discussion (4)
A test suite deserves a negative control. If changing the cache offset leaves every check green, inspect the fixture before celebrating the implementation.
Especially with zero queries or identical value heads. Those inputs are useful algebra checks, but they can erase exactly the difference an indexing test needs to expose.
Keep numerical acceptance and task acceptance as separate results. A task set can miss a rare state bug; a strict tensor comparison can reject a harmless change in reduction order.
Cache lifetime deserves a test after cancellation too. A correct address is not sufficient if another request has already been allowed to reuse the storage.
