AI Infra Interviews logo
Interactive explainers

See how AI systems actually run

Each explainer follows one concrete thing, such as a single prompt, through the machine. You change the input and watch the time, bytes and memory change, with numbers computed from a real run rather than drawn for illustration.

How a language model serves your prompt

Follow one prompt through a real model running in your browser: tokens, prefill, the KV cache, decode and sampling, with the time, bytes and memory each step costs.

  • Tokens
  • Prefill
  • KV cache
  • Decode
  • Sampling
  • Production scale
Open the explainer →

Free to explore; the production-scale stage is Premium.

Your prompt, as the model sees it
To2068·make919·a253·cup7118·of282·tea7188,28·first808·boil17663·the260·water913
Measured cost per token
Prefill
5.6 ms
Decode
23.0 ms
Decode costs 4.1× more per token: every decode pass reads all 135 million weights to make one token. Recorded with the same 8-bit model the explainer runs.

How one server serves many people at once

Batching, continuous batching and the paged KV cache, run on a scheduler you can watch pass by pass, with every pass time measured on a real model.

  • Requests
  • Batching
  • Static batching
  • Continuous batching
  • Paged memory
  • Preemption
  • Tuning
  • Production scale
Open the explainer →

Two stages free; the scheduler simulations and production scale are Premium.

Measured: one pass for a whole batch
batch 1
41 tok/s
batch 2
56 tok/s
batch 4
95 tok/s
batch 8
126 tok/s
batch 16
160 tok/s
batch 32
181 tok/s
Serving 8 people takes 2.6× the time of one pass and makes 3.1× the tokens. Measured on the same model as the inference explainer.

How GPUs add up their gradients

The all-reduce behind every data-parallel training step, run on real numbers in your browser: the ring, its cost, trees for small messages, hierarchical topology and overlap with the backward pass.

  • Gradients
  • Collect at one GPU
  • Reduce-scatter
  • All-gather
  • Cost
  • Latency
  • Topology
  • Overlap
  • Production scale
Open the explainer →

Two stages free; the ring simulations, cost, topology, overlap and production scale are Premium.

Summing a 7B model's gradients
8 GPUs
436 ms
ring
54 ms
64 GPUs
31.8 s
ring
552 ms
Collecting every gradient at one GPU, against the ring all-reduce, on H100 servers of eight GPUs. Lower bounds from the link rates.

Real numbers, not illustrations

Every value comes from running the system being explained. Recorded runs say where and when they were measured.

Light until you ask for more

Pages open on a recorded run. Running a model on your own device is a separate choice, with the download size stated first.

Connected to the lessons

Each stage links to the concept that explains it in depth. New to the field? Start with the basics.