An illustrated engineering reference / October 2026
From GPU Kernels to Production Systems
A warp waits for a load. A collective waits for one rank. A request waits in a queue. Learn the words that explain each delay, then connect them to the measurements and decisions that matter.
110 terms across GPU architecture, CUDA, memory, networking, distributed training, inference and reliability. Every entry includes an explanation, an example, a common confusion and a route into deeper study.
Premium guide PDFs and companion files require an active paid Premium membership or approved complimentary access. Referral Premium includes the rest of the site, but excludes these downloads. Keep your downloaded copy.
Already a member? Sign in to download →
An AI infrastructure glossary explains the vocabulary used to build and operate systems that train and serve models. It connects hardware terms such as streaming multiprocessor and device memory to software terms such as kernel, collective and KV cache, then follows their effects into latency, capacity and reliability.
This reference is designed for reading a profiler report, discussing a deployment, or preparing for a technical interview. Definitions sit beside mechanisms and worked examples, so you can explain why a term matters in the system you are investigating.
Follow occupancy, warp execution state, scoreboard stalls, arithmetic intensity and the roofline model. Separate resident work from ready work, and put a denominator on the profiler’s percentages.
Read the free warp-state entry →Work through KV cache, prefill, decode, continuous batching and goodput. Distinguish a memory ceiling from a concurrency level that meets the service’s latency targets.
Continue with the KV Cache concept →Compare data, tensor, pipeline and expert parallelism. Trace all-gather, reduce-scatter and all-reduce through the chosen topology before treating more devices as a speedup.
Explore the Training at Scale syllabus →Define TTFT, inter-token latency, goodput, SLO and error budget with explicit measurement boundaries. Keep failed requests visible when evaluating a change.
Study latency metrics and goodput →Inspect the memory-request example, an eight-cycle scheduler trace and a KV-cache capacity calculation. These are selected pages from the complete PDF. Open a page to enlarge it; the expandable text below provides its opening explanation.
Inside the guide
Read these samples as text →
A load has several meanings: a thread asks for a value, a warp generates memory requests, caches may satisfy those requests, and only some traffic reaches device memory. Keep each level’s byte count separate.
Use the teaching model from the web glossary: 32 active lanes, one four-byte load per lane, 32-byte sectors, distinct addresses and a sector-aligned base. Adjacent words touch four sectors. A stride of eight words puts each lane in its own sector. Shift adjacent words by one word and the span crosses five sectors.
A percentage needs a counted event, a scope and a denominator. Occupancy describes resident work; eligibility describes work ready to issue; utilization describes activity under a particular metric definition.
One simplified scheduler has eight opportunities and can issue at most one warp instruction per opportunity. Four warps remain resident. The eligible counts are 0, 1, 2, 0, 2, 1, 0, 2. Five opportunities issue work and three have no ready warp.
Serving combines prompt processing, incremental generation and state management. More concurrency can reuse weights effectively, but every active sequence needs state and every scheduler decision affects latency.
Consider a standard full-attention model with 32 layers, eight KV heads, head dimension 128 and two-byte cache values. There is no cache sharding in this example. An 8,192-token sequence consumes 1 GiB of logical KV state.
Suppose 32 active lanes each load one distinct four-byte word. With a sector-aligned base, adjacent words span four 32-byte sectors. Spacing the words eight positions apart makes the same instruction touch 32 sectors.
Useful payload: 32 × 4 = 128 bytes
Adjacent words: 4 × 32 = 128 requested bytes
Eight-word stride: 32 × 32 = 1,024 requested bytes
Those are requested sector bytes in a teaching model. Cache hits and reuse can change the traffic that reaches HBM. That distinction is the point: a precise definition should tell you which observation supports your conclusion.
Change the address pattern in the free interactive example →Each chapter opens with a mechanism drawing, a worked calculation and a self-check. The reference also includes a comparison table, a review worksheet, an alphabetical index with abbreviations, linked study paths and primary-source notes. Page numbers below come from the current PDF.
The book includes all 77 entries from the free GPU glossary, plus 33 systems terms. Additions cover NVLink, NCCL collectives, RDMA, model parallelism, FSDP, checkpointing, KV-cache management, latency and service reliability. The PDF’s worked cases connect those terms across layers.
For the kernel path, use GPU Execution Model, the Nsight Compute report question and the Kernels and the Machine course. For serving, continue through Inference Engineering. Each linked page shows its access requirements.
For a practical network design, the GPU Networking for AI Engineers guide expands the transport and topology vocabulary into cabling, security and commissioning.
The reference links 60 primary documentation and research sources. Its GPU terminology focuses on NVIDIA and CUDA, with AMD differences called out where relevant. Architecture-specific features retain their target restrictions. Calculations are illustrative engineering models; the book does not claim new hardware benchmarks.
The terminology selection was inspired by the Modal GPU Glossary, by Modal Labs and its contributors, licensed under CC BY 4.0. We consolidated and reorganized terms, rewrote explanations and examples, added original diagrams and study connections, and extended coverage to production systems. The 77 GPU reference entries retain CC BY 4.0 reuse rights; the book’s publication notice explains their scope. No endorsement is implied.
Useful primary references include the CUDA Programming Guide, Nsight Compute Profiling Guide, NCCL collective definitions and vLLM metric definitions. Check the exact hardware and software version before applying a feature description.
An illustrated reference to GPU execution, memory, CUDA tools, performance, networking, distributed training, inference serving and reliability. Each term includes an explanation, example, common confusion and study links.
The PDF includes all 77 web entries, 33 additional systems terms, nine illustrated worked cases, a comparison reference, a review worksheet and a linked alphabetical index. The 77-entry web GPU glossary remains free to read.
Paid Premium members and readers with approved complimentary access can download the complete PDF. Referral Premium does not include paid guide downloads. The landing page and three selected samples are public.
Its numerical examples are worked teaching models. It does not report new GPU benchmarks or certify a deployment. Primary sources and architecture restrictions accompany the explanations.
Yes. The entries and study paths link to relevant concepts, interview questions, course lessons and calculators on AI Infra Interviews. Each linked resource retains its own access requirements.
Download the complete illustrated glossary for offline study. Use its index during a design review, work through the chapter cases, and follow the links when a definition deserves a full lesson.
Premium guide PDFs and companion files require an active paid Premium membership or approved complimentary access. Referral Premium includes the rest of the site, but excludes these downloads. See the Terms and the PDF’s source-license notice. Share this public page, and send corrections with the edition and page.
