Foundations of AI Infrastructure
What a GPU actually is, what happens between a prompt and a token, and how to size a model service before you deploy it. The trunk course every AI infrastructure path starts from, for engineers who can ship software but have never run a GPU in production.
A serving workload can fit its weights in device memory and still fail its latency target. This course teaches the calculations and measurements needed to explain why: arithmetic intensity, state growth, queueing, scheduling and startup.
The machine, request and arithmetic modules build the hardware model, trace a request and derive a provisional capacity plan. Worked examples distinguish GB from GiB, active sequences from user sessions, and per-user latency from aggregate throughput. The service module covers admission, cancellation and streaming correctness. The deployment module connects those decisions to probes and operational signals.
The capstone produces a capacity memo with explicit assumptions and a measurement plan. You can complete the arithmetic without a GPU; accepting the proposed fleet requires a representative benchmark and a failure-case test. Continue into the inference, training or fleet courses according to the work you want to do.
When you finish, you can
- Explain why a GPU is fast in terms of bandwidth and arithmetic intensity, not core count
- Trace a request from tokens through prefill and decode to the bytes each phase moves
- Derive weight and state budgets, then combine them with measured service capacity to size a fleet
- Report TTFT, token-gap behavior and end-to-end latency with explicit outcome counts
- Choose a batching policy and defend it against a stated latency target
- Diagnose a slow GPU service using request, scheduler and device evidence
- Produce a capacity plan a reviewer can argue with, in the capstone
BEFORE YOU START · You can program in Python and are comfortable on a Linux command line, with HTTP and with containers. No CUDA, no C++, and no machine learning maths beyond knowing what a matrix multiply is.
Syllabus
The machine
Reason about GPU throughput using arithmetic intensity, the memory hierarchy and measured bottlenecks. Derive a performance floor and choose a controlled experiment before changing hardware.
The request
Trace prefill and decode, distinguish weight reuse from state traffic, and measure TTFT, token gaps and end-to-end latency. Work through why a throughput gain can worsen one user’s latency.
The arithmetic
Calculate weights and architecture-specific state, check memory per rank, and turn session cadence into request demand. Combine memory limits with a mixed-workload benchmark to produce a provisional fleet plan.
The service
Define admission, departure and overload rules. Bound token work, propagate cancellation safely, reconcile retries and usage, and distinguish a completed stream from parseable partial content.
The deployment
Separate allocation, compatibility, startup and device-health failures. Connect request and scheduler signals to hardware evidence, then defend a capacity memo with measured inputs and explicit reversals.