Foundations of AI Infrastructure
What a GPU actually is, what happens between a prompt and a token, and how to size a model service before you deploy it. The trunk course every AI infrastructure path starts from, for engineers who can ship software but have never run a GPU in production.
Most GPU material is written for people training models. This is written for the people who have to run them, which is a different job with a different set of things that go wrong.
The role exists because a model is not a service. A checkpoint is a file, and turning it into something that answers ten thousand people at a latency somebody promised is infrastructure work: capacity, memory, scheduling, failure. The questions that decide whether it works are almost all arithmetic, and almost all of them can be done on a napkin before anything is deployed.
Five modules. The first three build the mental model and the arithmetic, and they are the ones people skip and then regret, because every later decision appeals back to them. The last two turn a model into a service and then run it somewhere other people can break it. The capstone sizes a real deployment and produces the memo you would take into a design review.
This is the trunk. Every learning path on the site starts here, and the specialisations assume you can do what this course teaches.
When you finish, you can
- Explain why a GPU is fast in terms of bandwidth and arithmetic intensity, not core count
- Trace a request from tokens through prefill and decode to the bytes each phase moves
- Size weights, KV cache and headroom well enough to say how many GPUs a service needs
- Report latency as the three numbers that matter, and say which one a change moved
- Choose a batching policy and defend it against a stated latency target
- Debug a GPU service that starts, runs and is quietly slower than it should be
- Produce a capacity plan a reviewer can argue with, in the capstone
BEFORE YOU START · You can program in Python and are comfortable on a Linux command line, with HTTP and with containers. No CUDA, no C++, and no machine learning maths beyond knowing what a matrix multiply is.
Syllabus
The machine
What a GPU is in the only terms that decide an infrastructure answer: bandwidth, the memory hierarchy, and whether the thing you are running is waiting on arithmetic or on memory.
The request
What actually happens between a prompt arriving and tokens leaving: the two phases with opposite hardware profiles, why they cannot be optimised together, and the three latency numbers that describe them honestly.
The arithmetic
Sizing a model service from first principles: what the weights cost, what the cache costs per concurrent user, and how to turn both into a GPU count and a defensible plan without a spreadsheet or a benchmark.
The service
Turning a model into something other people can call: the scheduling decision behind batching, the API contract that makes token cost and cancellation explicit, and the correctness bugs that only appear once responses stream.
The deployment
Running the service where other people can break it: how a GPU pod fails in ways no other pod does, the three signals that tell you what is wrong, and a capstone that produces the capacity plan you would take into a design review.