Inference Engineering
The mechanisms that decide throughput, time to first token and cost per token in a model serving fleet: how the scheduler and the cache allocator actually work, what to measure, and which lever to pull when a number moves the wrong way.
Foundations sized a service. This course examines the scheduling, state and execution decisions that determine whether it meets its quality, latency and cost requirements under a changing workload.
Paging and prefix reuse change state allocation and repeated work. Chunking changes how prompt processing shares execution with generation. Precision changes representation and kernel behavior. Phase separation changes placement and introduces a state handoff. Each mechanism needs its own resource model; the course connects them through worked examples and controlled measurements.
Eight modules cover the engine, cache, scheduler, precision, fleet, the tuning pass, measured optimizations and output contracts. You will calculate physical blocks and recovery costs, compare arrival models, test admission policies without hiding rejected demand, and evaluate quantized candidates with slice-level quality evidence. The fleet module covers attention state layouts, tensor parallelism and phase separation, and its capstone produces an SLO review and a runbook from a consistent synthetic scenario. The tuning pass works from measured runs on one GPU: which cap binds concurrency, how a batch job differs from live traffic, a tuning log with a prediction per step, and a lab capstone that ends in a release packet for the gateway and evaluation owners.
The measured-optimizations module measures complete speculative rounds, separates model routing from replica affinity, and joins request traces to service outcomes. Its teardown capstone requires configurations, raw attempts, an analysis recipe and a measured mechanism. Hardware-free calculations prepare the experiment; they do not stand in for device measurements.
Dated blocks connect these mechanisms to specific implementations and model configurations. Use them as starting points for checking the exact revision you intend to deploy.
The output-contracts module follows reasoning through answer delivery and tool calls through validation. Its CPU exercise tests completion, argument structure, caller permissions and request identity before the learner plans real-engine qualification.
When you finish, you can
- Explain how scheduler, state allocator and executor decisions interact
- Design a benchmark whose numbers survive contact with production traffic
- Choose components by the layer they occupy rather than by a throughput claim
- Reason about paged cache allocation, prefix reuse and eviction as capacity decisions
- Evaluate scheduling and admission against first-token latency, token gaps and all-offered success
- Specify tensor formats and validate quality, kernel behavior and serving cost
- Decide when splitting prefill from decode is worth its transfer cost, and when it is not
- Produce an SLO review and a runbook for a fleet, in the capstone
- Find the binding concurrency cap, plan a batch job to a deadline, and tune a measured replica into a priced, handed-off release
- Evaluate speculative decoding and model routing with full-path cost and quality evidence
- Produce a reproducible serving teardown with feasible latency-throughput-cost comparisons
- Budget reasoning latency and qualify streamed tool outputs across supported feature combinations
BEFORE YOU START · Foundations of AI Infrastructure, or the equivalent: you can derive the KV cache formula, identify when decode is bandwidth-limited and when other resources bind, and size a fleet from a workload description.
Put the course into practice
Build a project after the linked lesson. Keep the scripts, measurements and written decision as evidence of your work. Hardware requirements and execution limits appear on each project.
- Premium · Performance project
Tune vLLM serving parameters with controlled experiments →
Recorded serving configurations, raw request evidence and a comparison that rejects mismatched experiments.
- Premium · Distributed service project
Build a prefix-aware routing proxy and test its failure paths →
A tested local routing proxy, policy experiment records and a cache-residency limitation analysis.
- Premium · Portfolio capstone
Defend an inference benchmark with quality gates and cost evidence →
A reproducible comparison, an eligible Pareto frontier and a written decision with a reversal condition.
After: SLO review and runbook
- Premium · Intermediate
Trace inference requests with OpenTelemetry, Jaeger and Grafana →
A checked trace and metrics pipeline, four request journals, and an acceptance record that distinguishes missing telemetry from failed inference.
- Premium · Intermediate
Speculative decoding: measure when it helps →
Paired GPU request records and a decision report that separates accepted drafts, complete output and correct answers.
After: Measure whether accepted proposals improve useful throughput
Syllabus
The engine
Connect scheduler, allocator and executor decisions. Build workload-aware benchmarks and distinguish execution, request handling and lifecycle control when choosing components.
The cache
Calculate paged allocation and prefix reuse with physical blocks. Separate unused-cache eviction from active preemption, and compare state recovery with offload using measured costs.
The scheduler
Track selected work and bounded state growth, compare chunk schedules, and combine admission constraints while reporting both latency and refused demand.
The precision
Specify weight, activation, accumulation and state formats. Verify execution paths, calculate metadata and speedup bounds, and evaluate quality by task and workload slice.
The fleet
Calculate architecture-specific state, compare tensor-parallel layouts and phase separation, and produce an evidence-based SLO review with an actionable runbook.
- Attention architecture determines which state the serving engine must retain29 minpremium
- Splitting a model across devices buys memory and spends interconnect15 minpremium
- Splitting prefill from decode answers a problem you must have first30 minpremium
- Capstone: investigate an SLO miss and write a usable runbookproject330 minpremium
The tuning pass
Find the cap that binds concurrency, plan a batch job against its deadline, tune one change at a time with a measured log, and hand a priced, tested deployment to the gateway and evaluation owners.
- Concurrency is set by whichever cap binds first22 minsign in
- A batch job is judged at its last item, not its average throughput22 minsign in
- Tune in the order the bottleneck moves, and log a prediction for every step26 minpremium
- Capstone lab: deploy one model, tune it, run a batch and hand it offproject360 minpremium
Measured optimizations
Test speculative decoding and model routing against complete-path cost and quality, then produce a reproducible serving-engine teardown.
Reasoning and output contracts
Budget the wait for useful answers, validate streamed tool calls, and qualify a serving configuration against the features used together.