Training at Scale
Planning, running and rescuing training jobs large enough that failure is a schedule item: what a step actually costs, how memory decides the parallel layout, what the collectives are worth, and what to do when a run degrades or diverges.
Serving a model is a steady-state problem. Training one is a project with a deadline, a budget and a failure rate, and the engineering is mostly about those three meeting.
The arithmetic here is unforgiving in a useful way. A run is a fixed amount of work divided by a rate you can estimate, on hardware that fails at a rate you can measure, and almost every decision is a consequence of those two numbers rather than a preference. That makes it a good subject to reason about carefully and a bad one to improvise.
Four modules. The run first, because you cannot plan what you cannot cost. Then the layout, which is where memory forces choices people think are theirs. Then the economics: utilisation honestly measured, checkpoints priced against failure, and the schedule that falls out. Then the rescue, because at this scale runs degrade and diverge, and the difference between a good and a bad infrastructure engineer is mostly how quickly they establish which.
The capstone is a run plan: the document you would take into the review that decides whether to spend the compute.
When you finish, you can
- Break a training step into its four costs and say which one binds
- Derive a parallel layout from the memory budget rather than choosing one by reputation
- Treat mixed precision as a numerics contract and know where the wide accumulations must stay
- Price each parallel dimension by the collective it buys and the link it needs
- Compute MFU honestly and explain the gap between it and device utilisation
- Choose a checkpoint interval against a measured failure rate rather than a habit
- Diagnose a degraded or diverging run, including the hardware hypotheses
- Produce a run plan you can defend to a reviewer, in the capstone
BEFORE YOU START · Foundations of AI Infrastructure, or the equivalent: the roofline, the memory budget, and the habit of naming a binding resource before proposing a fix. Inference Engineering helps and is not required.
Syllabus
The run
What a training step is made of and what it costs: the four terms in a step, the memory budget that decides your parallel layout before you get an opinion, and the numerics contract that mixed precision is actually asking you to sign.
The layout
Choosing and placing the parallel mesh: what each dimension costs in communication rather than what it is called, how to map the mesh onto the machine so the chattiest traffic takes the shortest path, and what sharding gives up in exchange for fitting.