Training at Scale
Planning, running and rescuing training jobs large enough that failure is a schedule item: what a step actually costs, how memory decides the parallel layout, what the collectives are worth, and what to do when a run degrades or diverges.
A large training run needs a model of its work, a feasible memory and communication layout, and a recovery plan. The completion date depends on retained progress over the full calendar, including time spent saving, restarting and replaying.
Four modules develop that plan. The run covers dependency timing, per-rank memory and mixed-precision behavior. The layout compares communication patterns, physical placement and sharding schedules. The economics connects MFU with explicit accounting, checkpoint cadence and recovery costs. The rescue uses controlled evidence to investigate wrong updates and slow ranks.
The capstone combines these decisions in a synthetic 70B-model run plan. Its worked sensitivity table shows why a nominal fit within the allocation can still be too uncertain to promise. You will distinguish supplied assumptions from measurements and define the validation that could change the decision.
When you finish, you can
- Trace the critical path through input, compute, communication and update
- Derive a parallel layout from the memory budget rather than choosing one by reputation
- Specify and validate precision for weights, gradients, accumulation and optimizer state
- Price each parallel dimension by the collective it buys and the link it needs
- Compute MFU honestly and explain the gap between it and device utilisation
- Choose a checkpoint interval against a measured failure rate rather than a habit
- Diagnose a degraded or diverging run, including the hardware hypotheses
- Produce a run plan you can defend to a reviewer, in the capstone
BEFORE YOU START · Foundations of AI Infrastructure, or the equivalent: the roofline, the memory budget, and the habit of naming a binding resource before proposing a fix. Inference Engineering helps and is not required.
Syllabus
The run
Trace a training step's critical path, derive memory from the actual precision recipe, and verify how gradients and updates behave. Worked timelines, rank budgets and a rounding example connect the estimates to observable failures.
The layout
Compare parallel communication over consistent units of work, map a concrete mesh onto devices, and calculate sharding's persistent and transient memory. Choose between feasible layouts using measured exposed time and throughput.