A training step is four costs and only one of them is the model

TL;DR: Four terms: compute, communication, optimiser, and input. Compute is the one everyone models and often not the one that binds. The others hide, because they overlap with compute when things go well and become visible only when they stop. Name which term binds before optimising anything, exactly as in serving.

Where you are. First lesson of the training course. It opens on the decomposition every later decision refers back to, because a run's schedule is this arithmetic multiplied by a token count.

The four terms

step time  ~  max( compute, communication, optimiser, input )
              when overlap works

step time  ~  compute + communication + ...
              when it does not, which is the more common case

Compute. Forward and backward through the model. For a dense transformer, roughly six floating-point operations per parameter per token: two for the forward multiply-accumulate and four for the backward, which computes gradients with respect to both inputs and weights. This is the term with a clean formula, which is why it gets all the attention.

Communication. Gradients have to be combined across data-parallel replicas, and sharded setups also exchange parameters. This scales with model size and with how the job is laid out, and it is the term that most often surprises people.

Optimiser. Applying the update. Small in operations and not free in bytes: it reads and writes the optimiser state, which is several times the parameter count, so it is a memory-bandwidth cost rather than a compute one.

Input. Getting the next batch to the device: reading, decoding, transforming, transferring. Invisible when it keeps up and total when it does not.

The useful framing, and the one carried from Course 1: these are four candidate binding resources, and the fix for each is different. A run limited by communication does not improve with a faster accelerator. A run limited by the input pipeline does not improve with either.

Why compute gets over-modelled

It has a formula that fits on one line, it is what benchmarks report, and it is the term a model architect can discuss. So it becomes the number people plan with, and the plan is optimistic by however much the other three cost.

The correction is the gap between theoretical and achieved throughput. If your compute model says a step should take one second and it takes two, the missing second is in the other three terms, and finding which one is the entire job. MFU and HFU is the concept that formalises this and the next module treats it properly; the point here is that the gap is not measurement error, it is the other terms becoming visible.

That pair of pictures is most of what a training-infrastructure engineer does: keeping a run in the top state, and diagnosing which arrow moved it into the bottom one.

Overlap is the whole game

Communication and input can both be hidden behind compute, and the difference between a job at 40% utilisation and one at 15% is usually whether they are.

Gradients reduce as they are produced. The backward pass produces gradients layer by layer, starting from the output. Each layer's gradient can begin its exchange while earlier layers are still computing, so most of the communication happens underneath the compute rather than after it. Requires the framework to bucket gradients and start collectives early, which is standard and worth verifying rather than assuming.

Input is prefetched. The next batch is read and prepared while the current one is being processed, so the device never waits on storage.

Both have the same failure signature, from Course 1: the device idle in a regular rhythm. Bursts of work separated by consistent gaps means something is not overlapping, and which one it is depends on where the gap sits relative to the collective.

The residual that cannot be hidden is worth knowing about: the last gradient bucket cannot start its exchange until the first layer's gradient exists, so there is always a tail of communication after compute finishes. Making that tail small is what bucket sizing is for.

The arithmetic to carry

total work        = 6 x N x D          N parameters, D training tokens
achievable rate   = devices x peak x MFU
wall clock        = total work / achievable rate

MFU for a well-tuned large run: 0.35 to 0.5. Use 0.4 and say you assumed it.

Two lines and a caveat, and it is enough to answer "how long will this take" and "how many devices do we need", which are the two questions this course exists to answer properly. Training FLOPs: 6ND has the derivation.

The caveat matters: the MFU figure is where all four terms are hiding. Quoting 40% is assuming that communication, optimiser and input are mostly overlapped and the compute is reasonably efficient. When a real run lands at 15%, the arithmetic was not wrong, the assumption was, and the diagnosis is which of the four terms is exposed.

Do this before moving on

For a model and token budget you can name, compute the total work, then the wall clock on a device count of your choosing at 40% MFU. Write down the number.

Then ask what would have to be true for that number to hold, and list the three assumptions: that gradient exchange overlaps the backward pass, that input prefetching keeps up, and that the compute itself is efficient. Those three are what you will spend the run defending, and having named them in advance is what makes a slow run diagnosable rather than disappointing.

Go deeper

• Training FLOPs: 6ND is the compute term derived properly, including where the factor of six comes from.
• MFU and HFU is the gap between theoretical and achieved, and how to measure it without fooling yourself.
• Data Parallelism and DDP is the communication term in its simplest form, and where bucketing lives.
• Collective Communication Primitives is what the exchange actually does and what it costs.
• Data Loading Pipelines for Training is the input term, which is the one most often left out of a plan entirely.
• Your training run is slow: compute, memory, network or IO? is this decomposition as a debugging question.

Key takeaways

• Four terms: compute, communication, optimiser, input. Step time is their maximum when overlap works and their sum when it does not.
• Compute is over-modelled because it has a clean formula; the gap between predicted and achieved is where the other three live.
• Gradient exchange hides behind the backward pass and input hides behind compute, and both fail with the same signature: idle in a regular rhythm.
• There is always a communication tail that cannot be hidden, which is what bucket sizing addresses.
• Total work is 6ND, wall clock is that over devices times peak times MFU, and the MFU assumption is where all four terms are concealed.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1A team models a run using 6ND and a 40% MFU assumption, then measures 15% in practice. What does the gap represent?

   Thermal throttling across the fleetThe 6ND formula not applying to this architectureOne or more of communication, optimiser and input becoming exposed rather than overlapped, since the MFU assumption presumes they are hidden behind computeMeasurement error in the utilisation figure

2. 2Why can most gradient communication be hidden, and what part cannot?

   Answer it, then reveal

3. 3A training job shows the devices busy in regular bursts separated by consistent gaps. Which two terms could be responsible, and how do you separate them?

   Communication and input, both of which produce this signature when they stop overlapping; separate them by where the gap sits relative to the collective and by whether enlarging the prefetch closes itCompute and communication; separate them by device countCompute and optimiser; separate them by profiling kernel timeOptimiser and input; separate them by checking optimiser state size