RL Post-Training Infrastructure

Record a memory timeline for initialization, generation, log-probability computation, backward, optimizer step and weight publication. Capture every rank's maximum, not just rank zero. The first successful update is necessary evidence; a later long episode or second publication can expose a larger allocation.

03 / UNDERSTAND AND SIZE THE LOOP

Choose where the phases run

The trainer spends two minutes waiting for attempts, then the samplers wait while it updates the policy. It is tempting to split them into separate fleets immediately. That may shorten an iteration, but it also changes how many GPUs are rented, which weights each attempt sees and what happens when either side fails. First draw the time spent by each phase.

In a colocated design, generation and training use the same GPU pool at different times. In a separated design, each has its own devices. Synchronous describes coordination: an update waits for the required batch and the next batch uses the published policy. Asynchronous describes overlap between generation and learning. These axes are related but different. Separate fleets can still execute synchronously, and specialized implementations can share devices while overlapping selected work.

Compare time and resource use together

Assume one batch takes 120 seconds to generate, 20 seconds to finish scoring, 45 seconds to train and 15 seconds to publish weights and prepare the next phase. The serial cycle is 200 seconds. If the relevant stages can overlap across separate pools, an optimistic steady-state interval is the larger of generation-plus-scoring, 140 seconds, and training-plus-publication, 60 seconds. That is 140 seconds, not 60.

What the diagram shows

Illustrative serial timeline: generation 120 seconds, scoring 20, training 45 and publication 15, totaling 200 seconds. Separate pools can ideally overlap a 140-second collection path with a 60-second learning path. At 16 versus 24 allocated GPUs, the examples cost 3,200 versus 3,360 GPU-seconds per batch. Rates must be remeasured after placement changes.

Figure 3. Overlap shortens the cycle but uses more GPUs

What the diagram shows

Active version 41 keeps serving while version 42 is staged. Every required shard must match its manifest. Only a complete validated replica activates version 42. A missing or corrupt shard rejects the staged version and leaves 41 active. In-flight attempts require an explicit version and cache policy.

Figure 4. Stage and validate before activating a policy

Put a lower bound on transfer

The interview prompt asks for a 200B-parameter policy copied after every update. At two bytes per parameter, its raw weights are 400 GB, or 372.53 GiB. An effective aggregate path of 50 GB/s needs at least eight seconds to move one complete copy across that boundary. Here GB and GB/s are decimal, and the rate is an assumed payload rate after protocol overhead. An advertised link rate in Gb/s must first be divided by eight and then adjusted for the usable path.

If packing takes three seconds, installation takes two and validation takes one, a serial publication costs 14 seconds. Those phase times are assumptions. With a 45-second training step, that is substantial exposed work. Compression, overlap or publishing less frequently may help, but each changes a different part of the system. Publishing less often increases policy age; lossy transfer changes the sampler's distribution.

Four independent rollout replicas each need a complete model. If all four copies must cross one 50 GB/s bottleneck, the byte floor becomes 1,600/50 = 32 seconds. A broadcast tree can avoid sending four complete copies over that particular edge if downstream nodes forward data. It still has to deliver every required tensor to every receiver. Draw the network cuts and bytes crossing each; do not multiply the advertised bandwidth of unrelated links.

NCCL broadcast sends a root buffer to participating ranks, while all-gather assembles contributions from them. Collective calls require matching participation, count and datatype. Training shards and inference shards may differ, so a literal broadcast of the full model to every rank can waste memory and traffic. Define tensor ownership and the resharding operation before choosing the collective. [6]

What the diagram shows

One immutable checkpoint and exact token IDs feed the rollout probability record and a teacher-forced trainer evaluation. Both paths align assistant-token positions and sampling conventions before comparing signed log-probability differences. Only after fixed-policy agreement is understood should an optimizer update be introduced.

Figure 8. Compare identical actions through both execution paths

Diagnose from identity toward arithmetic

Document whether saved probabilities are before or after temperature and filtering. A top-p or top-k sampler changes the support of the behavior distribution; a probability from an unfiltered model is not automatically its action probability. Use the convention required by the chosen framework objective, and test it with a tiny distribution where the expected values can be calculated by hand.