rlhf

Generation is a decode workload and the update is a training workload, and they want different software on different GPUs. The per-token arithmetic for each side, the 141 GB broadcast that has to happen every step, the ratio of actor to learner GPUs that follows, and the idle time that on-policy training builds in.

DPO deletes the rollout engine, the reward service and the weight broadcast, turning post-training back into one supervised job: a policy, a frozen reference and a fixed dataset. The memory, the FLOPs per token, the precompute that removes the reference from the loop, and the exploration you give up.