26Design the infrastructure for RLHF on a 70B: where do rollouts and the learner run, and how do weights move between them every step?▼hardNewOpenAIAnthropicNVIDIA4 replies◆ premiumGeneration is a decode workload and the update is a training workload, and they want different software on different GPUs. The per-token arithmetic for each side, the 141 GB broadcast that has to happen every step, the ratio of actor to learner GPUs that follows, and the idle time that on-policy training builds in.Open full answer →
28We are running GRPO at scale. What does the infrastructure have to do that plain RLHF did not, and where do the GPUs sit idle?▼hardNewDeepSeekOpenAINVIDIA4 replies◆ premiumSixteen samples per prompt, a verifier that runs code, no value model, responses to 16k tokens: GRPO moves the cost from the learner to rollouts and rewards. The token arithmetic for one step, the tail that holds a batch for the longest sample, and the three places a fleet idles.Open full answer →
