We are running GRPO at scale. What does the infrastructure have to do that plain RLHF did not, and where do the GPUs sit idle?

Sixteen samples per prompt, a verifier that runs code, no value model, responses to 16k tokens: GRPO moves the cost from the learner to rollouts and rewards. The token arithmetic for one step, the tail that holds a batch for the longest sample, and the three places a fleet idles.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🕸️ Distributed Training🔒 Premium

RL Post-Training InfrastructureReinforcement-learning post-training alternates between generating long responses with an inference engine and updating the policy with a training framework, and the two phases want different hardware, different parallel layouts and a copy of the same weights each. The infrastructure problem is keeping the generators busy while the learner trains, syncing terabytes of weights between them every step, and surviving the long tail of response lengths.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPData parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.

Core

🕸️ Distributed TrainingSign in

ZeRO and FSDPZeRO and FSDP keep data parallelism's simple programming model but shard the optimizer state, gradients and parameters across ranks, cutting per-GPU memory from 16 bytes per parameter toward 16/N. The price is 1.5x DDP's communication and a dependence on tokens per GPU that decides when sharding stops paying and tensor parallelism takes over.

Advanced

🕸️ Distributed Training🔒 Premium

Tensor ParallelismTensor parallelism splits individual weight matrices across GPUs so each rank computes a slice of every layer, which is how a model whose single layer does not fit one GPU gets trained at all. It costs four all-reduces per transformer block on the critical path, which is why it stays inside the NVLink domain and rarely exceeds 8 ranks.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhy is fp8 training hard, and how did DeepSeek-V3 make it work?Next in this trackDesign a checkpoint format for thousands of GPUs: no gather on write, resumable at a different world size, no stall.Next in this trackYour 4,096-GPU run loses about 2% of every day to restarts. Fix it, and tell me where the floor is.

FEDITOR'S NOTE

Scored on knowing what GRPO removes (the critic) and what it multiplies (samples per prompt), on doing the per-step token budget, and on naming the tail latency of the longest sample as the dominant idle.

DISCUSSION · 0

No comments yet — be the first to share your approach.