Design the infrastructure for RLHF on a 70B: where do rollouts and the learner run, and how do weights move between them every step?

Generation is a decode workload and the update is a training workload, and they want different software on different GPUs. The per-token arithmetic for each side, the 141 GB broadcast that has to happen every step, the ratio of actor to learner GPUs that follows, and the idle time that on-policy training builds in.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🕸️ Distributed Training🔒 Premium

RL Post-Training InfrastructureReinforcement-learning post-training alternates between generating long responses with an inference engine and updating the policy with a training framework, and the two phases want different hardware, different parallel layouts and a copy of the same weights each. The infrastructure problem is keeping the generators busy while the learner trains, syncing terabytes of weights between them every step, and surviving the long tail of response lengths.

Core

🚀 Inference & ServingSign in

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

Advanced

🚀 Inference & Serving🔒 Premium

Chunked PrefillA long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.

Foundational

🧮 Open Weights & Serving Engines

vLLM Server Arguments That MatterA vLLM deployment is mostly decided by a dozen flags, and the ones that matter fall into four groups: how the model is split across GPUs, how memory is divided between weights and cache, how requests are batched, and which specialized backends the model needs. Getting the first two wrong produces an engine that will not start or that runs out of memory under load. Getting the third wrong produces an engine that starts, serves, and misses its latency target by a wide margin.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour team wants to replace a PPO-style RLHF pipeline with DPO. What leaves the cluster, and what does the training job look like afterwards?Next in this trackWe are running GRPO at scale. What does the infrastructure have to do that plain RLHF did not, and where do the GPUs sit idle?Next in this trackWhy is fp8 training hard, and how did DeepSeek-V3 make it work?

FEDITOR'S NOTE

Scored on separating the two workloads by their arithmetic intensity, on sizing the weight broadcast and knowing where it hides, and on deriving the actor-to-learner ratio from per-token costs rather than asserting it.

DISCUSSION · 0

No comments yet — be the first to share your approach.