AI Infra Interviews logo
← Distributed Training & Parallelism / 27
mediumNewAnthropicOpenAIDatabricks

Your team wants to replace a PPO-style RLHF pipeline with DPO. What leaves the cluster, and what does the training job look like afterwards?

DPO deletes the rollout engine, the reward service and the weight broadcast, turning post-training back into one supervised job: a policy, a frozen reference and a fixed dataset. The memory, the FLOPs per token, the precompute that removes the reference from the loop, and the exploration you give up.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

DPO deletes the rollout engine, the reward service and the weight broadcast, turning post-training back into one supervised job: a policy, a frozen reference and a fixed dataset. The memory, the FLOPs per token, the precompute that removes the reference from the loop, and the exploration you give up.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced
🕸️ Distributed Training🔒 Premium
RL Post-Training InfrastructureReinforcement-learning post-training alternates between generating long responses with an inference engine and updating the policy with a training framework, and the two phases want different hardware, different parallel layouts and a copy of the same weights each. The infrastructure problem is keeping the generators busy while the learner trains, syncing terabytes of weights between them every step, and surviving the long tail of response lengths.
Foundational
🕸️ Distributed Training
Data Parallelism and DDPData parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.
Core
🕸️ Distributed TrainingSign in
ZeRO and FSDPZeRO and FSDP keep data parallelism's simple programming model but shard the optimizer state, gradients and parameters across ranks, cutting per-GPU memory from 16 bytes per parameter toward 16/N. The price is 1.5x DDP's communication and a dependence on tokens per GPU that decides when sharding stops paying and tensor parallelism takes over.
Advanced
🕸️ Distributed Training🔒 Premium
Tensor ParallelismTensor parallelism splits individual weight matrices across GPUs so each rank computes a slice of every layer, which is how a model whose single layer does not fit one GPU gets trained at all. It costs four all-reduces per transformer block on the critical path, which is why it stays inside the NVLink domain and rarely exceeds 8 ranks.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackWe are running GRPO at scale. What does the infrastructure have to do that plain RLHF did not, and where do the GPUs sit idle?Next in this trackWhy is fp8 training hard, and how did DeepSeek-V3 make it work?Next in this trackDesign a checkpoint format for thousands of GPUs: no gather on write, resumable at a different world size, no stall.
FEDITOR'S NOTE

Scored on naming what leaves the cluster rather than reciting the loss, on the reference-logprob precompute as the obvious infrastructure win, and on saying clearly that the cost is exploration, not quality per se.

DISCUSSION · 0

No comments yet — be the first to share your approach.