AI Infra Interviews logo
← Distributed Training & Parallelism / 27
mediumAnthropicOpenAIDatabricks

Your team wants to replace a PPO-style RLHF pipeline with DPO. What leaves the cluster, and what does the training job look like afterwards?

Offline DPO trains on existing preference pairs, so online rollout and reward services leave the update loop. Account for both responses, decide whether reference scores can be cached, and retain the data-generation and evaluation costs outside training.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Offline DPO trains on existing preference pairs, so online rollout and reward services leave the update loop. Account for both responses, decide whether reference scores can be cached, and retain the data-generation and evaluation costs outside training.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced
🕸️ Distributed Training🔒 Premium
RL Post-Training InfrastructureRL post-training connects response generation, scoring and policy updates. The infrastructure must coordinate model versions while handling variable response lengths and uneven phase costs.
Foundational
🕸️ Distributed Training
Data Parallelism and DDPReplicas combine gradients from different data. Follow a two-worker update, fix unequal-token weighting, and see why queued gradient buckets can leave more than one reduction exposed.
Core
🕸️ Distributed TrainingSign in
ZeRO and FSDPZeRO and FSDP reduce replicated training state by distributing it across workers. The memory saving comes with communication and temporary-buffer costs.
Advanced
🕸️ Distributed Training🔒 Premium
Tensor ParallelismTensor parallelism splits operations within a layer across GPUs. Derive the column/row matrix split, count training and inference collectives, and choose a group size from memory, latency and actual interconnect performance.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackWe are running GRPO at scale. What does the infrastructure have to do that plain RLHF did not, and where do the GPUs sit idle?Next in this trackWhy is fp8 training hard, and how did DeepSeek-V3 make it work?Next in this trackDesign a checkpoint format for thousands of GPUs: no gather on write, resumable at a different world size, no stall.
FEDITOR'S NOTE

Separate the offline training loop from dataset creation and evaluation. Count policy and reference work under an explicit recipe, and define the cache identity before removing the resident reference.

DISCUSSION · 0

No comments yet — be the first to share your approach.