Advanced

On-Policy Distillation

On-policy distillation asks a teacher to guide the student on prefixes the student generated. Understand distribution matching, token alignment, teacher-scoring cost and the evaluation needed to trust a smaller policy.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Pretraining, Fine-Tuning and Distillation RL Post-Training Infrastructure Forward Pass, Backpropagation and Optimizer Steps Evaluation and Data Pipeline Infrastructure Model Memory Footprint

LESSONS THAT TEACH THIS

Agent Post-Training SystemsBudget teacher feedback on student traces18 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismThe student generates faster, but distillation throughput is flat. How do you find the bottleneck?→Distributed Training & ParallelismDesign a distillation pipeline that improves a target task without hiding capability regressions→Distributed Training & ParallelismDesign the infrastructure for RLHF on a 70B: where do rollouts and the learner run, and how do weights move between them every step?→Distributed Training & ParallelismWe are running GRPO at scale. What does the infrastructure have to do that plain RLHF did not, and where do the GPUs sit idle?→Distributed Training & ParallelismYour team wants to replace a PPO-style RLHF pipeline with DPO. What leaves the cluster, and what does the training job look like afterwards?→Distributed Training & ParallelismIn data-parallel training, what actually gets communicated between GPUs, and how much is it per step?→