How to Train AI Agents: SFT, Distillation, RL and the Infrastructure Between Them
Choose an agent-training method from the evidence you have. Compare SFT, on-policy distillation and RL, then trace the data, reward and rollout failures that can invalidate a training run.
BY OWEN HARTLEY · AIINFRAINTERVIEWS EDITORIAL · UPDATED OCTOBER 29, 2026 · 4 MIN READ
PRACTICE THIS:Start the free agent-training lesson ·Explore the Agent Post-Training Systems course ·Practice distributed-training interviews ·Read the Premium distillation reference
Training an AI agent means choosing a learning signal for its decisions and building a reliable path from interaction to update. Supervised fine-tuning (SFT) uses demonstrations. Distillation uses teacher feedback, sometimes on the student's own generated histories. Reinforcement learning (RL) uses scored outcomes. The right starting point depends on the failure you can reproduce and the supervision you can trust.
Before training, check the surrounding system. A tool parser, missing dependency, dirty workspace or incomplete verifier can make a good model appear to fail. Training against those mistakes can make an experiment more expensive without repairing the cause.
Hugging Face's agent post-training curriculum provides a useful progression from demonstrations to full agent harnesses. We used it as a starting point for an original Agent Post-Training Systems course, focused on the engineering decisions between those stages. Its opening lesson is free; the deeper exercises are Premium.
Choose the method from the evidence
A repair agent that repeatedly emits invalid tool arguments may benefit from demonstrations of valid calls. An agent that takes an unhelpful path and cannot recover may benefit from guidance at the states it actually visits. A task with a reliable outcome check can support learning from attempted solutions even when writing ideal demonstrations is difficult.
| Approach | Signal and systems check |
|---|---|
| SFT | Reviewed demonstration targets. Did preprocessing retain the intended tokens? |
| On-policy distillation | Teacher feedback on student prefixes. Can compatible events be scored within budget? |
| Outcome-based RL | Scores for sampled attempts. Does each score represent a valid task outcome? |
| Stateful training | Actions, observations and an episode outcome. Are reset and stopping rules reproducible? |
| Async execution | Overlapping rollout production and updates. Does accepted work satisfy the policy-age rule? |
The last two rows describe the execution setting. They can coexist with different learning objectives. There is no requirement to complete every row before training a useful agent.
SFT starts with the target selection
A conversation contains tokens written by the user, the assistant and tools. Under an assistant-action training recipe, the model needs the tool's observation as context but should not be trained to produce that observation as its own action.
The mistake can be difficult to spot from a loss curve. Adding many easy tool-output targets can lower an average even when the assistant's predictions are unchanged. Truncating a long observation can remove the final patch while leaving earlier assistant targets, so a simple “some targets remain” check misses the problem.
Inspect final token IDs, role boundaries, selected labels and prediction positions after all preprocessing. TRL's SFT documentation distinguishes assistant-only and completion-only objectives; chat-template documentation explains the serialization boundary. The Premium SFT regression interview question works through the diagnostic comparison and a numerical counterexample.
Distillation adds a teacher service to the budget
A teacher can provide generated demonstrations or probabilities for possible next tokens. On-policy distillation uses student-generated prefixes, which lets guidance cover mistakes the student actually makes. The GKD paper develops this approach and examines different losses.
The systems implication is a second scoring workload. Count the teacher's input positions, selected output positions, batch shapes and feedback representation. A tool observation can add substantial context-processing work even when its target loss is zero. Equal vocabulary sizes do not prove that teacher and student token IDs describe the same events.
The Premium On-Policy Distillation reference includes a small executable divergence example and a full-distribution memory calculation. The course then turns those quantities into a teacher capacity worksheet. Neither teacher agreement nor a lower divergence replaces an independent task evaluation.
Rewards need an outcome contract
A failed test is a task outcome when the test executed correctly. A verifier process that disappeared supplied no such evidence. Substituting zero for an unknown score can change the relative advantages in a group and teach the policy about platform reliability.
Task budgets also need explicit meaning. Exhausting a declared action limit may be a valid failure under that task definition. A shared service queue causing an external timeout may instead make the attempt unscorable. Preserve the typed cause and specify the group's handling before training starts.
The existing GRPO infrastructure question covers response groups, verifier capacity and zero-variance rewards. The AgentGym guide, a Premium resource, develops reset, reward and evaluation exercises in greater depth.
A training record needs more than a final score.
- Start from a known task state
Pin task identity, environment and harness configuration.
- Capture actions and observations
Preserve sampled tokens, eligibility masks and policy identity.
- Validate the outcome
Record a task score or an explicit unscorable cause.
- Admit the training record
Apply the recipe's group, provenance and age rules.
Faster rollouts do not guarantee faster learning
The complete path may wait on model generation, tool execution, reward verification, teacher scoring, learner updates or policy publication. Measure the boundary where time accumulates. A generation optimization has little value when its outputs wait behind a saturated verifier and then expire.
Asynchronous execution overlaps stages but introduces queue and sampling-delay decisions. TRL's asynchronous trainer documentation describes one implementation. Its controls do not provide a universal safe staleness threshold. Compare synchronous and bounded asynchronous experiments using the same task population and total resource budget.
Report accepted learning work alongside raw generated volume. Also retain scheduled tasks, valid scores, infrastructure failures and exclusions. A pipeline that drops slow, difficult episodes can look faster while changing the learning problem.
Build a reviewable training plan
Start with the free lesson on choosing a training signal. Then work through the Premium trace audit, teacher budget, reward contract and rollout-capacity exercises. The final project joins them in a launch decision where an aggregate gain hides a protected-task regression.
For actual small-model training, the separate tiny language-model post-training project provides an executed experiment. The new course uses original calculations and CPU exercises to establish systems reasoning. It makes no claim that those exercises train a production agent or validate a GPU deployment.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
SFT learns from selected target tokens in demonstrations. Reinforcement learning changes the probability of sampled behavior using rewards. Both require careful data and evaluation contracts; a successful-looking transcript or a reward number can still be misleading.
The student generates a continuation, and a teacher provides guidance at the resulting prefixes. This lets training visit states the student actually reaches. The loss, token alignment and teacher-scoring cost must still be specified.
Stateful tasks need an explicit account of how actions change later observations. An environment provides that interaction and reset contract. A particular environment framework is optional, and its presence alone does not establish secure execution or a correct reward.
The opening lesson is free to read without signing in. The five deeper lessons, including the launch-review project, require Premium. This article is public.
Discussion (4)
The hardest evaluation tasks deserve a second look before training. If every one requires a missing dependency, the benchmark is measuring the image build. Keep the failed run; it is useful evidence even when it contributes no policy reward.
And rerun the unchanged baseline after fixing that image. Otherwise the infrastructure gain gets credited to the new policy.
Teacher scoring needs its own workload description. A rate measured on short plain-text prompts tells us little about long tool histories.
Accepted positions per second needs a cost beside it. Moving a bottleneck to a much larger teacher pool can improve that rate while making each completed experiment more expensive.