Choose a training signal before choosing a trainer
Choose SFT for demonstrations, distillation for teacher feedback and reinforcement learning for scored outcomes. First reproduce the agent failure and test whether the harness or environment caused it.
12 MIN
TL;DR: Choose the learning signal you can trust: demonstrations, teacher probabilities or task outcomes. Establish a reproducible evaluation first. A broken tool interface or missing dependency can make a capable model look as though it needs training.
Where you are. This is the opening lesson. You will produce a one-page proposal for improving a repository-repair agent, then carry its assumptions through the rest of the course.
Find the failure in a complete attempt
Suppose an agent edits a repository but fails to repair a bug. Its model chooses actions; its harness assembles messages, invokes tools and decides when to stop; its environment contains the filesystem, dependencies and task state. Together they produce a trajectory: the ordered record of actions and observations through one attempt.
A correct edit followed by a missing test runner is an environment failure. A malformed tool call can arise from the model, serialization or parsing. A successful local test against an incomplete test suite may be a verifier problem. Preserve the initial repository revision, task, model, tool schema, observations and stopping reason before selecting a remedy.
Begin with a small held-out task set and inspect failures end to end. Try a corrected template, tool schema or environment image when the evidence points there. Training earns its place when the policy still makes a reproducible decision error under a functioning setup. A policy is the model's distribution over its next actions, represented here as generated tokens.
Trace one repair attempt before changing model parameters.
- Model proposes a test command
Preserve the exact generated arguments and message template.
- Harness submits the command
Compare proposed arguments with the command actually executed.
- Environment returns a result
Distinguish a failed test from a missing interpreter or lost worker.
- Verifier checks the repair
Check the resulting repository against independent requirements.
Match evidence to an objective
Supervised fine-tuning, SFT, increases the likelihood of selected target tokens in demonstrations. A demonstration can teach a tool format, an effective repair sequence or recovery after an error. A successful episode is not automatically a good demonstration: it may include unnecessary steps or an accidental shortcut. Inspect the actions you intend to imitate.
On-policy distillation lets the student generate a continuation and asks a teacher for guidance at the prefixes the student actually reached. That guidance requires an explicit distribution-matching recipe and compatible token events. It is useful when a strong teacher can guide relevant states and the scoring cost fits the training budget. The teacher can still be wrong.
Reinforcement learning, RL, uses rewards to change the probability of sampled behavior. Group Relative Policy Optimization, GRPO, compares rewards among responses to the same task without a separately learned value baseline. A reliable terminal reward can evaluate a whole repair even when writing the ideal sequence of edits is difficult. Sparse success, weak tests and service failures can make that signal unhelpful.
| Evidence | First experiment and check |
|---|---|
| Reviewed examples | Try SFT with an explicit target mask. Confirm the intended assistant targets survive preprocessing. |
| Teacher guidance | Try on-policy distillation. Align token events and evaluate held-out task quality. |
| Task verifier | Try outcome-based RL. Verify reward meaning and handling of unscorable attempts. |
| Vague complaint | Improve evaluation first. Reproduce and categorize the failure. |
These methods can be combined. They are not a mandatory ladder. A strong initial policy may learn from outcomes directly; another may need examples before it produces enough valid actions to explore.
Compare the same population
Use a constructed baseline of 80 scheduled tasks: 36 verified successes, 36 task failures and eight environment failures. The system succeeds on 36/80 = 45% of scheduled tasks. Among the 72 scored tasks, success is 36/72 = 50%. Both numbers are useful when their denominators are visible.
Suppose a proposed harness fix removes all eight environment failures, and four of those repaired attempts succeed. End-to-end success becomes 40/80 = 50% without changing the policy. This is a hypothetical calculation, not evidence that infrastructure repair always produces that gain. It shows why an initial policy-training proposal needs a functioning baseline.
Split training and evaluation by the source of related tasks, such as repository or task family, before generating demonstrations. Near-identical repairs on both sides of the split can make memorization look like generalization. Keep a final holdout apart from the set repeatedly used to choose prompts and checkpoints.
Do this before moving on
Write a proposal containing the failing behavior, the evidence that isolates it to policy decisions, the supervision available and one release requirement. For the repository agent, an acceptable draft might require valid tool calls, increased held-out repair success and no regression on protected task families.
Then change one assumption: the verifier can be edited by the agent. Your next action should be to move evaluation outside the writable workspace and check its integrity. Running more optimization against that verifier would give a misleading experiment.
Go deeper
-
Pretraining, Fine-Tuning and Distillation separates the objective from adapters and numerical storage; Premium reference.
-
RL Post-Training Infrastructure explains the state and compute roles inside an online training cycle; Premium reference.
-
Training at Scale supplies the memory and communication background for the learner pool.
-
AgentGym reinforcement-learning guide develops the environment contract in a longer Premium implementation guide.
-
Tokens, Embeddings and Logits connects the next-token policy definition to the tensors a model produces.
-
Structured Outputs and Constrained Decoding explains the format controls worth evaluating before training on malformed calls.
-
Model Onboarding helps pin the model, template and runtime needed for a reproducible baseline.
Key takeaways
- Preserve a complete attempt before deciding which component needs improvement.
- Demonstrations, teacher distributions and rewards provide different supervision.
- In the example, 45% scheduled-task success and 50% scored-task success describe the same baseline.
- Fix an invalid verifier before using its output as a training objective.
Check yourself
Answer before you look. Recalling it is what makes it stick; recognising it does not.
1The edit is correct, but the environment cannot find its test runner. What should change first?
2What is the difference between 36/80 and 36/72 in the baseline?
3Must SFT always precede outcome-based RL?
Sign in to track which lessons you have finished.