Post-train an LLM from scratch in PyTorch: SFT, DPO, PPO and GRPO
Implement the full LLM post-training pipeline in pure PyTorch on a CPU. Pretrain a tiny Transformer, then compare SFT, a reward model, DPO, PPO and GRPO with a verifiable reward.
How do you implement SFT, a reward model, DPO, PPO and GRPO from scratch in PyTorch?
Start from a pretrained causal Transformer. SFT minimizes cross-entropy on answer tokens only. A reward model adds a scalar head trained with the Bradley-Terry loss, -log sigmoid(score_chosen - score_rejected). DPO trains on the same pairs against a frozen reference with -log sigmoid(beta * (policy log-ratio of chosen - that of rejected)), so it needs no reward model or sampling. PPO samples answers, scores them with the reward model minus a KL penalty, estimates advantages with a critic and GAE, and applies the clipped ratio objective. GRPO drops the critic: it samples a group of answers per prompt, checks them with a program and normalizes rewards within the group. This project runs all of them on a CPU in about 15 seconds.
Six checkpoints with lineage, per-stage metrics and a five-seed comparison of SFT, DPO, PPO and GRPO.
Environment
CPU · Python 3.12 · PyTorch 2.6.0 · no GPU required
Plan your session
60–90 minutes of guided work; every stage runs in about 15 seconds. A planning estimate, not measured runtime.
Before you begin
Run Python scripts in a terminal and read basic PyTorch tensor code. Know what next-token prediction and a training loop are; the free tiny language model guide covers both.
CPU workflow executed across five seeds. All six stages, the five-seed comparison and 12 unit tests executed with PyTorch 2.6.0+cpu on 21 October 2026; 13 of 13 deliberate source mutations failed the tests. Results describe a 134,977-parameter model on synthetic single-digit addition. They do not rank the methods for real models or natural language.
01Base model
Pretraining on plain a+b=c text stores facts, but the model cannot answer in Q/A format.
↓
02Supervised format
SFT on 40 examples, with loss on answer bytes only, teaches the format and surfaces facts it was never shown in that format.
↓
03Preference or reward
DPO learns from answer pairs; PPO from a learned reward model; GRPO from a program that checks each sampled answer.
Follow the changing state through the project. Each boundary has a check in the walkthrough.
Continue with Premium
Premium includes this guide’s complete walkthrough and downloadable project files. The technical overview, prerequisites and verification notes above are public.
