post-training
Everything that happens to a model after pretraining and before deployment: instruction tuning, preference optimization, reasoning RL, alignment. By 2026 this is the layer where most of the perceived capability gap between modern models lives. 2026-04-27-llm-training-principles-paths-practices‘s reference point: a 1.3B InstructGPT model, post-trained with alignment + preference optimization, beat a 175B GPT-3 on human preference — two orders of magnitude in parameter count, the smaller model won.
The four routes
Modern post-training is rarely a single technique; the landscape is four overlapping methods:
| Method | What it does | Trade |
|---|---|---|
| SFT (supervised fine-tuning) | Train on (instruction, response) pairs. Teaches form: how to answer, how to format, how to be an assistant. | Cheap, well-understood; teaches style as much as content. |
| RLHF (reinforcement learning from human feedback) | First imitate high-quality responses (SFT), then use a reward model trained on human preference comparisons to do RL. | The original recipe; expensive, hard to keep stable, vulnerable to reward overfitting. |
| DPO (direct preference optimization) | Skip the separate reward model — learn directly from preference pairs against a reference policy. | Simpler stack, no separate reward model, often comparable to RLHF on common tasks. |
| RFT (reinforcement fine-tuning) | The productized interface ([[openai | OpenAI]]‘s framing): user supplies task, grader, reward signal. The training infrastructure is hidden. |
These aren’t strict rivals; production recipes often chain them.
SFT teaches style, not just knowledge
A subtle but consequential point in the article: SFT data length, formatting choices, citation style, list-vs-prose preferences propagate directly into model output shape. Combined with the empirical bias of preference evaluations toward longer, well-structured responses, this means leaderboard differences between models are often style differences, not capability differences.
Practical implication: leaderboard wins from “improved post-training” sometimes mean “post-trained to produce the kind of output the eval rewards”. Real-task results, cost, and stability are the calibration set.
The DeepSeek-R1 four-stage recipe (the public reference)
DeepSeek-R1‘s technical report is the cleanest public account of a modern post-training pipeline, and the article walks through all four stages:
| Stage | What | Why |
|---|---|---|
| 1. Cold-start SFT | Small high-quality CoT dataset, light supervised fine-tune of the base model | DeepSeek-R1-Zero showed pure RL on a base model is feasible but produces repetition, language mixing, unreadable output. The cold start gives RL a stable starting point. |
| **2. Reasoning RL with [[grpo | GRPO]]** | RL on math / code / logic with verifiable correctness as the reward |
| 3. Rejection-sampling fine-tune | Filter successful trajectories from stage 2 → new SFT dataset → another supervised pass | Bridges RL → SFT: the good RL trajectories become high-quality training examples for the next round. |
| 4. Alignment RL | Helpfulness + safety preference reward, RL pass | Final shaping into a release-ready assistant. |
Stages 3 and 4 iterate. The article’s point: the gap between “just SFT” and the full four-stage recipe is visible in published results, not a theoretical improvement.
The cold-start stage is the article’s specific anti-pattern flag — R1-Zero proved cold start isn’t strictly required for RL to work, but skipping it gives you a model that’s strong on benchmarks and unusable for users.
Why RLHF is being supplemented by verified rewards
Within RL post-training, the field is shifting toward verified rewards — programmatic correctness checks (math, code, logic) instead of (or alongside) human preference labels. The benefits: cheaper, more consistent, less prone to label drift. The catch: works only where verification is automatable. Open-ended tasks (writing, advice, design) still depend on preference signals, with all the reward-hacking risks that entails.
Where this layer fits
- Upstream — pretraining decides what the model can learn to do; post-training decides what it will do.
- Adjacent — eval-grader-reward design is post-training, in the sense that the reward / grader / eval triple is the actual training target, not a separate evaluation step.
- Downstream — reasoning-models (o1, R1) emerged from pushing reasoning RL hard within post-training. Agent training is the further generalization: post-training in environments rather than on static tasks.
- Sideways — constitutional-ai and deliberative-alignment are alignment-specific post-training techniques that fold safety into the training target rather than tacking it on at the end.
Diagnostic implication
If a model “got better” between releases, post-training is the most common single source — instruction following, refusal behavior, tool-use shape, response style. Pretraining changes are slower, more expensive, and harder to attribute. Per llm-training-pipeline‘s closing checklist: when you can’t explain a release-to-release improvement by pretraining, look at post-training first.
Post-training as “RL with reasoning in the action space” (Yao 2025)
Shunyu Yao’s The Second Half reframes the recent post-training wave from a different angle. The author’s RL-side decomposition is algorithm + environment + priors. By 2024, two pieces had quietly fallen into place: language pretraining provided priors (pretraining), and adding reasoning to the action space (react) provided a way to use them generatively. Once those two were locked, the RL algorithm became the easy part — that’s why the o-series, R1, deep research, and computer-using agents arrived close together.
Read this way, the four-stage DeepSeek-R1 recipe above is one instantiation of the broader pattern; the GRPO / cold-start / verifiable-rewards details matter, but the load-bearing move was earlier — adding reasoning as an action so RL on top of language priors could generalize. The author’s own pre-2022 work on text-game agents (CALM, the first agent built via pre-trained LMs) needed millions of RL steps per game and didn’t transfer; the same model class with reasoning-as-action transfers across SWE / math / web / computer-use without retargeting.
Practical consequence for diagnosing post-training improvements: ask what’s changed at the action-space layer, not only at the algorithm-and-data layer.
- Did the model learn to allocate inference compute over reasoning tokens differently? → reasoning-models axis.
- Did the action space gain new tools or environments? → harness axis.
- Did the cold-start / preference / verified-reward stages change? → the four-stage recipe axis above.
The three are coupled (the four-stage recipe is how the reasoning policy gets trained), but they fail differently and want different diagnostic moves.