reasoning-models
The model class that emerged in 2024–2025 — OpenAI’s o1, DeepSeek-R1, and their successors — characterized by spending variable inference compute on internal reasoning before producing a final answer. 2026-04-27-llm-training-principles-paths-practices frames this as opening a second scaling axis: training compute remains one axis, inference compute per response is now another.
What changed
Pre-reasoning-model era:
- Inference cost per response was roughly fixed by output length.
- Capability scaled almost entirely with training compute.
- Chain-of-thought was a prompting trick, not a trained capacity.
Reasoning-model era:
- The model is trained to spend more or less inference compute depending on task difficulty.
- A simple question gets a short response; a hard problem gets a long internal reasoning trace before the answer.
- The same training-compute investment produces a different model — one that scales further by spending more at inference time.
The article’s diagram (Figure 9) names four quadrants in the (training compute) × (inference compute) plane:
| Fixed inference | Variable / longer inference | |
|---|---|---|
| Smaller training scale | ”GPT-3 era” — scale training, fixed inference | (rare) |
| Larger training scale | ”Pre-reasoning era” — larger pretraining, fixed output length | Reasoning models — same training scale, variable inference: o1, R1 |
| + environment | Agent era — longer trajectories, more tool calls, much larger inference budget |
The frontier moves diagonally across both axes simultaneously.
How they’re trained
The standard recipe is post-training heavy, with reasoning RL the central stage:
- Cold-start SFT on a small high-quality CoT dataset to anchor format.
- Reasoning RL with GRPO on tasks with verifiable rewards (math, code, logic).
- Rejection-sampling FT to convert good RL trajectories into more SFT data.
- Alignment RL for helpfulness and safety.
DeepSeek-R1‘s technical report is the canonical public reference. The cold-start step is the article’s specific anti-pattern flag: R1-Zero proved cold start isn’t strictly required, but pure RL without it produces repetition, language mixing, and unreadable output.
Why RL teaches inference-budget allocation
The point that most distinguishes reasoning models from prior CoT-prompted models: the RL is teaching the model when to think more vs. less, not just how to think.
- Reward signals from verifiable tasks let the policy learn that some inputs reward a long chain of reasoning and others don’t.
- The model learns a budget allocation policy over inference compute — how many “thinking” tokens to produce before answering.
- This is what makes the second scaling axis useful: you don’t manually configure how much the model thinks; the model has internalized when to spend.
The user-facing dial over this policy is the effort level. 2026-07-12-claude-model-effort-level (anthropic, Jul 2026) confirms the mechanism from the vendor side: the effort level is sent as one more input the model responds to, and per-level behavior — not just thinking length but files read, verification depth, how far to push before checking in — is trained into the frozen weights and reconsidered every turn. Anthropic reports explicitly tuning against “overthinking” during training because it degrades effectiveness. See model-and-effort-selection for the practical selection heuristics.
Reasoning model vs. Agent model
The article distinguishes (Figure 10):
| Reasoning model | Agent model | |
|---|---|---|
| Unit of optimization | A single answer | A trajectory in an environment |
| Loop shape | Prompt → reasoning trace → answer → verifier | Goal → planner → tool call → environment feedback → memory / summary / context editing → next action |
| Main bottleneck | Verifier accuracy | [[meta-harness |
| Typical reward | Outcome reward | Outcome + process + context |
| Common failure | Shortcut reasoning | Tool misuse, context drift, [[reward-hacking |
Reasoning models optimize a thinking budget; Agent models extend that to an acting budget across an environment. The training infrastructure for Agents is correspondingly heavier — see meta-harness for the further generalization.
The Junyang Lin reflection
The article highlights Qwen’s former model lead Junyang Lin’s framing of the open problem in 2026: thinking and instruct modes optimize fundamentally different objectives. One pursues directness, compliance, low latency; the other pursues exploration, accuracy, more compute. A single model trying to switch between modes via a “thinking on/off” knob is not solving one problem cheaper — it’s two different optimization targets.
The article presents this as a problem statement, not a solved question. Worth tracking when more vendor reports land.
Reasoning as a strange RL action (Yao 2025)
Shunyu Yao’s The Second Half gives reasoning models a complementary framing from the RL side. Yao decomposes RL into algorithm + environment + priors, and argues the reasoning-model regime is what happens when you finally get the action space right: reasoning is added as an action, and that’s the bridge between language pretraining priors and generalizable agents. See react for the conceptual page.
Three implications specific to reasoning models:
- Reasoning is a “strange” action — it doesn’t change the environment, the action space is combinatorially infinite (a turn can produce a word, a sentence, or 10K random words with the same external effect), and classical MDP intuition says decision-making should collapse. Empirically it generalizes because the language priors have already “seen” the reasoning space during pretraining. Yao’s compressed claim: “language generalizes through reasoning in agents.”
- The Junyang Lin observability question, restated. The thinking-vs-instruct mode tension that 2026-04-27-llm-training-principles-paths-practices presents as an open problem is, under Yao’s frame, a problem about reward design over a strange action space: the reward shape that produces good thinking and the reward shape that produces good direct compliance pull in different directions, so the same model trying to multiplex them isn’t paying one cost — it’s paying two.
- Why RL teaches inference-budget allocation, restated. From the priors-as-priors perspective: the model isn’t learning to think “from scratch” — it’s learning when to spend the priors it already has. The reasoning trace is a learned policy over an open-ended action space whose value comes from pretraining, not from the RL-time data alone.
This view is consistent with the four-stage DeepSeek-R1 recipe above; it adds a different why on top of the how. The wiki’s react page consolidates the reasoning-as-action framing across the agent and reasoning-model contexts.
Cross-link
- react — reasoning-as-action as a unifying frame; the conceptual ancestor of both prompted CoT agents and trained reasoning models.
- chain-of-thought — the prompted ancestor (Wei et al. 2022, surveyed in 2026-06-04-llm-powered-autonomous-agents); reasoning models are CoT trained into the weights as inference-budget allocation.
- post-training — where reasoning capacity gets installed, via the four-stage pipeline.
- grpo — the reasoning-RL algorithm.
- eval-grader-reward — the verified-reward channels (math, code, logic) that make reasoning RL practical.
- meta-harness — the next generalization: optimize the harness program around the reasoning model, not just the model.
- deliberative-alignment — depends on reasoning models; safety reasoning at inference time is feasible because reasoning at inference time is.
- deepseek, openai — the two labs whose reasoning-model work the article centers.