distillation

The transfer of capability from a larger teacher model to a smaller student. By 2026 it isn’t a secondary deployment trick — it’s a structural step in the training pipeline and one of the core ways frontier models earn their value: they generate the training data the next generation of smaller, deployable models will see. 2026-04-27-llm-training-principles-paths-practices‘s formulation: models must get bigger before they can get smaller.

The standard pattern

1. Train a large teacher model — often MoE for training-side compute efficiency, or a dense reasoning model with heavy RL post-training.
2. Use the teacher to generate trajectories — for reasoning models, this means full chain-of-thought + answer; for instruction-tuned models, it means high-quality response examples.
3. Train a smaller (typically dense) student on those trajectories, often with SFT followed by a lighter RL pass.

The student doesn’t need to discover the reasoning; it learns from the teacher’s worked-out version.

Why bigger first, smaller later

The article offers a working hypothesis the wiki tracks but doesn’t claim is settled:

• In raw internet text, knowledge memorization and reasoning ability are coupled. The pretraining objective (next-token prediction on natural text) forces the model to do both at once.
• A model has to be big enough to carry both before it has reliable reasoning capacity. Small models trained on raw text can’t easily decouple the two.
• A teacher of sufficient size can then generate pure-reasoning supervision — trajectories where the reasoning is explicit and the knowledge load is minimized. A small student trained on this synthetic data can specialize in the reasoning half without having to memorize the encyclopedic half.

So “first scale up, then scale down” isn’t only a cost story (cheaper inference). It’s a structural one: the small deployable model is only achievable via the large teacher’s data generation. (data-engineering)

Public reference points

• DeepSeek-R1-Distill series — the canonical 2026 public example. RL trajectories from R1 train dense models from 1.5B to 70B. The article frames this as evidence that capability transfer at this scale is real, not a benchmark artifact.
• Llama 3.1 405B — explicitly used to improve the post-training quality of Llama 3.1 8B and 70B. The frontier model exists partly to manufacture training data for the smaller members of the family.
• TranslateGemma — a specialization rather than pure size compression — narrow target task, dedicated reward design, smaller deployable footprint. Cited as evidence the same toolkit handles “make it smaller” and “make it specialized”.

The industry diffusion staircase

The article’s Figure 12 names a staircase across model generations:

The point: frontier model value isn’t only its own inference — it’s also the training data source for the rest of the industry. Each generation manufactures the supervision for the next. The ”↓ Parameters” reversal between Step 3 and Step 4 is where distillation is doing the work.

Why dense for the distilled targets

The R1-Distill series targets dense models, not MoE — see mixture-of-experts for the trade. Briefly: dense is the deployment-friendly form (simpler serving, lower memory floor, no routing complexity), and the goal of distillation is deployability. MoE makes sense for training-side compute efficiency at the teacher; dense makes sense for serving-side simplicity at the student.

Release is a checkpoint choice

The article’s closing point on this layer: the released model isn’t necessarily the rightmost training checkpoint. Vendors compare multiple checkpoints on real-task results, refusal style, tool stability, cost, and regression risk before picking. The released version is a product decision over a continuously running training pipeline.

Implication for reading vendor announcements: “model X.5 is better than X” depends on what axis. Cheaper / faster / more reliable on a different task / more specialized — these are all “better” in a release context, not necessarily a higher peak on a capability curve.

Cross-link

• data-engineering — synthetic data via teacher trajectories is the data-engineering manifestation of distillation.
• post-training — distillation targets typically need their own SFT + light RL pass; not just trajectory imitation.
• reasoning-models — the canonical teachers. Reasoning RL produces the trajectories that distillation transfers.
• mixture-of-experts — explains the dense-target choice for distilled students.
• deepseek — most of the public reference points come from there.