llm-training-pipeline
The end-to-end pipeline that turns raw text into a deployed model. 2026-04-27-llm-training-principles-paths-practices‘s organizing frame: by 2026 the gap users perceive between models is increasingly not about pretraining scale — it’s the back half of this pipeline that decides what a model feels like in production.
The six-layer view
| Layer | What it actually optimizes | What users perceive |
|---|---|---|
| **[[pretraining | Pretraining]]** | Knowledge coverage, representation quality, scaling efficiency |
| **[[data-engineering | Data engineering]]** | Distribution, quality, dedup, synthetic supervision |
| System & architecture | Throughput, memory, context length, active params, cost | ”Why does this support 128K context” / “runs on one GPU” |
| **[[post-training | Post-training]]** | Instruction following, style, refusal, tool use |
| **[[eval-grader-reward | Eval & reward design]]** | What “good”, “safe”, “robust” mean as training signal |
| **[[distillation | Distillation]] & deployment** | Latency, cost, specialization, online improvement |
Layers are coupled — a change in one almost always propagates. The article’s stronger claim: the value of a frontier model now sits disproportionately in the bottom four layers, not the top one.
The nine-stage view
The same pipeline, expanded to surface the substages that matter as engineering decisions:
- Raw data
- Data engineering — see data-engineering
- System recipe — parallel strategy, MoE / dense choice, mixed-precision (FP8 etc.),
muP, WSD learning rate, batch size, context length budget - Pretraining — see pretraining
- Post-training — see post-training
- Eval / reward design — see eval-grader-reward
- Agent harness — see harness for the runtime view; meta-harness for the outer-loop optimizer; and, since the Jul 2026 Weng survey, the recursive-self-improvement hierarchy above both (self-improving-harness systems make this stage self-modifying)
- Distillation / specialization — see distillation
- Deployment
Two feedback loops cross the whole pipeline:
- Production traffic → data engineering. Real usage seeds the next data mix. Cursor’s real-time RL (cursor) is one published example.
- Offline benchmark results → pretraining. Eval signal informs the next pretraining decision (data ratios, tokenizer, architecture).
These loops are why the pipeline is “the product” — the released model is a snapshot (distillation §“Release is a checkpoint choice”), the loops are what’s running.
Why the back half matters now
InstructGPT’s 1.3B beating 175B GPT-3 on human-preference evaluation is the canonical historical proof that post-training can reorder capability ranks across two orders of magnitude in parameter count. By 2026 the same dynamic shows up across more layers:
- Tokenizer + context-window choices made before pretraining run with every inference.
- Data mixture decisions shape capability distribution (code vs. math vs. encyclopedic) in ways more compute cannot fix later.
- Post-training pipelines (e.g. DeepSeek-R1’s 4-stage recipe) determine instruction quality, reasoning shape, refusal behavior.
- Reward and grader design decide what “good” means at all — see eval-grader-reward and reward-hacking.
- Harness alone produces 6× spreads on the same base model in the Meta-Harness paper — and the harness stage is now itself an optimization loop, not a hand-written artifact (recursive-self-improvement).
- The release version is rarely the rightmost training checkpoint — it’s a product decision balancing capability ceiling vs. cost, latency, regression risk.
The “why did this model suddenly get stronger?” diagnostic
2026-04-27-llm-training-principles-paths-practices‘s closing prescription, rephrased as a checklist:
- Layer location — is the change in pretraining, or after it? Style / instruction-following / tool-use shifts almost never come from “more pretraining text”.
- Component — weights & training recipe, vs. reward / eval / grader, vs. harness code, vs. deployment loop. Recent “model got better” reports often resolve to one of the latter three.
- Release intent — is the new checkpoint pursuing a higher capability ceiling, lower cost / latency, or specialization for a use case? “Better than v1” depends on which axis.
The point: model name → training curve is a fiction. Model name → snapshot from a specific checkpoint of a specific layer of a continuously-running pipeline is closer to reality.