pretraining

The first stage of the llm-training-pipeline: predicting next tokens at scale on a curated text mixture, until the model has compressed enough language distribution and world knowledge to be useful as a base for everything downstream. By 2026 it is necessary but no longer the differentiator — most of the perceived capability gap between modern models is from the back half of the pipeline (post-training, eval-grader-reward, harness, distillation).

What it actually does

Token prediction is the form; the function is more layered:

• Learn the language distribution (syntax, semantic structure, register).
• Compress the knowledge and patterns visible in the training corpus into parameters.
• Lay down representations that can be activated by post-training for instruction-following, reasoning, tool use — none of which appear automatically.

Token prediction by itself doesn’t explain why scaling causes new capabilities to emerge. The article is careful: capability shape is decided over the whole pipeline, not by pretraining alone.

What gets locked in here

Pretraining isn’t only about “how much knowledge”. Several decisions made before training starts ship with every inference forever:

• Tokenizer & vocabulary. Llama 2’s 32K vocabulary → Llama 3’s 128K compressed sequence length by ~15% with downstream performance gains. A tokenizer that fragments Chinese / code / math poorly pays a tax on every forward pass; the cost is permanent. (2026-04-27-llm-training-principles-paths-practices)
• Context window. Long context (e.g. 128K) changes attention cost, batch size, training curriculum, and parallel strategy. It’s a pretraining-architectural commitment, not a knob flipped at release.
• Multimodality. Whether vision / audio / etc. enter at pretraining or are bolted on post-hoc affects data mixing, encoder design, safety eval surface — listed by Gemma 3 as one of its explicit pretraining commitments alongside single accelerator and 128K context.
• Single-accelerator runnability. If “must run on one GPU” is a release goal, parameter count, quantization path, and model-family branching all get constrained from the start.

User-visible features (“can run locally”, “can read images”, “handles long documents”) are mostly pretraining-time decisions, not deployment add-ons.

Compute, params, tokens — see chinchilla-scaling

Pretraining is governed by a budget triangle: parameters, training tokens, total compute (FLOPs). The historical question “more parameters or more data?” is now answered by chinchilla-scaling: if you have a fixed compute budget, there’s a data-optimal point; modern models routinely over-train past it (Llama 3 8B at ~75× Chinchilla-optimal) trading more compute for higher capability density per parameter. Total training FLOPs predicts quality better than parameter count does.

Stability is itself a capability

Training a frontier-scale model isn’t only a deep-learning problem — it’s a distributed-systems problem. 2026-04-27-llm-training-principles-paths-practices highlights several failure modes that almost never make it into research papers but determine whether a training run completes:

• Loss spikes large enough to require rolling back days of work.
• Silent GPU faults producing wrong gradients without crashing.
• NVLink jitter / inter-node communication anomalies poisoning a few steps at a time.

DeepSeek-V3’s report (deepseek) is cited as the unusual public reference for “training that ran clean”: 14.8T tokens, ~2.788M H800 GPU hours, no irrecoverable loss spikes, no rollbacks, and FP8 mixed-precision verified at frontier scale. The implicit point: the ability to run a frontier-scale pretraining cleanly is itself a research-lab capability not equally distributed.

Recent recipe-level work

Public training reports increasingly include details that were research-internal a year ago:

• muP — hyperparameter transfer from small-scale experiments to frontier scale.
• WSD learning rate — Warmup / Stable / Decay schedule.
• Batch size + data-to-parameter ratio as deliberate choices, not defaults.
• Architectural variants like Forgetting Transformer or Kimi’s Attention Residuals — work on making longer contexts trainable without information dilution. (moonshot-ai)

The article’s framing: these used to be the kinds of details that distinguished labs; now they’re table stakes appearing in every modern training report.

Pretraining is the floor, not the ceiling

The recurring point: pretraining decides what the model could learn to do later, but whether it follows instructions, refuses appropriately, uses tools well, reasons reliably — none of that comes from more pretraining text. Those properties live in post-training and downstream stages.

If you fix pretraining and improve downstream layers, capability ranks reorder (InstructGPT 1.3B > GPT-3 175B). If you scale pretraining and ignore downstream layers, you get a more knowledgeable but less usable model.

Pretraining as RL priors (Yao 2025)

Shunyu Yao’s The Second Half gives a complementary framing from the RL side: pretraining is the priors half of RL’s three-component decomposition (algorithm + environment + priors). Yao’s stronger claim is that for decades RL focused on algorithms, then on environments, and no one paid attention to priors — and that’s what most of the recipe’s leverage actually came from. Per the source: “The most important part of RL might not even be the RL algorithm or environment, but the priors, which can be obtained in a way totally unrelated from RL.”

Concrete implications this view sharpens:

• The transfer gap was a priors gap. OpenAI‘s pre-2022 environment-first plan (gym → World of Bits → Universe → Dota → robotic Rubik’s Cube) solved individual environments but didn’t transfer across domains. Yao’s diagnosis: it was missing language priors. Once GPT-2 / GPT-3 existed, fine-tuning on top of language priors produced WebGPT and ChatGPT — the same RL machinery now generalized.
• Reasoning is the action that lets priors bind. Pretraining priors are useless if the action space is only environment-grounded (react). Adding reasoning to the action space gives the priors something to do per turn, which is why the LLM-agent recipe works without exotic algorithms.
• Tokenizer / context / multimodal commitments are priors-shaping decisions, not just architectural ones. The Llama 3 tokenizer expansion (32K → 128K) doesn’t just compress sequences — it changes which token-level structures the priors can represent. Same for multimodal pretraining: adding vision changes the priors’ coverage, not just the input pipeline.

Yao’s framing doesn’t contradict the back-half-of-the-pipeline thesis from 2026-04-27-llm-training-principles-paths-practices; the two stack. Both are true: the priors decided the recipe was possible at all, and the back-half stages decide perceived capability between modern models.