chinchilla-scaling

The data-optimal training rule from Hoffmann et al. 2022’s Training Compute-Optimal Large Language Models (Chinchilla). For a fixed compute budget, there’s a balance between parameter count and training tokens that minimizes loss; “bigger model” without proportionate “more tokens” leaves capability on the table. By 2026 the rule has been complicated, not invalidated — modern models deliberately over-train past the Chinchilla point to extract higher capability density at a given parameter count.

The data-optimal point

Roughly: for N parameters, train on ~20×N tokens. So an 8B model is data-optimal at ~200B tokens; a 70B at ~1.4T tokens.

This is the compute-optimal point — the choice that minimizes pretraining loss given a fixed FLOP budget. It is not the choice that minimizes inference cost or maximizes capability per parameter.

The “over-training zone”

Modern recipes routinely sit far to the right of the Chinchilla point:

• Llama 3 8B was trained on 15T tokens — about 75× the Chinchilla-optimal point for an 8B model.

The trade is intentional: spend more training compute, get a smaller model with the same (or higher) capability — which means cheaper inference, smaller deployable footprint, easier to fit on consumer hardware. Inference happens billions of times; training happens once. (2026-04-27-llm-training-principles-paths-practices)

The headline number: total FLOPs > parameter count

Once you accept that compute, params, and tokens trade off, total training FLOPs is the single best predictor of model quality, better than parameter count alone. A 70B trained on 200B tokens and a 7B trained on 2T tokens occupy similar quality bands by FLOP count, even though the parameter counts differ 10×.

This is why “the 8B model” or “the 70B model” tells you less than people assume — the same parameter count can mean very different total compute investments and very different capability points.

How to read training-report claims

The article’s framing makes this an everyday diagnostic:

• “Trained at the Chinchilla-optimal point” — compute-efficient, but the model is probably bigger than necessary for a given quality target.
• “Over-trained 10–75× past Chinchilla” — modern dense models with consumer-deployment intent (Llama 3, Qwen 8B, Gemma).
• “Trained well under Chinchilla” — usually under-trained — the model is probably worse than it could be for its size, not better. The article’s diagnostic question: “Is this model actually too small, or is it just under-trained?”

Where it fits in the pipeline

Chinchilla is a planning tool inside pretraining. It does not explain:

• The capability differences from post-training — RLHF, DPO, RFT, and the four-stage R1 recipe operate inside the same FLOP budget but on different data and objectives.
• The capability differences from data-engineering — same FLOPs on different mixtures produce very different capability distributions.
• The serving-cost question — Chinchilla doesn’t penalize inference compute; the over-training trade does.

So Chinchilla still answers the inside-the-pretraining-stage question well. It just stopped being the only question that mattered.

Open questions

• The crossover point between “more pretraining tokens helps” and “diminishing returns from data quality” is empirical and varies per model family. The article doesn’t claim a universal threshold.
• Synthetic-data pretraining (data-engineering) puts pressure on the original Chinchilla regression, since the regression assumed natural-language tokens — the relevant equivalence is unclear when 30%+ of tokens are model-generated.