2 pages tagged

#pretraining

chinchilla-scaling

Compute-optimal data/params rule; models over-train past it; FLOPs, not params, predict quality.

Floor not ceiling; tokenizer/context/multimodal commitments lock in here; pretraining = RL priors.