data-engineering

The full data-production stack that sits between raw text and pretraining tokens, and increasingly the place where modern training recipes differentiate. 2026-04-27-llm-training-principles-paths-practices‘s framing: data engineering looks like cleaning, but is actually capability design — what the model sees, in what proportion, decides the shape of its capabilities, not just how much it knows.

The funnel

The article’s pipeline view:

[Raw crawl, Code repos, Books, Forums, Docs, Synthetic data]
        ↓
Text extraction
        ↓
Language ID
        ↓
Quality filter
        ↓
PII redaction
        ↓
Safety filter
        ↓
Deduplication
        ↓
Mixture design
        ↓
Training shards

Every stage drops something — the discarded set defines the model as much as the kept set does.

What data engineering actually controls

Three layers of decisions, ordered by how much they affect the final model:

1. Mixture composition. Web text vs. code vs. books vs. forums vs. domain-specific corpora. Code-heavy mixes produce code-strong models; the ratio is doing real work, not just topping up volume.
2. Quality filtering. Document-level + line-level quality gates. The same source, filtered differently, produces meaningfully different downstream models.
3. Deduplication and contamination control. Often underweighted but high-impact:
   • Document-level dedup avoids reabsorbing the same content via mirrors, syndicated articles, repeated boilerplate.
   • Line-level dedup avoids high-frequency templates (license headers, navigation menus) crowding out signal.
   • Benchmark-leakage filtering — keeping the eval set out of the training set — has become a serious operational concern.

The article’s diagnostic: a lot of what looks like “uneven quality” in open-weights models traces back to data-pipeline differences, not architecture or compute differences.

Data Mixing Laws

Mixture ratios are now their own research topic — Data Mixing Laws and similar work treat the mixture as a design variable, asking what proportions of code / math / natural language produce what shape of downstream capability. The framing has moved from “how much can we collect?” to “given a budget, what mix maximizes the capability profile we want?”

This is the data-side analogue of chinchilla-scaling: a quantitative tool for budget allocation, not just intuition.

Synthetic data is no longer auxiliary

By 2026, synthetic data is a first-class part of pretraining and post-training, not a fallback for when human data runs out. The article cites several public examples:

• Self-Instruct — earliest pattern: model generates its own instruction-following training data.
• DeepSeek-R1 distillation traces (deepseek) — RL-trained reasoning trajectories become the training set for smaller dense models. (distillation)
• Qwen and Kimi — rising synthetic-supervision share visible in public training reports.

The structural pattern: each generation of stronger models reconstructs the training data the next generation sees. Early models generate basic instruction data; stronger ones produce high-quality reasoning traces and CoT; RL-trained reasoning models produce distillation-grade trajectories for smaller dense models.

Why this works isn’t fully settled, but the article offers a working hypothesis: in raw internet text, knowledge memorization and reasoning are coupled — the pretraining objective forces the model to learn both at once. A model has to be big enough to carry both before it can be used to generate pure-reasoning supervision that lets smaller models specialize on the reasoning half. “Models must get bigger before they can get smaller” is structural, not just a cost story. (Cross-link: distillation.)

How decisions here propagate

A bad data decision in this layer is hard to fix later:

• Tokenizer mismatch with the data mixture (pretraining) — fragmenting Chinese / code / math costs every inference, forever.
• Benchmark contamination — destroys the integrity of every eval downstream.
• Mixture imbalance — produces capability gaps that more compute alone will not close.
• Repeated low-quality content via inadequate dedup — the model learns templates instead of patterns.

Data engineering and pretraining are tightly coupled — most of the capability distribution the model will ever have is decided here, before any post-training or harness work begins.