mixture-of-experts
MoE (Mixture of Experts) — the architectural pattern where a model has many “expert” subnetworks but only a small subset is activated for any given token. Lets total parameters scale up while keeping per-token compute roughly fixed. The canonical 2026 trade-off in the systems & architecture layer; not a free win.
What it actually does
Conceptually: replace each FFN layer with N parallel expert FFNs plus a router. For each token, the router picks the top k experts (typically k=1 or k=2 out of dozens or hundreds). The token is processed only by those experts; the rest are inactive for that token.
Result: total parameters grow with N, but compute per token grows with k, not N. A 600B-parameter MoE with k=2 activates roughly the compute of a 30B-ish dense model.
The contrast with dense models — where every parameter participates in every forward pass — is the headline trade. (2026-04-27-llm-training-principles-paths-practices)
Why it’s a compromise, not a free lunch
The cost structure shifts rather than shrinks:
- Routing complexity — the router itself is a learned component; bad routing destroys the win.
- Load balancing — uneven expert utilization wastes capacity and creates training instability. Auxiliary losses to balance routing add their own tuning surface.
- Infrastructure weight — distributed serving has to handle expert sharding, all-to-all communication, and KV cache layout that differs from dense models.
- Memory — total parameter count is still the GPU memory footprint, even though compute is small. MoE lets you cheap compute without cheaping memory.
The article’s framing: DeepSeek-V3, Qwen’s MoE variants, and similar designs are all picking specific compromises in this space, not making “MoE > dense” architectural statements.
Public reference points
- DeepSeek-V3 — the canonical 2026 worked example of frontier-scale MoE plus FP8 mixed precision; cited in pretraining for stability claims.
- Qwen series — multiple MoE variants, also cited as deliberate cost / quality compromises.
The article doesn’t claim MoE is universally better — it specifically calls these designs cost / effect compromises, not architectural preferences.
Dense vs. MoE in distillation
When the article distinguishes “dense” from “MoE” in distillation, it matters:
- Dense = all parameters always active. DeepSeek-R1-Distill targets dense models from 1.5B to 70B because dense is the deployment-friendly form — simpler serving, lower memory floor, no routing infrastructure.
- MoE is an upstream choice that helps you train a high-capability teacher cheaply. Distilling its trajectories into smaller dense models gives you the deployment story without the infrastructure tax.
So MoE and dense aren’t competing — they sit at different points in the pipeline: MoE for training-side compute efficiency at frontier scale, dense for serving-side simplicity at deployment scale.
When MoE gets cited as evidence
Two ways the article uses MoE as a reference point:
- As an example of architectural choices made before training that propagate everywhere downstream — you can’t switch a model from MoE to dense after the fact.
- As an example that modern training reports are detailed enough to debate routing strategies — public discussion has moved from “how big is your model?” to “what’s your activation pattern?”, which is the kind of detail the article treats as the new differentiator.