grpo

GRPO (Group Relative Policy Optimization) — the RL algorithm introduced in DeepSeekMath (Shao et al. 2024) and used as the reasoning-RL stage of DeepSeek-R1‘s post-training. The reason it shows up everywhere in 2026 isn’t theoretical superiority — it’s that it removes one whole component of PPO’s stack, which matters at LLM scale.

What it changes vs. PPO

PPO (Proximal Policy Optimization) is the workhorse of pre-2024 RLHF. It needs:

• A policy network — the model being trained.
• A separate value network — estimates expected return from a given state. The “critic” in actor-critic.

At LLM scale, maintaining a value network nearly the size of the policy is an expensive engineering and memory problem.

GRPO eliminates the value network entirely. Instead:

1. Sample multiple responses (a group) from the same prompt.
2. Score each response with the reward signal.
3. Use the group’s internal ranking — each response’s reward minus the group mean, normalized by std — as the advantage estimate.

That replaces “what’s the expected value of this state?” (estimated by a learned critic) with “is this response above or below average for this prompt?” (computed from the group). No critic network needed.

Why this matters at scale

The article’s framing is engineering-first, not theoretical:

• One model instead of two. Policy training only — drops a whole copy of the model from memory.
• No critic to train, debug, or tune. Value-network training is its own stability problem; GRPO sidesteps it.
• Compatible with verifiable-reward signals. When the reward comes from a programmatic check (math correctness, code passes tests), the group-relative formulation is a natural fit — sample several responses, check correctness, rank within the group.

Concrete examples cited: DeepSeek-R1, Cursor’s Composer 2 RL infrastructure. The article notes both converged on “GRPO-class” algorithms — same direction, possibly with implementation variations. (2026-04-27-llm-training-principles-paths-practices)

Where it fits

GRPO sits at stage 2 of the post-training pipeline (reasoning RL with verifiable rewards, in the DeepSeek-R1 framing). It doesn’t replace SFT, DPO, or RLHF outright — they answer different questions:

• SFT — teach behavior from labeled examples.
• DPO — optimize against preference pairs without a separate reward model.
• RLHF (PPO) — RL with a learned reward model + value network.
• GRPO — RL with an explicit reward (often verifiable) and group-relative advantage, no value network.

The trend the article highlights: as more of post-training moves toward verifiable rewards (math, code, logic), GRPO becomes the natural default for that subset. RLHF/DPO remain relevant where preference signals dominate.

Limits

GRPO’s group-relative advantage works best when:

• The reward is well-defined enough that comparisons within a group are meaningful.
• Sampling multiple responses per prompt is affordable (group size matters).
• Reward variance is high enough that group ranking is informative.

If responses cluster too tightly or rewards are noisy at the group level, the group-relative signal degrades. These are practical constraints rather than principled limits — but they explain why the algorithm pairs well with verified rewards (math correctness, unit-test pass) and less well with subtle preference signals.