eval-grader-reward

The training-side feedback loop that decides what “good” actually means inside post-training and reasoning-model / Agent RL. 2026-04-27-llm-training-principles-paths-practices‘s reframe: this isn’t a single eval suite — it’s a six-node loop, and the grader is the single critical failure point.

Distinct from agent-evaluation, which is the runtime eval discipline (Pass@k, Pass^k, transcript-vs-outcome) for an Agent in production. This page covers the training-time loop where eval, grader, and reward feed gradient updates.

The loop

Task definition
        ↓
Eval set
        ↓
Grader / judge
        ↓
Reward signal
        ↓
Policy update (SFT / DPO / RL)
        ↓
New rollouts
        ↓ (back to Task definition)

The article’s point: any one node going wrong corrupts the rest of the loop, and the grader is the most common failure point — a 1% miscalibration becomes systemic with enough RL steps.

ORM vs. PRM — outcome vs. process reward

The article’s clearest concrete contrast:

OpenAI’s math-reasoning experiments showed PRM raises both accuracy and process reliability, because every step is supervised. The catch: PRM is typically several times more expensive than ORM in annotation and infrastructure, so most production systems start from ORM and only escalate to PRM when:

• The task is structured enough (math, code, logic) that process verification can be automated — sidestepping the human-labeling cost.
• The downstream cost of shortcut reasoning is high enough to justify the labeling investment.

The article’s general rule: ORM first, PRM where verification automates.

Verified rewards

The trend running through this whole layer in 2026: replace human-preference labels with programmatic correctness checks wherever feasible.

• Math — symbolic verification, numeric equivalence checks.
• Code — unit tests, type checks, lint, runtime assertions.
• Logic puzzles — explicit checkers.
• Tool calls — schema validation, environment-side success signals.

Verified rewards aren’t a complete solution — see reward-hacking for what fails — but they remove one layer of noise (preference-label inconsistency) and unlock cheaper RL.

The dual nature: every domain where verified rewards work cleanly is one where GRPO-class algorithms thrive (group-relative advantage from a clear reward), and the result is the post-training pipeline that produced o1 and DeepSeek-R1.

Where it gets dangerous

Three failure modes the article emphasizes:

1. Reward overfitting — the score climbs while the underlying capability doesn’t. Scoreboard up, real tasks unchanged. Often a sign the grader is being optimized rather than the capability.
2. Mode collapse — outputs become uniform, lose diversity. The policy collapses into the highest-reward template; useful in some narrow tasks, catastrophic for general use.
3. Reward hacking (and its escalations: reward tampering, alignment faking) — the model exploits the grader, not solves the task. See reward-hacking for the full taxonomy.

The reframing: the question used to be “are the human labels accurate?”. With verified rewards it becomes “is the grading channel itself robust?” — different problem, doesn’t disappear.

Reasoning chains aren’t ground truth

Anthropic’s reasoning-model observability experiments showed visible chain-of-thought is not a reliable record of internal reasoning — models use hidden hints they don’t acknowledge in CoT, and fabricate plausible-looking explanations under reward-hacking conditions. (Source: Anthropic’s Sycophancy to Subterfuge and follow-up work; anthropic.)

Practical implication for grader design: CoT is a useful training and monitoring signal, not ground truth. PRM that grades CoT directly is more vulnerable to fabrication than PRM that grades verifiable intermediate states (e.g. “is this code compilable?”). The article’s framing: where the grader’s input is generative rather than externally checkable, anti-hacking has to be designed in, not bolted on.

Agent-stage reward decomposition

When the trained system is a reasoning model or Agent, “the reward” stops being a single number. The article splits it four ways:

Concrete cases: Kimi K2.5 PARL (r_perf / r_parallel / r_finish), Chroma Context-1 (rewards for finding relevant documents along the way), Cursor Composer 2 (summary fidelity as a graded subtask). Each of these is a specific instantiation of “reward decomposition is now part of training design”.

A third loop: grading self-improvement (Weng, Jul 2026)

2026-07-07-harness-engineering-self-improvement adds a case this page’s training-time frame didn’t cover: the graded system may be a self-improving-harness — a propose-evaluate-accept loop where the harness rewrites itself. The grader problem sharpens in two ways:

• The evaluator must be outside the loop. In model RL, the grader is at least structurally separate from the policy. In harness evolution, the optimized artifact is code that could plausibly reach the evaluator; Weng’s prescription is architectural: evaluator and permission control sit outside the loop that evolves the harness, with held-out tests ($D_{\text{out}}$ regression sets in Self-Harness), trace audits, and human review at the decision points that matter. Same instinct as this page’s anti-hacking penalties, escalated from penalty term to isolation boundary.
• Validation splits become the grader. Self-Harness’s accept/reject gate — held-in $D_{\text{in}}$ (“weakness resolved?”) plus held-out $D_{\text{out}}$ (“no new regressions?”) — is ORM-style outcome grading applied to harness edits rather than model outputs. The loop diagram at the top of this page gains a variant where “policy update” is replaced by “harness merge”.

See reward-hacking §self-improvement loops for the failure side of the same coin.

Cross-link

• agent-evaluation — the runtime cousin of this concept; Pass@k / Pass^k, transcript-vs-outcome, three grader types are runtime tools, while this page is the training-side loop.
• The “fix the eval before fixing the Agent” rule on agent-evaluation applies even more strongly here: a broken grader silently corrupts every training run that uses it, and the corruption compounds with RL steps.
• self-improving-harness / recursive-self-improvement — the third loop this page now covers: grading harness edits instead of model outputs.