The Second Half

A short essay by Shunyu Yao (openai; author of CALM 2019 and ReAct 2022) based on talks given at Stanford 224N and Columbia. The thesis: a working “recipe” — language pretraining + scale + reasoning-as-action — has now generalized broadly enough that the recipe will solve any benchmark you can construct, in months. So the field’s bottleneck shifts from training methods to evaluation design. Yao calls the open problem the utility problem — AI saturates exams but the world hasn’t changed much by GDP measure — and argues the second half of AI is about questioning evaluation assumptions the field has taken for laws.

Source file: raw/2026-04-27-the-second-half-of-ai.md.

Image evidence carries weight in this post — three of the four figures are load-bearing for distinct claims, summarized inline below.

Summary

• The first half was a methods game. New training methods / model architectures hillclimbed benchmarks; benchmarks were treated as commodities. The asymmetry shows up in citation counts (image 1: a balance scale tipped toward “Training / Models / Methods” against “Evaluation / Benchmark / Environment”): Transformer ~168K vs. WMT’14 workshop report ~1.3K (~130× ratio); AlexNet ~172K vs. ImageNet ~49K (~3.5×); GPT-3 ~49K vs. SuperGLUE ~2.5K (~20×); Atari deep RL ~17K vs. ALE ~3.8K (~4.5×). The author’s framing: methods were “harder and more exciting than tasks”, and a single great method generalized across many benchmarks, so the impact of methods went well beyond any individual task.
• A working recipe quietly assembled itself. Three ingredients — massive language pretraining (priors), scale (data + compute), and reasoning-and-acting (action-space augmentation) — together solve a wide range of RL tasks using language. A year before this essay, the author argues, no one would have believed a single recipe could tackle software engineering, creative writing, IMO-level math, mouse-and-keyboard manipulation, and long-form QA. By 2025 it does.
• Read through the lens of RL, the recipe is “priors mattered most”. The author breaks RL into three components — algorithm, environment, priors — and walks the historical mis-prioritization:
  • For decades, RL focused on algorithm (REINFORCE → DQN → TD-learning → actor-critic → PPO → TRPO). Sutton & Barto’s textbook is “all about algorithms and almost nothing about environments or priors”.
  • Deep RL made it clear that environment matters: an algorithm is often as specific as the environment it was tested in. OpenAI’s early plan (gym → World of Bits → Universe → Dota → Rubik’s Cube) was “make every digital world an environment, then solve them”. It worked on individual environments but didn’t transfer.
  • Post GPT-2 / GPT-3, the missing piece turned out to be priors. Language pretraining distills general commonsense + language knowledge into models, which can then be fine-tuned into web (WebGPT) or chat (ChatGPT) agents. The most important RL ingredient came from a path totally unrelated to RL.
• Reasoning is a strange action — and it’s the bridge that makes priors generalize. The author’s pre-2022 work on text-game agents (CALM, the first agent built via pre-trained language models) showed millions of RL steps for one game, no transfer across games. The eureka moment: humans don’t only choose go to cabinet 2 / open chest 3 — they also choose to think: “the dungeon is dangerous and I need a weapon. Chest 3 is in cabinet 2 — let me go unlock it”. Adding reasoning to the action space (image 2: the canonical ReAct figure — Obs → Reasoning → Act with the cooking-and-soy-sauce trajectory) gives language pretraining priors something to do, and lets the model afford flexible test-time compute per decision. In classical RL this should be terrible — adding infinite “do-nothing” actions makes decision-making impossible — yet empirically it works. Author’s distilled formulation: language generalizes through reasoning in agents. See react for the concept page.
• Once priors + reasoning-as-action are in place, the algorithm becomes nearly trivial. That’s how we got the o-series, R1, deep research, computer-using agents. “What a sarcastic turn of events: for so long RL researchers cared about algorithms way more than environments, and no one paid attention to priors — all RL experiments essentially started from scratch.”
• The first-half game is being ruined by the recipe. Two mechanisms:
  • The recipe has standardized and industrialized benchmark-hillclimbing. A novel method targeted at one task might lift it 5%; the next o-series model lifts it 30% without targeting it.
  • Even harder benchmarks now get solved increasingly soon. Image 3 (Jason Wei’s chart, Progress on AI benchmarks in the past five years) shows the trend — TriviaQA, MMLU, GSM8K saturating by 2022–2023; MATH from ~5% in 2021 to ~95% by 2024; GPQA from ~40% in 2023 to ~95% in early 2025; SWE-bench-verified from ~5% to ~80% in 2024–2025; AIME from ~10% to ~95% in roughly a year; “Humanity’s last exam” debuted late 2024 at ~5% and climbed within months. Construction-rate of harder benchmarks lags the recipe’s rate of solving them.
• The second half is an evaluation game. Yao’s claim is sharper than “make harder benchmarks”: fundamentally re-think evaluation. Don’t only crank difficulty — question the setups existing benchmarks treat as laws. The challenge is inertia: humans build harder MATH after solving MATH, find harder coding after solving HumanEval, etc., without questioning the assumptions baked into the eval frame.
• The utility problem is the load-bearing motivation. AI has beaten world champions at chess and Go, surpassed most humans on SAT and bar exams, reached gold at IMO and IOI. “But the world hasn’t changed much, at least judged by economics and GDP.” Yao calls this the utility problem, “the most important problem for AI”. The root cause, he argues, is deceptively simple: eval setups differ from real-world setups in basic ways.
• Two specific assumptions to question. The article names two — both treated as universal but neither inevitable:
  • Evaluation “should” run autonomously. Standard frame: agent receives task input, runs autonomously to completion, gets a single reward at the end. Real frame: a customer-service agent doesn’t take one super-long message, work for 10 minutes, then return a final response — it engages with the human throughout. Counter-examples that break this assumption: Chatbot Arena (real humans in the loop) and tau-bench (user simulator in the loop). Image 4 shows tau-bench’s setup: tools (get_user_details, book_reservation, …) + domain policy as system prompt + user instructions as system prompt; the example trajectory has the agent mid-task asking the simulated user “JK9O19 is basic economy and cannot be changed. But since it is within 24h, I can cancel it and book a new one. Do you want me to do it?” — the agent is forced to negotiate, not just execute.
  • Evaluation “should” run i.i.d. Standard frame: 500-task test set, run each independently, average. Real frame: tasks are sequential. A Google SWE gets better at the google3 repo over time; a SWE-agent solves many issues in the same repo without any such accumulation. The wiki has long-term memory and cross-session continuity mechanisms in principle, but academia lacks benchmarks that justify them or even question the i.i.d. assumption. The assumption hides the gap.
• Why these assumptions held for so long. Yao’s compressed answer: “when intelligence is low, improving intelligence generally improves utility.” So benchmarks built under autonomous + i.i.d. assumptions worked fine across the first half — every uplift translated. Now intelligence is high enough that the recipe is guaranteed to lift any benchmark within those assumptions, so any further uplift along the existing axes barely moves real-world utility.
• Closing prescription for second-half work. Two-step loop:
  • Develop novel evaluation setups (or tasks) targeted at real-world utility.
  • Solve them with the recipe, or augment the recipe with novel components.
  • The recipe will crush incremental methods unless you create new assumptions that break it. Then you get to do truly game-changing research. “While players in the first half solve video games and exams, players in the second half get to build billion or trillion dollar companies by building useful products out of intelligence.”

Notable claims

• Method papers got 50–130× the citations of their “main” benchmark papers in the first half. Concrete pairs from image 1: Transformer / WMT’14 workshop report (~168K / ~1.3K); AlexNet / ImageNet (~172K / ~49K); GPT-3 / SuperGLUE (~49K / ~2.5K). Used as direct evidence that methods were treated as the prestige object and benchmarks as commodities.
• Benchmark saturation now happens in months, not years. From image 3: SWE-bench-verified ~5% → ~80% in 2024–2025; AIME ~10% → ~95% in roughly a year; “Humanity’s last exam” debuted ~5% in late 2024 and climbed within months. Pattern: any benchmark within the recipe’s reach gets solved before the community can build a successor.
• CALM (the author’s 2019 paper) was the first agent in the world built via pre-trained language models — text-based games on top of GPT-2, taking “millions of RL steps to hillclimb a single game” and not transferring across games. Cited as the empirical anomaly that drove the reasoning-as-action insight: humans generalize because they can also choose to think.
• Adding reasoning to the action space “afford[s] flexible test-time compute for different decisions” — the framing for what later became inference-compute scaling in reasoning-models. Yao explicitly calls reasoning a strange action: it doesn’t change the external world, the action space is combinatorially infinite, and classical RL theory says decision-making should collapse — yet empirically it generalizes precisely because of the language priors the model already has.
• “Language generalizes through reasoning in agents” — Yao’s compressed thesis statement. Cited in react as the page’s load-bearing line.
• OpenAI’s pre-2022 environment-first plan didn’t transfer. Concrete trajectory named: gym → World of Bits → Universe → Dota → robotic Rubik’s Cube. “It never came close to solving computer use or web navigation, and the RL agents working in one domain do not transfer to another.” Resolution came from priors (language pretraining), not from a better algorithm or better environment.
• The utility problem is named explicitly: “AI has beat world champions at chess and Go, surpassed most humans on SAT and bar exams, and reached gold medal level on IOI and IMO. But the world hasn’t changed much, at least judged by economics and GDP. I call this the utility problem, and deem it the most important problem for AI.”
• Two evaluation assumptions identified as load-bearing: autonomous-eval and i.i.d.-eval. Both are presented as questionable, neither dismissed — Yao’s point is that the field treats them as laws when they’re assumptions specific to first-half setups.
• Acknowledgement: the post itself was drafted with OpenAI deep research from the talk slides (“This blog post is based on my talk given at Stanford 224N and Columbia. I used OpenAI deep research to read my slides and write a draft.”). A small but striking instance of the recipe writing about itself.

Notable quotes

RL finally works. More precisely: RL finally generalizes.

The most important part of RL might not even be the RL algorithm or environment, but the priors, which can be obtained in a way totally unrelated from RL.

Even though you add infinite empty boxes, you have seen them throughout your life in all kinds of games, and choosing these boxes prepares you to better choose the box with money for any given game. […] Language generalizes through reasoning in agents.

AI has beat world champions at chess and Go, surpassed most humans on SAT and bar exams, and reached gold medal level on IOI and IMO. But the world hasn’t changed much, at least judged by economics and GDP. I call this the utility problem, and deem it the most important problem for AI.

Our evaluation setups are different from real-world setups in many basic ways.

When the intelligence is low, improving intelligence generally improves utility. But now, the general recipe is guaranteed to work under these assumptions.

While players in the first half solve video games and exams, players in the second half get to build billion or trillion dollar companies by building useful products out of intelligence.

Open questions

• The essay names the utility problem but doesn’t define it precisely or propose a metric. “Judged by economics and GDP” is the gestural standard; the wiki should track whether later sources operationalize it (a “utility” measurement that isn’t just another benchmark).
• The two assumptions called out (autonomous + i.i.d.) are illustrative, not exhaustive. What other assumptions baked into current evaluation are similarly load-bearing? The essay ends with the loop (“create new assumptions that break the recipe”) but doesn’t list candidates beyond these two.
• The reasoning-as-action argument is presented as intuitive (“you’ve seen empty boxes throughout your life, so the choice prepares you”) rather than mechanistic. The author flags this himself: “It is a really magical thing and I apologize for not fully making sense of it here, I might need to write another blog post just for it.” Worth tracking if a follow-up appears.
• Empirical scope of the autonomous-eval critique: Chatbot Arena and tau-bench are the two examples; both still constrain the loop heavily (a single conversation, a defined task domain). What does fully non-autonomous evaluation look like — multi-week customer-service workflows, ongoing product collaborations, etc.? The essay doesn’t say.
• Cross-link with meta-harness: Yao’s “augment the recipe with novel components” prescription is structurally similar to outer-loop harness optimization, but he doesn’t name it. Worth a note if a future ingest connects them. Partially resolved (2026-07-12): 2026-07-07-harness-engineering-self-improvement made the connection — Meta-Harness now sits inside the recursive-self-improvement progression (prompts → context → workflow → harness code → optimizer code), which is “augmenting the recipe with novel components” made systematic. What remains open is whether Yao would count harness search as second-half work (it still optimizes benchmark scores) or first-half work in new clothes.

Pointers

• The author’s prior papers cited in the essay: CALM (2019, text-game agents on GPT-2 — pre-ReAct attempt); ReAct (2022 — reasoning-as-action). The wiki’s react concept page consolidates the framing.
• Co-cited benchmarks: MMLU (Hendrycks et al. 2020) (the “successful evals based on human exams” example); “Humanity’s last exam” / agi.safe.ai (the “much harder” successor); the HumanEval-style coding benchmarks and the harder LiveCodeBench-style successor; Chatbot Arena (lmarena.ai); tau-bench; long-term memory papers cited in passing (Liu et al., Yitao Liu CER).
• Companion sources in this wiki: 2026-04-27-llm-training-principles-paths-practices (back-half-of-the-pipeline frame; intersects on reasoning-models and post-training but stays inside the training stack — Yao’s essay is one layer up, asking what the training stack should be optimizing toward); 2026-04-27-agent-principles-architecture-engineering (runtime Agent engineering; intersects on agent-loop and agent-evaluation).
• The chart in image 3 is by Jason Wei, OpenAI colleague — attributed inline; not its own page.