react

ReAct — Reasoning + Acting. Shunyu Yao‘s framing (Yao et al., ReAct, 2022) for the loop that underlies essentially every modern LLM agent: the model alternates between reasoning (a “thought” — a free-form natural-language step that doesn’t change the external world) and acting (a tool call or environment action that does). The wiki’s primary source for the conceptual claim is Yao’s later essay, which retroactively explains why this works: reasoning-as-action is the bridge that lets language-pretraining priors generalize across RL environments.

The shape

Per turn:

Obs t      → environment observation
Reasoning  → free-form thought ("the dish should be savory and salt is out, I should find soy sauce — it's in the cabinet to my right")
Act t      → environment action ("turn right")
Obs t+1    → new observation ("you see a cabinet and a table")
Act t+1    → ...

(Verbatim trajectory from the figure in 2026-04-27-the-second-half-of-ai.)

The reasoning step is “in the action space” — the agent chooses to think. The thought tokens land in the conversation just like any other action, the model conditions on them on the next turn, and the agent gets cheap test-time compute that scales with task difficulty.

Why this works — the three-component RL view

Yao decomposes RL into algorithm + environment + priors (2026-04-27-the-second-half-of-ai):

• For decades, RL focused on algorithm (REINFORCE → DQN → PPO → TRPO → …). Sutton & Barto’s textbook is “all about algorithms and almost nothing about environments or priors.”
• Deep RL made it clear that environment matters; you can build an “optimal” algorithm that only excels in toy settings if you ignore which environment you’re targeting. OpenAI‘s gym → World of Bits → Universe → Dota → Rubik’s Cube arc is the canonical “environment-first” attempt — it solved individual environments but didn’t transfer.
• The missing piece was priors. Language pretraining (pretraining) compresses general commonsense and language knowledge into the model. With strong priors and the right action space, the algorithm becomes nearly trivial.

ReAct is what makes the priors useful: it gives the language priors something to do per turn. Without reasoning-as-action, the agent has only environment-grounded actions (go to cabinet 2, open chest 3), and the priors leak out only weakly via tool-name selection. With reasoning-as-action, the priors drive a thought process that can compose plans, recover from errors, and cover for sparse training signal.

Why it’s a strange action

Yao explicitly calls reasoning a strange action in RL terms (2026-04-27-the-second-half-of-ai):

• It doesn’t change the external world. No state transition. Classical MDP intuition says it’s a no-op.
• The space is open-ended and combinatorially infinite. A turn can produce a word, a sentence, a paragraph, or 10,000 random English words — same external effect.
• Classical theory says this should be terrible. “Imagine you need to choose one out of two boxes, one with $1Mandoneempty.Yo{u}^{\prime }reexpectedtoearn$500k. Now imagine I add infinite empty boxes — you’re expected to earn nothing.”
• Empirically it generalizes. Yao’s intuitive resolution: language pretraining means the agent has effectively “seen” those infinite empty boxes throughout pretraining; choosing among them prepares the agent to choose the right one in any given game.

His distilled formulation, the page’s load-bearing line:

Language generalizes through reasoning in agents.

The author flags the explanation is more intuitive than mechanistic — “It is a really magical thing and I apologize for not fully making sense of it here, I might need to write another blog post just for it.” Worth tracking if a follow-up appears.

What ReAct enabled

Once the recipe (language priors + scale + reasoning-as-action) was assembled, the bottleneck inverted: the algorithm became the easy part. Per Yao, this is the throughline behind:

• The o-series (reasoning-models) — RL teaches the model how to allocate inference budget over reasoning tokens.
• DeepSeek-R1 — open-weights instance of the same recipe.
• OpenAI deep research, computer-using agents — the same recipe extended to longer horizons / external environments.
• Most production agent loops — the ~20-line ReAct loop is the structural core.

Practical instances in this wiki

• agent-loop — the minimal ~20-line ReAct loop is the structural mental model for every Agent runtime (2026-04-27-agent-principles-architecture-engineering). The “Workflow vs. Agent” distinction on that page is essentially “code-decided control flow vs. ReAct-decided control flow”.
• reasoning-models — reasoning-as-action trained into the model rather than prompted. RL teaches the model when to think more vs. less; ReAct’s thought step becomes a learned policy, not a prompt convention.
• ralph-wiggum-loop — the agent-self-review pattern is a ReAct loop with a structured reviewer step in the action space, looping until reviewers + verifiers agree.
• harness — every named harness in the wiki wraps a ReAct loop with acceptance / boundary / signal / fallback layers.

What ReAct is not

• Not a prompt template. “Thought / Action / Observation” formatting was the original paper’s prompting trick; the concept is the augmented action space, not the literal Thought: token convention.
• Not chain-of-thought. CoT is reasoning that produces an answer in one turn; ReAct interleaves reasoning with environment-changing actions across turns. The two ideas overlap but aren’t the same.
• Not multi-agent orchestration. A single ReAct agent reasons + acts within one loop. Multi-agent systems (multi-agent-orchestration, claude-subagents) coordinate multiple ReAct agents; the underlying loop is unchanged.

Cross-link

• 2026-04-27-the-second-half-of-ai — Yao’s later essay, where the reasoning-as-action insight is named and re-explained as “language generalizes through reasoning in agents”.
• agent-loop — the runtime structure ReAct describes; the wiki’s loop-shape page.
• reasoning-models — what happens when ReAct-style reasoning gets RL-trained into the model.
• pretraining — the priors half of the three-component RL frame.
• post-training — where reasoning-as-action capacity is installed via the four-stage recipe.
• 2026-06-04-llm-powered-autonomous-agents — Lilian Weng’s 2023 survey, the wiki’s earliest synthesis to catalog ReAct (filed there under self-reflection, alongside Reflexion); the conceptual ancestor of this page.