self-reflection

Self-reflection — the second half of the agent anatomy‘s Planning component: an agent improving iteratively by critiquing past actions and correcting mistakes (Weng 2023). “It plays a crucial role in real-world tasks where trial and error are inevitable.” Where task-decomposition makes the plan, reflection fixes it.

The methods Weng surveys

Reflexion (Shinn & Labash 2023)

A standard RL setup (binary reward) plus a self-reflection loop layered on a ReAct-style action space. The framework diagram has two bands:

• Act band: Query → LLM → Action → Environment → Reward, producing a trajectory {a₀,o₀,…,a_N,o_N} and rewards {r₀,…,r_N}.
• Reflect band: a heuristic h computed from the trajectory feeds a Reflection (LLM) step whose output is appended back into the query context.

The heuristic detects two failure modes and may reset the environment to start a fresh trial:

• Inefficient planning — trajectories that run too long without success.
• Hallucination — a sequence of consecutive identical actions yielding the same observation (a concrete, mechanical definition of a stuck agent).

Reflections are produced from two-shot (failed trajectory, ideal reflection) examples and kept in working memory (up to three) as context for the next attempt.

Chain of Hindsight (CoH; Liu et al. 2023)

Fine-tunes the model on sequences of its own past outputs, each annotated with human feedback and ordered by reward (r_n ≥ … ≥ r₁), so the model learns to continue the trend of improvement and produce a better output conditioned on the feedback history. Two anti-shortcut tricks: a regularizer on the pretraining log-likelihood, and random masking of 0–5% of past tokens (feedback sequences share many words, so copying is a real failure mode).

Algorithm Distillation (AD; Laskin et al. 2023)

Applies the CoH idea to cross-episode RL learning histories. An agent gets a little better each episode; AD concatenates that improving history and trains a causal transformer (behavioral cloning) to predict the next action — so the network internalizes the RL process itself, not a single task’s policy. Properties: task-agnostic (random task per training run); needs multi-episode (2–4) context for in-context RL to emerge; approaches RL² despite being purely offline, and beats expert distillation (ED, which clones expert trajectories rather than learning histories).

A note on ReAct’s placement

Weng files ReAct under “Self-Reflection”, but the wiki treats ReAct as the loop primitive (reasoning + acting) rather than a reflection method — see react. Reflexion is the page where ReAct’s action space actually gets a reflection step bolted on.

The 2026 descendants

Self-reflection here is prompt-/SFT-level. Its engineered descendants externalize and harden the same instinct:

• ralph-wiggum-loop — the agent requests further agent reviews and loops until reviewers + verifiers are satisfied. Reflexion’s “compute heuristic, reflect, retry” generalized to multiple reviewers.
• verifier-loop — binds explicit acceptance criteria so “the model said done” isn’t trusted; the deterministic counterpart to LLM self-critique.
• reward-hacking — the caution rail: a model reflecting on / grading itself can also learn to game the signal. ChemCrow’s finding (LLM self-evaluation fails in expert domains — see agent-evaluation) is the early warning.

Cross-link

• 2026-06-04-llm-powered-autonomous-agents — the source.
• task-decomposition — the other half of Planning.
• react — the action-space primitive Reflexion builds on.
• ralph-wiggum-loop, verifier-loop — engineered descendants.
• llm-agent — the anatomy this sits inside.