recursive-self-improvement
Recursive Self-Improvement (RSI) — the feedback loop where an AI system uses its current intelligence to improve the cognitive machinery that produces its intelligence. The concept dates to I. J. Good (1965, “ultraintelligent machine”) and was named by Yudkowsky (2008).
Modern interpretation
In modern AI, RSI does not (yet) mean a model rewriting its own weights directly. The practical near-term path runs through improving the training pipeline and deployment system (i.e. the harness), which enables a better successor model (2026-07-07-harness-engineering-self-improvement).
The spectrum of what can be recursively improved, from least to most radical:
| Level | What’s optimized | Example |
|---|---|---|
| Prompts | Static text instructions | context-engineering, prompt engineering |
| Structured context | What enters the window, in what form | [[agentic-context-engineering |
| Workflow | The control-flow graph of agent execution | ADAS, AFlow |
| Harness code | The full program orchestrating the model | meta-harness, self-improving-harness |
| Optimizer code | The code that optimizes the harness | STOP, Meta-Harness outer loop |
| Model weights | The parameters themselves | SIA, future continual learning |
Weng’s near-term prediction
- Harness engineering evolves toward meta-methodology — fewer heuristic rules, more general mechanisms.
- Mature harnesses enable auto-research loops; smarter models prevent overengineering.
- Eventually, many harness improvements will be internalized into core model behavior (as prompt tricks were internalized by instruction tuning), but the interface with external context and tools should remain.
Why harness-level RSI first
- Base-model capability threshold: STOP (Zelikman et al. 2023) showed recursive improvement worked with GPT-4 but degraded with weaker models. The base model must be capable enough to improve its own machinery (2026-07-07-harness-engineering-self-improvement).
- Evaluability: Harness-level changes are testable — run the harness, score the output, compare. Weight-level changes require expensive training loops and are harder to validate incrementally.
- Inspectability: Harness code is readable; weight changes are opaque. Permission control and safety review can operate on code artifacts.
The RSI concern
The same feedback loop that enables productive self-improvement also creates risks:
- reward-hacking — the system optimizes whatever signal it’s given, including exploiting the evaluator.
- Diversity collapse — evolutionary/RL loops converge on local optima (evolutionary-search-llm).
- Broken abstraction boundaries — if the program can edit the OS/evaluator, all guarantees break (self-improving-harness).
Weng’s prescription: evaluator and permission control should sit outside the optimization loop. How much oversight can be scaled up and automated remains open (2026-07-07-harness-engineering-self-improvement).
Cross-references
- harness — the runtime engineering frame; RSI’s near-term substrate.
- meta-harness — the outer-loop optimizer over harness programs.
- self-improving-harness — STOP, Self-Harness, DGM: specific systems that implement RSI at the harness level.
- evolutionary-search-llm — evolutionary methods as the search algorithm for RSI.