harness

The Harness is everything around the Agent Loop that decides whether the system actually works in production: verification baseline, execution boundaries, feedback signals, fallback paths. tw93‘s central thesis in 2026-04-27-agent-principles-architecture-engineering is that Harness quality moves success rate more than model quality does — and the leverage in Agent engineering is in the layers around the loop, not inside it. 2026-04-27-harness-engineering-codex-agent-first (Ryan Lopopolo / openai / codex, Apr 2026) is the primary-source counterpart: a five-month case study reporting ~10× throughput from the same model when the team’s job becomes designing the harness.

The four parts

Skip any one and the loop will eventually destabilize. The author’s framing: don’t release autonomy until all four are present.

Why Harness > model

Three engineering observations from the source justify the priority order:

1. More expensive models help less than expected. Once a baseline capability is reached, success rate is dominated by Harness — the marginal model upgrade often returns less than a marginal Harness investment.
2. Most “wrong tool selected” is a description problem. Debug the agent-computer-interface before suspecting the model.
3. Eval-system bugs look identical to model regressions. agent-evaluation failures are often Harness failures dressed up as model failures.

The OpenAI Codex case (now from primary source)

2026-04-27-harness-engineering-codex-agent-first (Ryan Lopopolo, Apr 2026) is the primary source for the throughput numbers Tw93’s essays cited at one remove: 3 engineers (now 7), 5 months, ~1M lines / ~1,500 PRs, ~10× a hand-coded baseline. The multiplier was not model strength; the post enumerates the engineering moves explicitly. The wiki now has dedicated pages for several of them:

1. Anything the agent can’t see doesn’t exist. Knowledge lives in the codebase, not in Slack or Google Docs. AGENTS.md ≈ 100 lines, used as an index only — details live in structured docs/ (claude-md, codebase-as-system-of-record).
2. Constraints encoded, not documented. Custom linters and structural tests enforce strict architectural layering (Types → Config → Repo → Service → Runtime → UI; cross-cutting via Providers). Lint error messages embed fix instructions for the next agent run (architectural-invariants).
3. End-to-end autonomy. Verify state → reproduce bug → record video → implement fix → drive the app to verify → record fix video → open PR → handle review feedback → fix CI → merge. Logs / metrics / traces are queried by the agent itself (LogQL / PromQL); the Chrome DevTools Protocol is wired into the agent runtime so it can drive the UI (verifier-loop, agent-legibility).
4. Self-review until quiet. PRs go through a ralph-wiggum-loop: the agent requests further agent reviews, responds to feedback, loops until reviewers + verifiers are satisfied. Most PRs no longer need human review.
5. Minimum merge friction. Flaky tests are retried, not blocked on. “In a system where agent throughput far exceeds human attention, the cost of fixing a small mistake after the fact is less than the cost of waiting.”
6. Cleanup runs continuously. “Golden principles” + scheduled cleanup-Codex tasks open small auto-merging refactor PRs to keep entropy down (entropy-and-garbage-collection).

Single Codex runs over six hours on one task are routine; many run overnight (long-running-agents). The post also explicitly cautions: this depends on the codebase’s specific structure and tooling and should not be assumed to generalize without similar investment.

The 2×2 of where Agents work

Plotting task clarity against verification automation:

Most production failures cluster in the bottom-right (vague goals, automated feedback). The system happily makes “progress” toward something nobody asked for.

Releasing autonomy in order

2026-04-27-agent-principles-architecture-engineering asserts a hard ordering: Harness → rollback → autonomy. Skipping any step is the common cause of Agent incidents.

1. Harness in place — acceptance, boundary, signal, fallback.
2. Rollback in place — provider switch, workspace isolation, allowlist, audit log. Irreversible operations have an undo path.
3. Autonomy expanded — explicit confirmation on sensitive ops, source-sink path-cutting (see prompt-injection), independent LLM verification on critical paths.

Relation to other concepts

• verifier-loop is the acceptance-baseline and feedback-signal parts of Harness instantiated for claude-code. Harness is the broader framing; verifier-loop is one of its layers.
• claude-hooks is one of the cleanest execution-boundary primitives in the claude-code ecosystem.
• agent-sandboxing is the execution boundary pushed down to the operating system: constraints the process tree cannot opt out of. 2026-07-12-codex-windows-sandbox-engineering is the wiki’s best evidence of what that row costs to build when the OS doesn’t cooperate — on Windows it took openai dedicated local users, firewall rules, and a four-binary architecture to make codex‘s default mode actually enforceable.
• agent-evaluation‘s “fix the eval before the Agent” rule is a direct corollary: if the feedback-signal component of the Harness is broken, all downstream signals about the Agent are unreliable.

”Harness” at a different scale: Meta-Harness

2026-04-27-llm-training-principles-paths-practices introduces a different use of the word: Meta-Harness is an outer-loop optimizer that keeps the model fixed and search-optimizes the harness program itself (prompt construction, retrieval, memory, tool orchestration). Same word, different scale:

• This page (Harness). A human-engineered runtime discipline: acceptance + boundary + signal + fallback for one product. Built once, governed continuously.
• Meta-Harness. A search procedure that proposes harness code, runs it, scores it, rewrites it. Its outputs are harness programs; its inputs are scores and execution traces.

The two are complementary. The headline number from the Meta-Harness paper — same base model, harness alone produces ~6× spread on the same benchmark — is the strongest empirical support so far for Tw93’s “Harness > model” thesis on this page. If anything, it argues the case more aggressively: the harness ceiling is high enough that automated search over harness programs is now its own research target.

Cross-link meta-harness for the search-loop view; this page remains the runtime-engineering view.

For the time-slice view — why this concept clustered into a thesis in Q1–Q2 2026 specifically, the convergence of six sources, and the enabling conditions that matured at once — see harness-why-it-matters-now.

Harness as RSI substrate

2026-07-07-harness-engineering-self-improvement (lilian-weng, Jul 2026) places harness engineering explicitly within the recursive-self-improvement frame. The near-term path to RSI runs not through models rewriting their own weights, but through improving the harness — the deployment system that orchestrates how the model thinks, acts, and evaluates.

Weng’s definition upgrades the earlier agent anatomy (LLM + memory + tools + planning) with workflow design, evaluation, permission controls, and persistent state management. A harness is closer to an OS than a prompt template: it encapsulates complex logic while keeping the interface simple.

Three design patterns recur across successful harnesses (2026-07-07-harness-engineering-self-improvement):

1. Workflow Automation — goal-oriented loop until acceptance criteria met.
2. File System as Persistent Memory — durable state in files, not carried in context.
3. Sub-agent and Backend Jobs — explicit, inspectable parallelism.

The progression of what gets optimized:

prompts → structured context → workflow → harness code → optimizer code

This progression directly feeds into self-improving-harness (systems that optimize their own harness) and evolutionary-search-llm (evolutionary methods searching the harness code space).

The coding-agent harness (stabilized interface)

Weng observes that the core interface across Claude Code, Codex, OpenCode, and Cursor-style agents has converged: Observe repo → Plan → Search/read files → Edit/write patches → Run tests → Inspect errors → Repeat. The tool set is grouped as:

This convergence supports the “harness > model” thesis: if the interface has stabilized, competing on harness quality rather than model strength is the margin that matters.