Harness engineering: Leveraging Codex in an agent-first world

A primary-source post from Ryan Lopopolo at openai reporting on a five-month internal experiment: a small team building and shipping a software product where no human wrote a line of code — every line of application logic, tests, CI, docs, observability, and internal tooling came from a codex agent. The team estimates ~10× the throughput of a hand-coded baseline. Approximately 1M LoC, ~1,500 PRs, three engineers grown to seven, with throughput increasing as the team grew. The post is the primary source for the “1500 PRs / 3 engineers / 5 months” Codex case that prior wiki sources cited at one remove via 2026-04-27-claude-code-architecture-governance-engineering / 2026-04-27-agent-principles-architecture-engineering — see harness.

Source file: raw/2026-04-27-harness-engineering-codex-agent-first.md.

Summary

The frame the post commits to up front: humans steer, agents execute. The team’s deliberate constraint is “no manual code.” That constraint forces a re-allocation of human attention away from authoring and toward designing the environment the agent operates in — tools, abstractions, feedback loops, and constraints. The article walks through eight specific moves that re-shaped engineering practice as the codebase grew from an empty git repo (Aug 2025) to ~1M LoC, with the subtext that the leverage points are not in the model — they’re in what surrounds it.

• Engineers stopped writing code; they started writing the environment. Early progress was slower than expected — not because codex couldn’t do the work, but because the environment was under-specified. The team’s job became “figure out what capability the agent is missing, then make that capability legible and enforceable in the repo.” This frames the role shift the post argues for: from author to environment designer.
• Application legibility for agents. Bottleneck moved from code throughput to human QA capacity. The team responded by making the application directly readable by the agent: launch one app instance per git worktree, hook the Chrome DevTools Protocol into the agent runtime so codex can drive the UI / take DOM snapshots / observe runtime events, and stand up an ephemeral observability stack (logs/metrics/traces) per worktree. The agent then queries logs with LogQL and metrics with PromQL — making prompts like “ensure service startup completes in under 800ms” mechanically actionable. Single Codex runs over six hours on one task are routine (often run overnight). See agent-legibility, long-running-agents.
• The repo is the system of record. Knowledge stored in Google Docs, chat logs, or people’s heads is invisible to the agent. The post’s core operating principle: anything not in versioned, repo-local artifacts (code, Markdown, schemas, executable plans) doesn’t exist for the agent. New hires onboard from docs/; the agent has to as well — except the agent has no Slack history, so the bar is even harder. See codebase-as-system-of-record.
• AGENTS.md as table of contents, not encyclopedia. The team tried “one big AGENTS.md” and it failed for four named reasons: context is scarce (a giant instruction file crowds out task content); too much guidance becomes ineffective (“when everything is important, nothing is”); it rots immediately (manual upkeep can’t keep pace); it’s hard to verify mechanically. The replacement: a ~100-line AGENTS.md that’s purely an index pointing into a structured docs/ knowledge base (design-docs/, exec-plans/active/, product-specs/, references/llms.txt per dependency, etc.). A scheduled “doc-gardening” agent scans for docs that no longer reflect code behavior and opens repair PRs. Same idea generalizes to claude-md.
• The bar is agent legibility. Because the codebase is entirely agent-generated, the team optimizes for Codex’s readability: anything the agent can’t see in context at runtime doesn’t exist. This reframes design tradeoffs — preference for “boring” technologies (composable, API-stable, well-represented in training data); occasional choice to re-implement a small subset of a public library rather than depend on its opaque upstream behavior (cited example: reimplemented a p-limit-style concurrent map helper rather than depending on p-limit, because the in-house version is integrated with their OpenTelemetry instrumentation, has 100% coverage, and is fully predictable). See agent-legibility.
• Spec the architecture, enforce invariants mechanically. Agents thrive on strict boundaries and predictable structure (citing logic.inc’s “AI is forcing us to write good code”). Each business domain is fixed-layered (Types → Config → Repo → Service → Runtime → UI), with cross-cutting concerns routed through a single Providers boundary. Allowed dependencies are mechanically enforced via custom linters (themselves Codex-written) and structural tests. Other “taste invariants” (structured logging, naming conventions, file size limits, platform reliability requirements) are also custom-linted; lint error messages include fix instructions for the agent inline. Architectural rigor that’s normally deferred until hundreds of engineers becomes an early prerequisite for agentic coding. See architectural-invariants.
• Throughput changed the merge philosophy. Pull request lifetimes are short. Flaky tests are usually retried, not blocked on. In a system where agent throughput far exceeds human attention, the cost of fixing a small mistake after the fact is less than the cost of waiting for human review. In low-throughput environments this would be irresponsible; here it’s the right call.
• Agents drive the full PR lifecycle (Ralph Wiggum loop). Humans interact almost entirely by prompting. To drive a PR to completion, the team instructs Codex to review its own changes locally, request additional agent reviews (local + cloud), respond to all human/agent feedback, and loop until all reviewers are satisfied — referenced as a “Ralph Wiggum loop” (citing ghuntley.com/loop). Codex uses the standard developer toolchain directly (gh, local scripts, repo-embedded skills) — no human copy/paste of context. Most PRs no longer require human review. See ralph-wiggum-loop.
• End-to-end autonomy crossed a recent threshold. Given a prompt, Codex can now: validate codebase state → reproduce a reported bug → record a video of the failure → implement the fix → run the app to verify → record a second video showing the fix → open the PR → respond to agent and human feedback → detect and fix CI breakages → escalate to a human only when judgment is required → merge. The post explicitly cautions that this depends on the codebase’s specific structure and tooling and should not be assumed to generalize.
• Entropy and garbage collection. Codex reproduces the patterns it finds in the repo — including the bad ones. Initial human-driven cleanup (“AI residue”) consumed ~20% of the team’s week (every Friday); not scalable. Replaced with what they call “golden principles” encoded directly into the codebase (e.g. prefer shared utility packages over hand-rolled helpers; never YOLO-probe shapes — validate at boundaries or rely on typed SDKs) plus a periodic background fleet of Codex tasks that scan for drift, update quality grades, and open targeted refactor PRs. Most are reviewable in under a minute and auto-mergeable. The framing is explicitly garbage-collection-shaped: technical debt is a high-interest loan; pay continuously, in small increments. See entropy-and-garbage-collection.
• What “agent-generated” actually means. Not just product code: also CI configuration, release tooling, internal developer tools, documentation, design history, eval frameworks, review comments and replies, scripts that manage the repo itself, and production dashboard definitions. Humans participate at a different abstraction level: prioritization, turning user feedback into acceptance criteria, validating outcomes. When the agent gets stuck, the team treats it as a signal to find what’s missing — tools, guidance, constraints, docs — and have Codex itself write the fix.
• Discipline moved from code to scaffolding. Closing posture: building software still requires discipline, but the discipline now lives in the supporting structure (tools, abstractions, feedback loops) more than in the code. The hardest current problems are environment design, feedback loops, and control systems. Open: how architectural coherence holds up over longer horizons; where human judgment remains highest-leverage and how to encode more of it.

Notable claims

• ~1M LoC, ~1,500 PRs, three engineers initially, in five months — i.e. ~3.5 PRs / engineer / day on average. Team has grown to seven engineers and throughput has increased. (harness, codex)
• The team estimates ~10× the throughput of a hand-coded baseline for the same product. (codex)
• The product is in production with hundreds of internal alpha users including daily power users. (codex)
• First commit was late August 2025 to an empty git repo; the initial AGENTS.md itself was written by Codex. (codex, AGENTS.md)
• The “one big AGENTS.md” approach was tried and failed for four reasons: context is a scarce resource; too much guidance becomes ineffective; the file rots; it’s mechanically unverifiable. (claude-md)
• The replacement is a ~100-line AGENTS.md acting as an index into structured docs/. (claude-md)
• A scheduled “doc-gardening” agent scans for stale docs and opens repair PRs. (entropy-and-garbage-collection)
• The architecture is enforced mechanically: business domains are layered Types → Config → Repo → Service → Runtime → UI with allowed dependencies, and cross-cutting concerns route through a single Providers interface; custom linters (Codex-written) and structural tests enforce this. (architectural-invariants)
• Linter error messages include fix instructions inline, written for the agent. (architectural-invariants)
• One Codex run over six hours on a single task is routine; many run overnight. (long-running-agents)
• The application can be launched per-git-worktree, with an ephemeral local observability stack (logs/metrics/traces) per worktree; logs queried via LogQL, metrics via PromQL. (agent-legibility, long-running-agents)
• The Chrome DevTools Protocol is hooked into the Codex runtime so the agent can drive the UI, take DOM snapshots, and observe runtime events. (agent-legibility, verifier-loop)
• The team chose to re-implement a p-limit-style concurrent map helper rather than depend on p-limit, citing OpenTelemetry integration, 100% test coverage, and predictable behavior under their runtime. (agent-legibility)
• The PR self-review pattern is named the “Ralph Wiggum loop”, citing ghuntley.com/loop. (ralph-wiggum-loop)
• The end-to-end Codex workflow: validate state → reproduce bug → record failure video → fix → run-to-verify → record fix video → open PR → handle reviews → fix CI → human-judgment escalation → merge. (codex, verifier-loop)
• “AI residue” cleanup originally consumed ~20% of the team’s week before being automated. (entropy-and-garbage-collection)
• Golden-principles examples cited verbatim: prefer shared utility packages over hand-rolled helpers (centralize invariants); don’t YOLO-probe shapes — validate at boundaries or rely on typed SDKs. (entropy-and-garbage-collection)
• Aardvark is named as another agent participating in this codebase, alongside Codex — i.e. the legibility investments are explicitly designed to pay off across multiple agents, not just Codex. (cited link to OpenAI’s Aardvark intro)

Notable quotes

人类掌舵。智能体执行。

(Humans steer. Agents execute.) — the post’s tagline; the constraint that motivates everything else.

代码仓库的知识库位于一个结构化了的 docs/ 目录中，此目录被当作记录系统来使用。一份简短的 AGENTS.md （大约 100 行）被注入到情境中，主要用作地图，并指向其他地方更深层次的真实信息来源。

(The repo’s knowledge base lives in a structured docs/ directory used as a system of record. A short AGENTS.md of ~100 lines is injected into context, used mainly as a map pointing to deeper sources of truth elsewhere.) — the table-of-contents framing.

从智能体的角度来看，它在运行时无法在情境中访问的任何内容都是不存在的。

(From the agent’s perspective, anything it can’t access in context at runtime doesn’t exist.) — the operating premise behind making the repo the single system of record.

在以人为本的工作流程中，这些规则可能会让人感到迂腐或束缚。有了智能体，它们就成了倍增器：一旦编码，它们就能立即应用于所有地方。

(In a human-centered workflow, these rules can feel pedantic or constraining. With agents, they become multipliers: once encoded, they apply everywhere instantly.) — on why architectural rigor that’s normally premature becomes an early prerequisite.

技术债务就像一笔高息贷款：不断地以小额贷款的方式偿还债务，总比让债务不断累积，再痛苦地一次解决要好得多。

(Technical debt is a high-interest loan: repeatedly paying small installments is far better than letting it accumulate and paying painfully all at once.) — the framing for the golden-principles + cleanup-loop combo on entropy-and-garbage-collection.

显而易见的是：构建软件仍然需要纪律，但纪律更多地体现在支撑结构上，而不是代码上。

(What’s clear: building software still demands discipline, but the discipline now lives in the scaffolding more than in the code.) — the closing line; reframes the engineering profession’s center of gravity.

Open questions

• The “~10× throughput” claim relative to a hand-coded baseline isn’t decomposed in the post: how much is from raw model speed, how much from in-house tooling investments (worktree-per-instance, ephemeral observability, custom linters), and how much from the strict no-manual-code constraint forcing scaffolding work that paid back compound interest? Worth comparing against any future OpenAI publication that breaks this down.
• Architectural coherence over time in a fully agent-generated codebase is named as an open question by the post itself. After 1M LoC the system has held; whether the same architecture holds at 10M LoC, with model regressions, with personnel turnover, isn’t claimed.
• The end-to-end autonomy claim is hedged: “depends on the codebase’s specific structure and tooling and should not be assumed to generalize without similar investment.” How much of the investment is portable to other repos / other agents (e.g. claude-code) is left open.
• The AGENTS.md “~100 lines” figure is consistent with the “short, hard, executable” framing on claude-md but the post doesn’t quote a token-count. Compare against anthropic‘s own CLAUDE.md for working on Claude Code (reported at ~2.5K tokens via 2026-04-27-claude-code-architecture-governance-engineering) — ballpark agreement, but worth noting if a future source publishes a definitive sizing.
• The post mentions Aardvark as another agent participating in the codebase. That’s a reference worth chasing into a follow-up source if Aardvark ships its own publication.
• “Custom linters with fix instructions in error messages” is a specific, replicable pattern that is not yet present in the existing wiki on claude-hooks or verifier-loop. If a future Anthropic / Tw93 source describes the same idiom, that’s a candidate concept page; for now, captured under architectural-invariants.

Pointers

• Companion sources in this wiki: this post is the primary source for several claims previously surfaced via 2026-04-27-claude-code-architecture-governance-engineering (the AGENTS.md table-of-contents framing) and 2026-04-27-agent-principles-architecture-engineering (the OpenAI Codex throughput numbers). Where the prior pages had (cite via Tw93's reading of OpenAI), this source replaces that with direct OpenAI authorship.
• External references the post cites: Ralph Wiggum loop (ghuntley.com), AGENTS.md, matklad’s ARCHITECTURE.md, parse don’t validate (Alexis King), AI is forcing us to write good code (logic.inc), OpenAI Cookbook on Codex exec plans, OpenAI’s Aardvark.
• The post is published in both English and Simplified Chinese on openai.com; this wiki ingested the zh-CN URL because that’s the version dropped into raw/. Quotations above are in the source’s original language (zh-CN) as ingested, with English glosses.