Research report · August 2026
Why the environment around the model, not the model, now decides what ships
An agent harness is everything in the system except the model: the instructions it reads, the tools it can call, the checks its output must pass, and the record of what it did. This report traces where the term came from, examines the public evidence behind it, and sets out what a production harness contains. Written for engineering leadership; every substantive claim is linked to its source.
Executive summary
~1MLines of agent-written code in 5 months, OpenAI
1,500Merged PRs from a team of 3, later 7
3.5PRs per engineer per day, rising as the team grew
+17.5ppPass-rate gain on one model from a harness change alone
The discipline crystallized in one week of February 2026. Mitchell Hashimoto named the practice on February 5, OpenAI published its field report on February 11, and an independent benchmark on February 12 showed 15 models improving at coding when only the harness changed. Anthropic had described the underlying pattern in November 2025.
The evidence points one way: with model capability increasingly interchangeable, the environment around the model is where output quality, safety, and velocity are actually determined. OpenAI, Anthropic, and independent engineers reached that conclusion separately, from production experience rather than theory.
The teams shipping reliable agents are not the ones with the best prompts. They are the ones that made every agent mistake structurally impossible to repeat.
1. What a harness is
A large language model is a stateless function: tokens in, tokens out. It keeps no memory between sessions, cannot execute code or reach a file system on its own, and cannot verify its own output. Every useful agent product wraps that function in software that supplies what the model lacks. deepset's working definition is the cleanest: the harness is everything in an AI agent system except the model itself.
The practice sits above two better-known layers. Each answers a different question, and the newer layers do not replace the older ones.
Prompt engineering
Optimizes a single exchange: one instruction, one output. Tactical and turn by turn. Peak attention 2023 to 2024.
Context engineering
Manages what the model sees in one run: retrieval, memory, compaction. Rose to prominence mid 2025.
Harness engineering
Designs the whole execution environment for multi-hour unsupervised runs: what the system blocks, measures, and repairs. Named February 2026.
2. How the term emerged
Feb 5, 2026Mitchell Hashimoto, co-founder of HashiCorp, names the practice in My AI adoption journey: "Anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again." In Ghostty, nearly every line of AGENTS.md traces to one bad agent behavior; once added, the mistake almost never recurs.
Feb 12, 2026Can Bölük publishes The harness problem, improving 15 models at coding in one afternoon by changing only the edit format.
3. The evidence
3.1 OpenAI: a million lines with zero manual code
The strongest public datapoint. Starting from an empty repository in late August 2025, a team of three engineers, later seven, shipped an internal product with no manually written code. Five months in, the repository held on the order of a million lines and roughly 1,500 merged pull requests, an average of 3.5 PRs per engineer per day. OpenAI estimates the build took about a tenth of the time hand-written code would have. The product has hundreds of internal users; what it is has not been disclosed. Their operating philosophy: humans steer, agents execute.
What the team actually built, in their own account, was almost entirely harness:
- AGENTS.md as a table of contents: roughly 100 lines pointing into a structured docs/ tree that is the system of record. The one big instruction file "rots instantly."
- The app boots per git worktree, and Codex drives it through the Chrome DevTools Protocol to reproduce bugs and validate its own fixes.
- Each worktree gets an ephemeral observability stack; agents query logs and metrics directly with LogQL and PromQL.
- A fixed layer order per domain, Types, Config, Repo, Service, Runtime, UI, enforced by custom linters whose error messages inject remediation instructions into agent context.
- Recurring cleanup agents replaced the old Friday "AI slop" sweeps, opening small refactoring PRs most of which are reviewed in under a minute.
- Single runs regularly work a task for upwards of six hours, often overnight.
OpenAI's own caveat: the end-to-end autonomy "depends heavily on the specific structure and tooling of this repository and should not be assumed to generalize without similar investment."
<div class="he-evidence-card">
<h3 id="can-boluk-15-models-one-afternoon-no-training">3.2 Can Bölük: 15 models, one afternoon, no training</h3>
<p>The cleanest controlled result. Bölük replaced the standard diff-based edit format with "hashline," which tags each line with a content hash, and benchmarked 3 runs of 180 tasks per model. The new format beat the standard patch format in 14 of 16 models tested; a v2 revision improved further in 12 of 16. The largest gain was GPT-5.1 Codex Mini, from 60.0% to 77.5% pass rate. Gemini gained 8 points, which he notes is <a href="https://blog.can.ac/2026/02/12/the-harness-problem/" target="_blank" rel="noopener noreferrer">"bigger than most model upgrades deliver, and it cost zero training compute"</a>, about $300 of benchmark spend in total. No weights changed. Only the harness did.</p>
</div>
<div class="he-evidence-card">
<h3 id="anthropic-surviving-the-context-window-boundary">3.3 Anthropic: surviving the context window boundary</h3>
<p>Anthropic's harness work targets the hardest structural problem in long-running agents: memory ends where the context window does. Its published pattern splits the run in two:</p>
<ul class="he-arrow-list">
<li>An initializer agent runs once: init script, git baseline, a JSON feature list, and a progress file that becomes the session-to-session log.</li>
<li>A coding agent then makes incremental progress each session, leaving a clean state at every boundary so the next window can resume.</li>
<li>The March 2026 follow-up adds planner, generator, and evaluator roles with structured handoff artifacts and verification gates.</li>
</ul>
</div>
4. One model, three harnesses
The July 9 ChatGPT desktop app is the clearest product illustration of the idea. Its three modes run on the same GPT-5.6 model family and keep separate execution contexts. What changes between them is the weight of the harness.
ChatThin
System prompt, a small tool set (search, image generation, sandboxed Python), and safety filters on both directions. Built for turn-by-turn conversation.
WorkMedium
Adds a planning loop, connectors into files and enterprise apps (Slack, Drive, SharePoint, calendars), approval checkpoints before high-risk actions, and scheduled background tasks.
CodexHeavy
Repo-aware coding agent: scoped sandbox, AGENTS.md ingestion, full terminal access with test and build feedback, diff generation and PR review. Same weights, autonomous developer.
Mode descriptions from OpenAI's July 9, 2026 release and contemporaneous coverage; capability lists per mode follow Srinivasan's breakdown.
5. Anatomy of a production harness
Across the OpenAI, Anthropic, and practitioner accounts, the same five components recur. This is the layer diagram of the discipline.
01System of record
Instruction files (AGENTS.md, CLAUDE.md) where the agent works. OpenAI's lesson: keep it a 100-line map into a docs/ tree, not an encyclopedia. Knowledge in Slack or Docs does not exist to the agent.
02Scoped tools
What the agent is permitted to do: edit files, run commands, query databases. A minimal, justified set; access to everything produces chaos, access to nothing produces nothing.
03Verification loops
The engine of autonomy: linters, type checkers, and tests run after every edit, with errors fed back so the agent corrects itself before a human sees the output.
04Guardrails and approvals
What the agent is not allowed to do: read-only paths, blocked commands, and the exact points where it must halt for human approval before a high-risk action.
05Observability and memory
Persistent state across sessions plus a structured log of every decision, tool call, and error. When a run fails, the log is the only way to reconstruct why.
Every major lab now ships a harness as a product, and the boundary between harness and model is itself becoming contested territory.
7. A practical roadmap
For a team starting from zero, the practitioner consensus reduces to five moves, in order.
1Establish a system of record. One page at the repository root: project structure, how to build, how to test, top conventions. Apply Hashimoto's rule: when the agent errs, write a permanent instruction instead of correcting it in chat.
2Wire a verification loop. Linters, type checks, and tests run after every edit, with failures fed back automatically. For non-code output, a validation script that inspects format and accuracy.
3Be deliberate about tools. Grant a minimal, justified set, and for each one ask what the worst thing the agent could do with it is, and what stops that.
4Debug from logs, not guesses. Every failure names what is missing from the harness: an instruction, a test case, or a guardrail.
5Ship one narrow loop. Pick a repetitive weekly task and build a complete small harness around it: a one-page instruction file, two or three tools, automated verification, and logging. Make it run end to end without intervention before scaling up.
One expectation to set with leadership: Hashimoto himself reports an agent running only 10 to 20 percent of his working day, a single agent rather than parallel fleets. The discipline compounds, but it starts small.
8. Where accounts disagree
The story circulates widely, and some retellings drift from the primary sources. Four points worth pinning down:
- Sequence: the briefing video and several posts say Anthropic published "shortly after" Hashimoto. Anthropic's first harness post is dated November 2025, three months before.
- Scale: OpenAI writes "on the order of a million lines," with a team that grew from three to seven engineers. Secondary accounts round this to "over one million lines by three engineers."
- The benchmark: the "15 LLMs" headline covers 16 models in the published charts; the hashline format wins in 14 of the 16.
- The product: OpenAI has not disclosed what the internal million-line product does. Claims about its nature are speculation.
9. Sources
- My AI adoption journeyHashimoto, M. Personal blog, Feb 5, 2026.
- Harness engineering: leveraging Codex in an agent-first worldLopopolo, R. OpenAI, Feb 11, 2026.
- Effective harnesses for long-running agentsAnthropic. Engineering blog, Nov 2025.
- Harness design for long-running application developmentRajasekaran, P. Anthropic, Mar 2026.
- We improved 15 LLMs at coding in one afternoon. Only the harness changed.Bölük, C. blog.can.ac, Feb 12, 2026.
- Developers Digest summaryOpenAI ChatGPT desktop unification, Jul 9, 2026.
- Codex Knowledge Base migration notesOpenAI ChatGPT desktop unification, Jul 9, 2026.
- cwc-long-running-agentsAnthropic. Code with Claude 2026 reference harness primitives.
- Srinivasan, A. Harness Engineering Explained in 20 Mins. YouTube; basis of the briefing document this report verifies and extends.
