Most of the Agent Is the Harness

Recently, Chris (my partner in HITL) and I were having a conversation about buying dedicated hardware to run an open-source model on it, something we own and control. Our goal was to see how it actually stacks up against the frontier labs. My first question back was simple. How do we use the model?

That question is the whole thing. A model sitting on a hard drive doesn't do anything. It's a faucet of intelligence with nothing plumbed to it. It answers when you ask, and forgets everything the second it's done.

Here's the plainest way I can put it. A model on its own is a text predictor. Words go in, words come out, and it forgets the exchange the moment it ends. A harness is the software wrapped around the model: a memory it can write to, tools it can call, accounts it can act through, and a place to keep running. The model decides what to do. The harness is what lets it actually do it, and remember that it did.

"Harness" used to be a word you'd only hear from people building agents on GitHub. It isn't anymore. Anthropic titled an engineering post "Effective harnesses for long-running agents" and named a Claude Code feature "A harness for every task." OpenAI built out a parallel discipline it calls harness engineering, treated internally as separate work from training the model. Both labs are pointing at the same split. The model reasons. The harness decides what that reasoning turns into.

The clearest number came out of a study published in May, "Stop Comparing LLM Agents Without Disclosing the Harness." Researchers ran the same three models through three different scaffolds and measured what actually explained the swing in results. The harness explained close to eight times more of the variance than which model was running. In one test, Claude Opus 4.5 scored 45.9% under one scaffold and 55.4% under another on the same benchmark. Same model, nearly ten points apart, from nothing but the wrapper around it.

That's one paper, and one paper is never the whole story. What makes me trust the direction is that shipping teams report the same shape. Replit split their coding agent's testing into a separate subagent, no model change, and watched autonomous work time climb from 20 minutes to over 200. Cognition's Devin Fusion hands execution from a frontier "main agent" to a cheaper model running alongside it, and calls the resulting 35% cost cut an architecture win. Harrison Chase at LangChain points out that Claude Code, built by the same lab that makes the model it runs on, reportedly ships with something like half a million lines of harness code wrapped around it.

The pattern holds. The model stayed the same. What changed was memory, tools, and scheduling, and none of those live inside a model's weights.

Here's the part I hadn't thought through until I was running one of these myself. The harness isn't one thing you either have or don't. It's a ladder. Each rung adds a capability, and each capability it adds is also a new thing that can break. Capability and fragility are the same climb.

The bottom rung is chat. ChatGPT, Claude.ai. The harness here is light but real. It remembers you across sessions, searches the web, runs a handful of built-in tools like code and file analysis, and even loads skills for certain tasks. What it doesn't do is act on its own. It answers one turn at a time, and you drive every step. The ceiling is low, and so is the risk. This is still where most people live. OpenAI's own usage research, published through the National Bureau of Economic Research, found "asking" for advice makes up roughly half of ChatGPT conversations. Programming dropped to 4 to 5% of messages, down from 12% in mid-2024. That work didn't disappear. It climbed to the next rung.

The middle rung is the session agent. Claude Code, Cowork. The harness adds the thing chat doesn't have: a task loop that runs in your actual environment. It works on your real files, runs commands, and chains steps toward an outcome on its own instead of waiting for your next message. The new failure class is that tool calls fail and context fills up and it goes wrong mid-task. But you're right there watching, and when you close the session, it stops. The blast radius ends at the session.

The top rung is always-on. Hermes Agent, OpenClaw, among the fastest-growing open-source projects on GitHub right now. The harness adds a heartbeat that runs on a schedule, its own logins to your live accounts, and the autonomy to act without being asked. Now it works across days with nobody standing over it. The new failure class is the whole reason the next section exists, because the same three things that make it powerful, persistence and autonomy and account access, are exactly what breaks badly when it breaks.

Almost nobody is up there yet. The most recent enterprise survey, from Menlo Ventures, found only 16% of enterprise deployments and 27% of startup deployments qualify as real agents, meaning something that plans, executes, and adjusts on its own. Copilots, the bottom two rungs, hold 86% of enterprise AI spend. The top rung gets most of the writing and the least of the actual use, and once you run something from up there, you understand the gap.

I climbed to the top rung with OpenClaw, and the fragility there isn't the kind you shrug off.

Start with security, because always-on plus account access plus tool execution is a genuinely dangerous combination. In a single month this spring, researchers logged close to 200 CVEs against OpenClaw. One of them, CVE-2026-32922, rated 9.9 out of 10, let a low-privilege caller mint broader tokens and walk up to admin access and remote code execution. It got patched, but the volume itself is the signal: this is young infrastructure holding the keys to your shell and your messaging accounts.

Then there's prompt injection, which IBM's X-Force and a security analysis of the framework both flag as structurally unsolved. When your agent reads incoming messages and also holds shell and file access, a message from anyone can carry an instruction that becomes a real action. The autonomy that makes it useful is the same autonomy that carries the attack.

And it can quietly burn money. A failed tool call gets retried, the error gets appended to the context, and each loop runs longer and costs more than the last. Left alone overnight, an agent can run up a bill with nobody watching. That last part is the theme of the whole rung: up here, the failures happen while you're not looking. Hermes Agent doesn't have a comparable public trail of incidents yet, though I'd read that as a smaller user base rather than a solved problem, and I'd want to run it myself before saying otherwise.

The honest question underneath all of it: does a good enough model eventually make the harness unnecessary, so the ladder collapses into a single rung?

Part of it does. Harrison Chase makes the sharpest version of this: when people say models will absorb the scaffolding, "This is not true. What has happened (and will continue to happen) is that a lot of the scaffolding needed in 2023 is no longer needed. But this has been replaced by other types of scaffolding." Old rungs dissolve. New ones appear higher up.

But memory, tools, and always-on operation aren't scaffolding that dissolves, because they aren't capability problems. A model's forward pass keeps no state between calls; anything it "remembers" has to be stored and fed back in from outside. Tool use is the same by construction: as one plain description of function calling puts it, "the LLM itself does not execute the function. Instead, it identifies the appropriate function... and provides the information in a structured JSON format." The model emits intent. Something external does the work. The "LLM as OS" framing from Rutgers casts the model as the kernel and everything around it, memory, file system, tools, as the operating system the kernel still needs.

I couldn't find a single lab, Anthropic, OpenAI, DeepMind, or Google, working on folding tool execution into the weights. The closest counterweight is continual learning, which could shrink how much memory lives outside the model. Even a16z, making that case, concedes that in-context memory "will remain the first line of adaptation."

Memory, tools, and always-on operation get built around the model. As far as I can tell, that's the shape of the thing, and it holds no matter how good the faucet gets.

Chris and I are still going to build that machine. When it's running, the real test isn't how smart the open model is on its own. It's how high up the ladder a good harness can carry a small model, against a frontier model with nothing plumbed to it.

Researched across parallel with Claude Code, with passes on harness terminology, benchmark data, security incidents, and whether the model can absorb any of it, then written from there. Fitting, for a piece about what a harness actually does.

Thanks for reading through.
I’d love to know how you felt about today’s newsletter. This will help me make the newsletter better.