What Is This?
Coding agents have crossed from demos into real software work, but the hard constraint is no longer only capability. It is cost per verified change.
The useful model is:
agent value = useful resolved tasks / total cost of attempted tasks
The numerator is not "tokens produced" or "impressive reasoning." It is shipped, reviewed, verified work. The denominator is not just the model price. It includes retries, context loading, failed branches, human review, CI time, and the opportunity cost of trusting a bad patch.
Frontier models are still the right default for ambiguous architecture, high-blast-radius edits, unclear bugs, and tasks where one missed constraint is expensive. But cheaper open or open-adjacent coding models can beat them on bounded work when the harness is good enough.
The cost wall appears when a team uses the strongest model for every step of an agent loop:
read repo -> search files -> edit -> run tests -> inspect failure -> retry -> summarize
Most of that loop is not frontier intelligence. It is controlled execution. Paying frontier prices for every token in that loop only makes sense when each extra unit of intelligence materially improves the chance of a correct patch.
Why Does It Matter?
Agentic software work is token-hungry by design. A coding agent repeatedly reads files, calls tools, observes errors, updates state, and tries again. Anthropic's engineering write-up on multi-agent research systems makes the economic problem explicit: in their data, agents typically used about 4x more tokens than chat interactions, and multi-agent systems used about 15x more. They also warn that the economics only work when the value of the task justifies the added token burn.
Coding tasks are often less parallel than research tasks, but they have the same cost shape: long context, repeated tool use, retries, and expensive self-explanation. A single successful change can be cheap; a failed autonomous run can burn a surprising amount of context before anyone notices it is drifting.
At the same time, open-model coding capability has improved quickly. Qwen's Qwen3-Coder announcement describes a 480B-parameter mixture-of-experts model with 35B active parameters, long-context support, and agentic coding/tool-use focus. OpenHands, SWE-agent, and similar systems show that the agent harness matters: the model is only one component in a loop that includes a shell, file editor, browser, sandbox, prompt, memory, and evaluation protocol.
So the practical question is not:
can an open model beat the frontier model?
The better question is:
on which task class, inside which harness, under which verification standard,
does the cheaper model produce the lower cost per accepted patch?
That is the real frontier for day-to-day engineering agents.
The Cost Wall
The cost wall has three layers.
First is direct inference cost. Frontier APIs charge more per million input and output tokens than cheaper open-model APIs or self-hosted models. Prices change, so the exact ratio should not be treated as timeless. The stable point is structural: agent loops multiply token consumption, and output tokens are usually the expensive side of the bill.
Second is retry cost. A model that is 20% cheaper but fails twice as often is not cheaper. A model that is much cheaper and fails only on tasks that the harness can detect early may be cheaper. The unit of comparison is not price per token; it is price per verified resolution.
Third is supervision cost. If a cheap agent produces patches that require deep human re-review, the saving disappears. If it operates inside a tight harness with narrow diffs, deterministic tests, and clear stop conditions, review cost stays bounded.
That gives the main rule:
cheap agents win when the task can be made small, observable, and falsifiable
They lose when success depends on broad architectural judgement that the harness cannot test.
Where Open Agents Can Win
Cheaper open-model agents are strongest when the work is bounded and the environment carries the intelligence.
Good candidates:
- fixing a failing unit test with a clear stack trace;
- updating typed call sites after an API rename;
- adding a small UI state that has an existing component pattern;
- writing glue code where examples already exist in the repo;
- converting a narrow lint/type error into a patch;
- applying a planned refactor to repeated files;
- adding tests for an already specified edge case;
- running a known verification command and iterating on the failure.
In these cases, the repo and test harness do much of the thinking. The agent is not discovering the product strategy. It is following a local gradient:
make a small edit -> run the check -> read the failure -> adjust
This is why prepared repositories matter. A repository with clear scripts, stable tests, fixture examples, coding conventions, and agent instructions shortens the effective task. It lets a cheaper model spend fewer tokens rediscovering how to work.
OpenHands is useful evidence here because it frames the agent as a platform, not a model wrapper. Its paper describes a sandboxed environment, command line, web browser, multi-agent coordination, and incorporated evaluation benchmarks. SWE-agent makes a similar point from a different angle: agent-computer interfaces change performance. The system around the model can raise or lower the difficulty of the same task.
Where Frontier Models Still Win
Frontier models are still the right call when the task is messy, under-specified, or high-blast-radius.
Use the stronger model for:
- identifying the real bug behind weak symptoms;
- reading unfamiliar architecture;
- choosing between competing fixes;
- designing a migration path;
- handling security, auth, billing, data loss, or production incidents;
- deciding what not to touch;
- writing the verification contract for another agent;
- reviewing whether a cheap agent's patch solved the right problem.
METR's work on AI task time horizons is the cleanest way to think about this. They argue that task length is a useful lens for agent capability because agents struggle with long sequences of actions, not only with isolated skill. As tasks get longer, state management, sequencing, recovery, and verification become the hard parts.
Open models can do many local steps. The frontier model earns its keep when the task cannot be safely localized.
A good operating pattern is therefore:
frontier model = auditor / planner / reviewer
open model = bounded executor
harness = source of truth
That is not model tribalism. It is division of labour.
Harness Quality Is The Multiplier
A weak harness turns every model into an expensive guesser. A strong harness turns weaker models into useful workers.
A good coding-agent harness has five properties.
1. It makes the task small
The agent should not be told "improve checkout." It should be told: reproduce this failing test, inspect these files, change only the payment-intent status mapping, and stop when the named test passes.
Small tasks reduce the amount of implicit planning required. They also make failure cheaper.
2. It exposes the right tools
SWE-agent's core claim is that language-model agents are a new kind of computer user and need interfaces designed for them. That matters. A model with a clumsy edit loop, noisy file search, or unbounded terminal access may waste tokens and make avoidable mistakes. A model with precise file editing, bounded command output, and clear observations can act more reliably.
3. It has executable checks
Autonomy without verification is drafting. The agent needs a command that can say no.
Good checks include unit tests, typechecks, lint rules, snapshot tests, integration probes, smoke tests, and live route checks. The best checks are fast enough to run repeatedly and specific enough to catch the intended failure.
4. It preserves a trail
The agent should leave a record of files changed, commands run, failures seen, and final evidence. The final message is not evidence. The trail is evidence.
This matters more for cheaper agents because the review burden must stay bounded. If the human has to reconstruct the whole run, the cost advantage is gone.
5. It stops early
A cheap model should not be allowed to wander. Give it budgets: max files, max commands, max minutes, max retries, and clear escalation triggers. If it cannot solve the task inside the bounded loop, hand the case to the frontier model or a human.
The stop rule is part of the economics.
Evaluation Discipline
The trap is comparing models on vibes:
frontier model felt smarter
open model was cheaper
That is not an evaluation. A useful comparison tracks task-level outcomes.
For a coding-agent bake-off, record:
- task type and estimated human time;
- model and harness version;
- context budget and retry budget;
- whether the final patch passed the required tests;
- whether a human accepted it;
- number of attempts;
- total input/output tokens or runtime cost;
- review time;
- defect discovered after merge, if any.
Then compare by bucket, not globally.
cost per accepted patch = total run cost + review cost + rerun cost
divided by accepted verified patches
A frontier model may dominate for ambiguous two-hour bug hunts. An open model may dominate for 15-minute mechanical fixes. A hybrid audit-then-execute workflow may beat both on medium tasks: frontier model writes the plan and verification contract; cheaper model executes; frontier or human reviews only the diff and evidence.
SWE-bench is useful because it moved evaluation toward real GitHub issues. SWE-bench Verified tightened that by filtering tasks for clearer statements and reliable tests. But production teams still need their own evals. Public benchmarks cannot tell you whether an agent works in your repo, with your scripts, your flaky tests, your deploy process, and your review standards.
Validation Surface
- Primary validation: SWE-bench and SWE-bench Verified define real-repository coding tasks and patch-based grading; SWE-agent and OpenHands show that agent interfaces and harnesses are central to performance.
- Independent validation: METR's task-horizon work supports the claim that longer multi-step tasks degrade reliability; Anthropic's engineering write-up supports the claim that agent loops materially increase token consumption.
- Limitation / counter-source: Model and API pricing changes quickly; Qwen3-Coder performance claims are first-party and should be treated as capability signals, not universal proof that open models beat frontier models.
- What remains uncertain: The exact break-even point for open agents depends on the repo, task distribution, hosted-vs-self-hosted economics, reviewer skill, and the quality of local evals.
Why Smart People Get This Wrong
The first mistake is equating cheaper tokens with cheaper work. If the agent makes bad patches, cheap tokens become expensive cleanup.
The second mistake is equating benchmark rank with local usefulness. A model can do well on a benchmark and still fail in a repo with poor setup, missing docs, slow tests, or unclear acceptance criteria.
The third mistake is using frontier models as universal labour. Strong models feel safer, so teams give them every step. That hides cost until usage scales. The better move is to reserve frontier intelligence for uncertainty reduction and use cheaper execution where uncertainty has already been crushed.
The fourth mistake is evaluating only successes. Failed attempts are part of the price. A model that succeeds once after four long failed runs is not equivalent to a model that succeeds first time inside a tight harness.
How To Use This
Use a three-tier routing rule.
Tier 1: Frontier-only
Use the best available model when the task is ambiguous, architectural, security-sensitive, production-sensitive, or hard to verify. The output should be a diagnosis, plan, verification contract, or reviewed patch.
Tier 2: Hybrid
Use a frontier model to inspect and plan, then hand a bounded work packet to a cheaper agent. This is the default for medium coding tasks.
A good packet says:
Goal:
Files likely involved:
Files not to touch:
Edit plan:
Verification commands:
Stop conditions:
Escalate if:
Required final evidence:
Tier 3: Cheap executor
Use an open-model agent directly when the task is local, reversible, and test-backed. Put hard bounds around the run. If it exceeds the bounds, stop rather than letting it spend its way into confusion.
For Jamie's world, this maps cleanly onto Jme-Loop stewardship: use the strong model to name the work and the checks; let a cheaper executor attempt one bounded ticket; require real test output before a change is treated as done.
What This Does Not Prove
This does not prove that open models are generally better coding agents than frontier models. They are not.
It does not prove that self-hosting is automatically cheaper. Hardware utilisation, operations, latency, batching, privacy requirements, and engineering maintenance can erase the apparent saving.
It does not prove that public benchmarks predict local ROI. They are useful external calibration, not a substitute for repo-specific evals.
The narrow claim is stronger and more useful:
when the task is bounded, the harness is strong, and evaluation is real,
cheaper open agents can produce lower cost per verified change
That is the cost wall frontier models are now running into.
Practical Takeaways For Jamie
- Treat model routing as a stewardship problem, not a taste preference.
- Keep frontier models on audit, planning, architecture, and review.
- Push bounded execution into cheaper agents only when tests and stop rules exist.
- Add repo-level instructions, trusted commands, and acceptance checks before expecting agents to perform.
- Track cost per accepted patch, not model cleverness.
Key Terms
- Cost wall: the point where using frontier inference for every step of an agent loop becomes economically dominated by routing easier work to cheaper systems.
- Open-model agent: a coding agent powered by an open-weight or open-adjacent model, often run through a local, hosted, or low-cost API backend.
- Harness: the tool, prompt, sandbox, command, memory, and evaluation environment around the model.
- Verified change: a patch that passes the agreed checks and is accepted by a human or automated review process.
- Task horizon: the length of task an AI agent can complete reliably, often measured in estimated human time.
Recall Questions
- Why is price per token the wrong unit for comparing coding agents?
- What kind of task lets a cheaper open agent beat a frontier model?
- What does a harness provide that the model does not?
- When should a frontier model stay in the loop?
- Why do failed attempts belong in the cost calculation?
Best Resources to Learn More
- SWE-bench for real-repository benchmark design.
- SWE-agent for the agent-computer-interface view of coding agents.
- OpenHands for the agent-platform view: sandbox, tools, browser, shell, and benchmarks.
- METR's task-horizon work for thinking about reliability as task length increases.
- Anthropic's multi-agent engineering write-up for the economics of token-heavy agent loops.
Sources
- Carlos E. Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" arXiv, 2023. https://arxiv.org/abs/2310.06770
- SWE-bench project site and leaderboard. https://www.swebench.com/
- OpenAI, "Introducing SWE-bench Verified," 2024. https://openai.com/index/introducing-swe-bench-verified/
- John Yang et al., "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering," arXiv, 2024. https://arxiv.org/abs/2405.15793
- Xingyao Wang et al., "OpenHands: An Open Platform for AI Software Developers as Generalist Agents," arXiv, 2024/2025. https://arxiv.org/abs/2407.16741
- METR, "Measuring AI Ability to Complete Long Tasks," 2025. https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/
- Anthropic Engineering, "How we built our multi-agent research system," 2025. https://www.anthropic.com/engineering/built-multi-agent-research-system
- Qwen Team, "Qwen3-Coder: Agentic Coding in the World," 2025. https://qwenlm.github.io/blog/qwen3-coder/
- Anthropic API pricing page, accessed during publication. https://www.anthropic.com/pricing#api
- DeepSeek API Models & Pricing page, accessed during publication. https://api-docs.deepseek.com/quick_start/pricing