What Is This?
Agent performance is not only a model property. It is also a repo-readiness property.
The clean model is:
agent output = model capability x repo preparation x verification quality
A stronger model helps. But a stronger model dropped into an unprepared repository still has to rediscover the same boring facts before it can do useful work:
- which dependency manager is authoritative;
- which install command actually works;
- which test command is trusted;
- which tests are slow, flaky, hidden, or irrelevant;
- which generated files should not be hand-edited;
- which conventions are enforced by maintainers;
- which deploy ritual turns a patch into a live change.
Those details look like admin. For an agent, they are part of the task environment.
This is why prepared repos can beat better models. The prepared repo turns vague autonomy into a tight loop: inspect, edit, run the right check, read the failure, fix, repeat. The unprepared repo turns the same agent into a detective, package-manager archaeologist, and build engineer before the real task even starts.
Why Does It Matter?
Software agents are now evaluated on real repositories, not just code snippets. SWE-bench is the canonical shift. It tests whether AI systems can resolve GitHub issues from popular open-source Python repositories. Each task is built from a real pull request and issue; the benchmark constructs a Docker execution environment with the repository installed at the pre-fix commit, then grades the proposed patch against fail-to-pass tests from the original pull request.
That design hides an important lesson. The benchmark is not only asking, "can the model reason about code?" It is also asking whether the whole system can operate inside a realistic repo ritual: local checkout, dependencies, tests, patch generation, and verification.
SWE-bench Verified made this even clearer. Its 500 tasks were human-filtered to check that problem descriptions were clear, test patches were correct, and tasks were solvable from the available information. The benchmark also separates full agent systems from a "bash only" comparison, where language models are evaluated through mini-SWE-agent in a minimal shell environment. That distinction matters: changing the harness, tools, prompt, and environment changes what the same model can do.
METR's work on software-task time horizons points in the same direction. Their analysis argues that agent capability is better understood by the length of tasks agents can complete reliably, not by isolated exam performance. They report that human task time strongly predicts model success: models are much more reliable on very short tasks than on tasks that take human experts hours. Long tasks fail because they require sustained state, sequencing, verification, and recovery from wrong turns.
Repo preparation shortens the effective task. It removes exploration debt. It makes the next correct action visible.
The Repo Is Part Of The Prompt
Most teams still think of prompting as text given at the start of a chat. Coding agents do not work that way.
A coding agent's prompt includes the repository itself:
files + dependency graph + scripts + tests + docs + comments + commit history + local failures
If those signals disagree, the agent inherits the confusion. A README says npm install; the lockfile says pnpm; CI uses npm ci; the app requires an undocumented environment variable; the real test command lives in a Makefile; generated files look editable; the only deploy script is in someone's shell history. A human maintainer can sometimes paper over that with memory. An agent cannot reliably infer tribal knowledge unless the repo exposes it.
Anthropic's Claude Code best-practices guide states this directly in operational terms. It recommends writing a concise CLAUDE.md with Bash commands, code style, workflow rules, testing instructions, repo etiquette, environment quirks, architectural decisions, and common gotchas. It also warns that bloated instruction files become counterproductive.
The lesson generalizes beyond Claude. Cursor has rules, OpenAI Codex uses repo instruction files and configuration, GitHub Copilot has custom instructions, and most agent harnesses eventually recreate the same primitive: a place for durable repo-specific instructions that the model reads before acting.
The instruction file is not magic. It is a compression layer for everything the maintainer would otherwise have to say repeatedly.
Verification Is The Real Autonomy Boundary
An agent that cannot verify its work is not autonomous. It is only drafting.
Anthropic's guide puts the failure mode plainly: Claude stops when the work looks done. Without a runnable check, "looks done" is the only signal available, and the human becomes the verification loop. The recommended fix is to give the agent a pass/fail signal it can run: tests, build exit codes, linters, fixture diffs, browser screenshots, or another check whose output can be read and acted on.
That is the difference between:
"fix the bug"
and:
"fix the bug, run npm test -- auth, then run npm run typecheck; stop only when both pass"
The second version is not just a better prompt. It changes the control loop. The agent can observe failure and iterate without waiting for a human to notice that the first patch was wrong.
GitHub's continuous-integration docs describe the human-team version of the same discipline: when code is committed, teams continuously build and test it to catch errors early. Agent work needs that ritual even more, because the agent's confidence is not evidence. The evidence is the command output, the diff, the reproduced bug, the passing test, and the live verification.
The Six Prepared-Repo Primitives
A repo is agent-ready when the agent can answer six questions without guessing.
1. Bootstrap
How do I get to a working local state?
This includes the package manager, install command, required runtimes, database setup, seed data, environment variables, generated clients, and service dependencies.
Good examples:
Use Node 20. Run npm ci. Copy .env.example to .env.local. No secrets are required for unit tests.
Bad examples:
Install dependencies as usual.
The problem with "as usual" is that the agent does not know whose usual is meant.
2. Trusted commands
Which commands prove the change?
A prepared repo names the smallest useful verification ladder:
npm run lint
npm run typecheck
npm test
npm run e2e -- --project=chromium
It also says when not to run something. If npm run build corrupts ownership on a production box, that belongs in the repo ritual. If one test suite takes 45 minutes, the agent needs the narrow command for local iteration and the full command for final verification.
3. Task-local tests
How do I test the thing I changed?
Large test suites create a search problem. A good repo gives the agent local handles: naming conventions, test directories, fixture locations, and examples of narrow test invocations.
This matters because agent loops are budgeted by context, time, and tool calls. A precise test command converts uncertainty into a fast feedback loop.
4. Non-obvious constraints
What would a maintainer know that the code does not say?
Examples:
- do not edit generated files;
- migration files are append-only;
- public routes must stay stable;
- UI copy follows a house style;
- a service runs under a different Unix user;
- deployment is not triggered by Git push;
- a flaky test should be rerun once but not ignored.
These are exactly the details that make agents look careless when they are missing.
5. Recovery paths
What should I do when the obvious command fails?
Prepared repos include fallback rituals:
If Playwright browsers are missing, run npx playwright install chromium.
If Prisma types are stale, run npm run db:generate.
If the dev server is already running, check port 3000 before starting another one.
This prevents the agent from improvising risky fixes for environmental failures.
6. Evidence expectations
What proof should the agent leave behind?
A good final state is not "done". It is:
changed files + commands run + outputs observed + remaining risks
For public-facing changes, that includes live checks. For library changes, it includes the route loading with the new title. For backend changes, it may include API responses. For data migrations, it includes before/after counts.
Validation Surface
- Primary validation: SWE-bench and SWE-bench Verified show that real-repo agent evaluation depends on installed environments, issue text, patches, and fail-to-pass tests.
- Independent validation: Anthropic's Claude Code guidance independently recommends explicit repo instructions, runnable verification, and concise environment configuration for coding agents.
- Operational validation: GitHub's CI documentation describes the same build-and-test discipline in normal software teams; agent workflows inherit that discipline rather than replacing it.
- Capability-context validation: METR's time-horizon work supports the broader claim that multi-step reliability depends on task length, sequencing, and sustained execution, not isolated model skill alone.
- What remains uncertain: the exact performance lift from any single repo-preparation practice varies by model, harness, language, task type, and baseline repo quality.
Why Smart People Get This Wrong
They confuse intelligence with operating leverage.
A better model can read more code, infer more intent, and recover from more mistakes. That makes it tempting to treat repo preparation as temporary scaffolding that will disappear when models improve.
But repo preparation is not only compensation for weak models. It is also how teams encode their own standards.
Humans also need setup docs, CI, tests, deploy scripts, architecture notes, and style guides. Better engineers do not make those artifacts obsolete. They make better use of them.
The same is true for agents. As models improve, the bottleneck shifts from "can it write code?" to "can it safely operate inside this system for longer without human rescue?" That second question is mostly about prepared state, trusted checks, and recovery rituals.
What This Does Not Prove
This does not prove that repo preparation matters more than model quality in every case.
If the task requires a capability the model lacks, no amount of AGENTS.md polish will solve it. A weak model still fails at deep architecture, ambiguous product judgment, unfamiliar APIs, or multi-file refactors that exceed its planning ability.
It also does not prove that instruction files should become giant manuals. Anthropic explicitly warns against bloated CLAUDE.md files because excess context can degrade performance. The point is not more text. The point is less ambiguity.
The practical claim is narrower:
for coding agents near the capability frontier, repo readiness often determines whether capability converts into completed work
How To Use This
Treat every important repo as an agent workplace.
Create one short repo-ritual file. Name the commands. Name the traps. Name the proof standard.
A useful template:
# Agent repo ritual
## Bootstrap
- Runtime:
- Install:
- Env vars:
- Services:
## Checks
- Fast local check:
- Full test:
- Typecheck/lint:
- Build/deploy:
## Change rules
- Do not edit:
- Generated by:
- Migration rule:
- Style rule:
## Recovery
- If install fails:
- If tests fail for environment reasons:
- If server already running:
## Evidence
- Final reply must include:
Then keep it honest. A stale ritual is worse than none because it creates false confidence.
Practical Takeaways For Jamie
For Jme-Loop, the paved path should not be "use a smarter agent." It should be:
prepare the repo so any competent agent can find the commands, run the checks, respect the rituals, and prove the change
This fits the steward-first direction. The steward should inspect each target repo before execution and ask:
- Is dependency installation deterministic?
- Are the repo's test and build commands explicit?
- Is there an agent instruction file?
- Are generated files and deploy rules named?
- Does the final proof require live or fixture verification?
The best target repos for field-testing are not necessarily the cleanest. They are the ones where a small amount of ritual-writing converts repeated agent confusion into reliable execution.
Key Terms
- Repo readiness: the degree to which a repository exposes the information an agent needs to bootstrap, modify, test, and prove work without maintainer intervention.
- Agent-computer interface: the tools, shell, file operations, prompts, and feedback channels through which an agent operates on software.
- Fail-to-pass tests: tests that fail before the target fix and pass after the correct patch, used by SWE-bench as the core grading signal.
- Verification loop: the cycle of making a change, running a check, observing output, and iterating until the check passes.
- Repo ritual: the project-specific commands, constraints, and proof standards that maintainers expect contributors to follow.
Recall Questions
- Why is the repository itself part of the agent's prompt?
- What are the six prepared-repo primitives?
- Why does a runnable test or build change the autonomy boundary?
- What does SWE-bench reveal about software-agent evaluation beyond model intelligence?
- When can instruction files become harmful instead of helpful?
- What proof should an agent leave after a code change?
Best Resources to Learn More
- Start with SWE-bench and SWE-bench Verified to understand real-repository coding-agent evaluation.
- Read the SWE-agent paper for the agent-computer-interface framing.
- Read Anthropic's Claude Code best practices for practical repo-ritual advice.
- Read METR's task-horizon work for the broader reliability lens.
- Compare those with your own repos: the fastest improvement is usually documenting the actual commands and checks, not changing models.
Sources
- Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan, "SWE-bench: Can Language Models Resolve Real-world Github Issues?" ICLR 2024. https://www.swebench.com/original.html and https://openreview.net/forum?id=VTF8yNQM66
- SWE-bench Verified overview, including the human-filtered 500-instance subset and bash-only mini-SWE-agent comparison. https://www.swebench.com/verified.html
- John Yang et al., "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering," arXiv:2405.15793. https://arxiv.org/abs/2405.15793
- Anthropic, "Best practices for Claude Code," especially the sections on verification and
CLAUDE.md. https://www.anthropic.com/engineering/claude-code-best-practices - METR, "Measuring AI Ability to Complete Long Software Tasks," March 2025. https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/
- GitHub Docs, "Continuous integration," for the standard build-and-test discipline that agent workflows inherit. https://docs.github.com/en/actions/automating-builds-and-tests/about-continuous-integration