Chapter 18 · Thinking like a designer
Model capability vs agent architecture
When an agent improves, the gain can come from the model or from a dozen other levers. Only a controlled comparison tells you which one.
Patch got better. Why?
Last month, Patch solved 62% of its 50-bug suite. This month it solves 74%. Three things changed in between. The provider shipped a new model version. A teammate added a run_tests tool that returns only the failing tests. And the compaction summary now keeps the last test output.
Everyone has a favorite explanation, but nobody has evidence yet. The answer matters, because each kind of gain has a different future:
- A model upgrade may need little implementation work, but it can change price, token use, and speed, so measure it like any other change. Your competitors get it too.
- Better architecture costs engineering time, and it may stop paying off as models improve.
- Some gains come from spending more tokens, so they show up on the bill.
Twelve levers, and how to test each one
Patch's 12-point jump could have come from any of twelve levers. Each lever changes one part of the loop, which tells you what to hold fixed when you test it.
| Lever | What it changes | How to test it |
|---|---|---|
| Better model | The weights (the numbers the model learned in training), so the same context gets different choices | Swap the model with everything else fixed, then retune the prompt and test again |
| More inference-time compute | Reasoning tokens, attempts, or steps spent per task | Raise the budget in steps, and plot success against cost |
| Better prompt | The instructions, including tool descriptions | Run old and new prompts on tasks you did not read while writing them |
| More context | How much the model sees: history, whole files, window size | Check failed traces for the missing fact, then vary the amount |
| Better retrieval | Which stored items reach the context | Score retrieval alone, then insert the right item by hand and see if the agent succeeds |
| Better memory | What carries over from one run to the next | Run a sequence of tasks with memory kept and with memory wiped |
| Better tools | Which actions exist, and what comes back | Remove the tool or restore the old one, and check that the model uses it |
| Better environment | Packages, test speed, error messages, repo docs | Sort failures into environment and agent failures. If the human's own fix also fails, the task is broken |
| Better verification | What counts as done | Score the checker against known outcomes: what bad work does it accept? |
| Better agent loop | Stop rules, budgets, retries | Sort runs by how they ended, and test the logic with a scripted model that gives fixed replies |
| Better planning | An explicit list of intended steps | Remove the planning tool, and split results by task length |
| Multiple agents | More loops, each with its own context and tools | Compare with one agent that gets the same model and the same total tokens |
More compute has two separate knobs. Suppose Patch keeps failing one hard bug. You can let the model reason longer inside each call, which raises the reasoning tokens per call. Or you can make more calls, for example three attempts at the fix, with the tests picking one that passes. Both knobs raise the total cost, so log reasoning tokens and calls separately, and compare designs at the same total.
A lever rarely changes only one thing. The extra changes that ride along are called confounds, because they could explain the gain instead. These are the ones that catch people most often.
Compute rides along with almost everything. Planning adds calls, extra agents and extra context add tokens, and a bigger step budget buys success directly. So compare designs at equal cost. Extra attempts also help only if a check can pick the one that worked.
Prompts and models depend on each other. A prompt tuned for one model can hurt another. A new tool brings a new description, so it is also a prompt change.
More context can hurt. Irrelevant text distracts the model. A bigger window also delays compaction, so the real gain may be fewer lossy summaries.
Memory breaks independent trials. A reset between trials removes the very effect you want to measure, so test memory on a sequence of tasks. Watch for memory that stores the answers to eval tasks.
Environment fixes change the test. When impossible tasks become solvable, the score rises with no change to the agent. Report those gains separately.
A checker cannot grade itself. If one check both stops the loop and scores the eval, the eval shares its blind spots. Self-checks are weak too: Anthropic found that "agents reliably skew positive when grading their own work".
Planning tools cost tokens. They should help long tasks and cost short ones, and a stronger model may plan well without one.
Parallel agents clash on shared work. Agents that edit the same code lose details and make conflicting choices. They do best on independent subtasks.
What gains from better models, and what needs architecture
A new model version arrives. Which parts of your agent get better on their own? These are tendencies to test, not laws.
A better model mostly improves judgment inside one call (Q29). That covers diagnosing a bug from evidence, writing correct code, choosing the next action, recovering from errors, and following long instructions.
Good architecture is needed whatever the model (Q30). That covers permissions and the security boundary, checkpoints and idempotency, the source of truth, budgets, checks against the real world, monitoring, and evaluation. For example, a smarter model does not make a retried payment safe. Only an idempotency key that the payment service checks does that.
Two hypotheses worth testing
First, features that make up for a model weakness fade as models improve. Deep Agents dropped its default todo tool after the benchmark above. In Anthropic's long-running harness work, Sonnet 4.5 needed context resets with handoff files, but with Opus 4.5 the author dropped them and relied on compaction. To test this on your tasks, run the old and new model, each with and without the feature. If the feature helps the new model less, retest it at every upgrade.
Second, gains from more agents are partly gains from more compute. Anthropic's multi-agent research system beat a single agent by 90.2% on its internal research eval. But multi-agent runs used about 15 times the tokens of a chat. On the BrowseComp evaluation, token usage alone explained 80% of the variance in performance. The model still mattered: a model upgrade gave a bigger gain than doubling the token budget. So compare designs at equal tokens.
Features that carry information or authority do not fade this way. A checkpoint, an approval gate, or your team's conventions file is not a workaround for a weak model.
How to attribute a gain
Back to Patch's jump from 62% to 74%. Asking which change caused it is a cause-and-effect question, so set it up like an experiment:
- Freeze the task set, and keep a held-out slice that you do not look at while tuning.
- Fix the budgets for model calls, tokens, cost, and time, and keep them equal across versions.
- Change one lever at a time: add it to the old version, or remove it from the new one (an ablation). If the two results disagree, the levers interact.
- Run paired, repeated trials on the same tasks, in the same window, from fresh environments.
- Compare task by task, and read the traces of the tasks that changed.
- Sort the failures into categories, before and after. The cause shows up as a shift between categories.
Chapter 16 covers the statistics behind steps 4 and 5.
| The trace shows | Suspect first |
|---|---|
| The needed fact never entered the context | Retrieval, more context, tools |
| The fact was there, and the model misread it | Model, prompt, compute |
| The right action did not exist, or failed with a vague error | Tools |
| A package was missing, a test was flaky, a build timed out | Environment |
| The run stopped early, looped, or died on an error | Loop, verification |
| It lost track of subtasks in a long run | Planning, compaction |
| It claimed done, and nothing caught it | Verification |
| It hit the budget while still making progress | Compute, budgets |
| It repeated a mistake from an earlier session | Memory |
Practice: what caused the gain?
Some of these cases have an answer. Others call for "insufficient evidence", plus the experiment that would settle it.
Pick an answer for each case.
Patch rose from 62% to 74% in the week that a new model version shipped and the run_tests tool landed. What caused the gain?
Answer
Two changes landed together, so a before-and-after number cannot separate them. The four combinations can.
A multi-agent research design beats a single agent by 9 points. It also uses four times the tokens. What can you conclude?
Answer
More agents means more compute. Until both designs spend the same, you cannot tell the design apart from the extra tokens.
You swap in the new model and change nothing else. Success rises by 9 points, and the traces show fewer misread test failures. What caused the gain?
Answer
The new model. Only the model changed, and the test was careful: 80 tasks, five trials each, run in the same window with the same budgets. The new model did better on 20 tasks and worse on 4, and the uncertainty range (+5 to +13 points) stays above zero. The traces also show how it helped.
Running example: which lever moved Patch?
The team reran each change on the same 50 bugs with fixed budgets, including all four combinations of model and tool. The run_tests tool added 9 points (uncertainty range +2 to +16). The new model added 3 points (−4 to +10), which this suite cannot tell apart from zero, so the team records it as unmeasured. The compaction fix left success flat but cut repeated test runs.
experiment = {
"tasks": "patch-bugs-v3", # 50 bugs, frozen
"trials": 3,
"budget": {"steps": 30, "usd": 3.00},
"arms": {
"baseline": {"model": "old", "run_tests": False},
"model": {"model": "new", "run_tests": False},
"tool": {"model": "old", "run_tests": True},
"both": {"model": "new", "run_tests": True},
"compaction": {"model": "old", "run_tests": False, "new_summary": True},
},
"order": "interleaved", # arms alternate, same afternoon
}What goes wrong
Crediting the last change
- Why it happens
- The newest or most visible change gets the credit, even when several changes landed, including a silent provider update.
- How to spot it
- More than one change landed between the two measurements.
- What to do
- Keep a change log next to every eval run, and test each change on its own.
Comparing at unequal cost
- Why it happens
- One design spends more tokens, steps, or attempts, so it buys its gain with compute. In one Anthropic test, a full harness built a playable game where a solo agent did not, but it cost over 20 times as much.
- How to spot it
- Cost per task differs between the two versions.
- What to do
- Keep budgets equal, or compare success against cost for each design.
You decide, AI builds
Decide yourself
- Which lever to try first, based on the failure categories in your traces.
- The frozen task set, the budgets, and what counts as a real gain.
- Which features make up for the current model, and when to retest them.
Delegate to AI
- The experiment runner: versions, interleaving, resets, and trial counts.
- Tagging traces with your failure categories, with spot checks by you.
- Intervals, per-task differences, and cost accounting across agents.
Check yourself
Which parts of an agent benefit primarily from better models? (Q29)
Answer
Judgment inside a call: diagnosing from evidence, writing code and arguments, choosing the next action, and recovering from errors. Even these depend on good context and tools.
Which parts require good systems architecture? (Q30)
Answer
Permissions and the security boundary, durability and idempotency, the source of truth, budgets, monitoring, and evaluation. A better model fixes none of these.
A feature adds 10 points with the old model and 1 point with the new one. What does that suggest?
Answer
The feature's value depends on the model. Check what it costs with the new model, and decide whether 1 point is worth it.
Takeaways
- Most levers also add compute or change the context on the side. So compare at equal budgets, and change one thing at a time.
- Attribute a gain with paired, repeated trials on a frozen task set. Then look in the traces for the cause.
- "Insufficient evidence" is a good answer when you can name the experiment that would settle it.
Going deeper
- How we built our multi-agent research system (Anthropic, June 2025). Token budgets, model choice, and when parallel agents pay off.
- Don't Build Multi-Agents (Cognition, June 2025). The case against parallel agents that make conflicting decisions.
- Harness design for long-running application development (Anthropic, March 2026). A separate evaluator agent, its cost, and a harness that shrinks as models improve.