A long agent run produces far more text than a model can read in one call. This chapter shows how your code picks what to send, what to shrink, and what to leave behind a pointer.
The model knows only what you send
Suppose Patch ran the tests ten minutes ago and they all passed. Your code saved the result, but if it leaves that result out of the next context, the model does not know it. It may run the tests again, or even report that they still fail.
On any call, the model knows two things: what it learned in training, and the context. The context is the text your code sends with that one call. It holds the instructions, the tool list, the conversation so far, tool results, and any files or notes your code adds. Nothing else reaches the model.
What a context contains
Patch's next call sends these parts, from top to bottom:
┌──────────────────────────────────┐ ┐
│ System instructions │ │ the same
├──────────────────────────────────┤ │ on every call
│ Tool definitions │ │
├──────────────────────────────────┤ │
│ Agent instructions (AGENTS.md) │ │
├──────────────────────────────────┤ ┘
│ Retrieved memory, skills list │ ← set per task
├──────────────────────────────────┤
│ Conversation: user requests, │
│ model replies, tool calls, │ ← grows every step
│ tool outputs │
├──────────────────────────────────┤
│ Relevant files for this step │ ← chosen per call
├──────────────────────────────────┤
│ Latest user request │ ← new this turn
└──────────────────────────────────┘
↓
LLM → reply, in space kept free for it
The conversation grows with every step. Tool outputs, such as file contents and test logs, usually make up most of it. The newest material goes last.
Provider APIs fix part of this order, and you choose the rest. Putting stable parts first and changing parts last makes calls cheaper, as the section on prompt caching explains.
A lot of state, a small view
This chapter's examples use a later Patch that already has the memory, skills, and helper agent from chapters 5, 7, and 8. It holds far more than one call can carry:
What Patch has vs what one call sees (illustrative)
What Patch has
What one call sees
2,400 files in the repository
The one file being fixed
A session log of 48 steps, 87,300 tokens
Recent steps, plus a summary of older ones
The full output of every tool call, saved to files
Short previews and file paths
Hundreds of saved notes
The three that match this task
An agent can have enormous external state while exposing only a carefully selected subset to the model.
A pointer is a short reference to something large, such as a file path or the id of a saved tool output. It costs a few tokens. The model can fetch the full text with a tool when it needs it. Patch already does this with long tool outputs (chapter 2).
Windows and budgets
Patch's model can read at most 128,000 tokens in one call. Tokens are pieces of words. For example, a short common word is usually one token, and a long or rare word can be several. That limit is the context window. Patch's next call uses it like this (illustrative numbers):
window 128,000
kept free for the reply -16,000
left for the input 112,000
system prompt, tools, AGENTS.md, skills list,
memory, and a helper agent's report 7,800
the file being fixed and the new user message 1,660
history: 48 steps of about 1,820 tokens each 87,300
total input 96,760 (86% of 112,000)
room left: 15,240 tokens, enough for about 8 more steps
Your code subtracts the reply first, because on many APIs the reply has to fit in the same window as the input. If Patch loaded all 40 tools from the team's catalog instead of its 6, their definitions would add 23,600 tokens. Then the call would no longer fit.
Keep your budget well below that limit. The provider charges for every input token on every call, and your code re-sends the history each time (chapter 1).
Relevance: choosing what goes in
Early on, Patch needed tests/test_auth.py to reproduce the bug. Now that the fix is in, that file matters less. The team's changelog rule matters more, because Patch is about to wrap up. So the best context changes from step to step.
For each part, ask whether it will change the model's next decision. Recent steps, the plan, and the latest test result almost always pass that test, because they show where the work stands.
You can also give the model file paths and a search tool instead of loading every file that might help. This saves tokens but costs extra calls, and the model may not think to look. So many designs preload what almost every step needs.
Even a fact in the context can be missed, and length makes this worse. A July 2025 study of 18 models found that answers grew less reliable as the input got longer, even on simple tasks. On one test, a focused prompt of about 300 tokens beat the full prompt of about 113,000 tokens for every model tested.
Retrieval: fetching what is not there
When Patch starts a task, its code can search Patch's saved notes for anything about token expiry. Then it adds the best matches to the context. Choosing items from a large store like this is called retrieval, and it runs in five stages:
Query and filters→Candidates→Selection→Provenance check→Budgeted injection
Your code searches within the right scope, such as this project, by matching words or meaning. It keeps the few best matches that fit the budget, checks where each came from, and adds them with their source. A close match can still be wrong: a note about a retired service can match "database" and describe the wrong system. Chapter 5 covers the details.
Compaction: when the window fills
At step 48, Patch's window is 86% full. Old tool outputs, such as files it read an hour ago, take up 75,620 of those tokens. Patch rarely needs them word for word. So the cheapest fix is to swap each old output for a short placeholder that names the saved file:
before: {"tool_call_id": "call_9", "content": "FAILED tests/test_auth.py ... (1,400 tokens)"}
after: {"tool_call_id": "call_9", "content": "[output cleared: 1,400 tokens.
Full text in .patch/outputs/call_9.log]"}
Each tool call keeps its result, with the same id. Provider APIs generally expect every call to have a matching result. The model can also still see what it asked for. For Patch, the 42 old outputs shrink from 75,620 tokens to 1,260.
If that is not enough, your code asks the model to summarize the old steps. It swaps the summary in for them and keeps the newest steps word for word. The originals stay in the session log on disk:
before: 96,760 tokens, 86% of the 112,000 left for input
fixed parts, the current file, the new message 9,460
steps 1 to 42: calls and user messages 5,090
steps 1 to 42: tool outputs 75,620
steps 43 to 48: calls and outputs 6,590
after: 17,450 tokens, 16%
fixed parts, the current file, the new message 9,460
summary of steps 1 to 42 1,400
steps 43 to 48, word for word 6,590
session log on disk: all 48 steps, untouched
Replacing part of the context with something shorter is called compaction. Your code decides when it runs, usually at a threshold such as 80% of the window. It also decides what must survive: the goal, the user's rules, decisions with their reasons, file paths, and open questions.
Summarization: the risky step
Two summaries of the same 42 steps show the risk:
# careful
Goal: make tests/test_auth.py::test_login_expired_token pass.
User rule (step 14, word for word): "Don't edit anything in
tests/fixtures/. The billing suite shares those files."
Original failure (step 1): assert 200 == 401.
Ruled out: a clock-skew allowance in auth/tokens.py (reverted).
Full tool outputs: .patch/outputs/call_<n>.log
# careless
Patch looked into a failing auth test. It read many files, tried a
change in auth/tokens.py that did not help, and narrowed the problem
down to auth/session.py.
Both read well, but the careless one dropped the user's rule, the exact failure, and the paths to the saved outputs. Ten steps later a billing test fails, and Patch sees that editing a fixture file would make it pass. Nothing in its context says no. The session log still has the user's message, but that does not help. Your code keeps the log outside the repository, and Patch's tools can only reach files inside it.
The model writing a summary cannot know which detail a later step will need. And each new compaction can blur the previous summary further. Four habits help. Use a fixed template with headings such as goal, rules, decisions, and files. Copy the user's messages word for word. Keep exact names, paths, and error text. And keep hard rules in a part of state that your code adds to every call, so they never depend on a summary.
Prompt caching
Patch's next call starts with the same 94,830 tokens as its previous call. A provider can save the work it did on such a repeated start, called the prefix. When the next request begins with exactly the same tokens, the provider reuses that work. This is prompt caching, and the reused part costs less and comes back faster:
reused new input reply cost
first call of a session 0 96,760 1,000 101
next call (same start) 94,830 1,930 1,000 15
after an AGENTS.md edit 3,600 93,160 1,000 98
prices per 1,000 tokens (illustrative): reused 0.1, new 1, reply 4
all three calls send 96,760 input tokens
In a long run almost every call is like this, so caching can cut the input bill several times over.
Reuse needs an exact match, though. If you edit one early part, such as a rule in AGENTS.md, everything after it counts as new. Compaction does the same to the history. So put stable parts first, add new instructions at the end, and keep the tool list and the model the same within a session. If a rule really changes, change it anyway and accept one expensive call.
Details such as the minimum length and how long the cache lasts differ by provider. This description dates from October 2026.
Context pollution
Suppose Patch's context holds a dump of every file in tests/, including the fixtures the user said not to touch. It also holds the copy of auth/session.py that Patch read before fixing it, and forty lines from a failed attempt in auth/tokens.py. Each of these can pull the next step the wrong way. Text that misleads like this is called context pollution.
Pollution also includes contradictions and prompt injection. Prompt injection is text in a file, web page, or tool result written to look like an instruction. To prevent it, send one file instead of a folder, and drop old reads of a file after editing it. Replace a failed attempt with one line saying what was ruled out. Labels on untrusted text lower the risk of injection, but never rely on the context for security (chapter 10).
You can also move noisy work into a subagent, which is a second agent loop with its own context. It can read thirty files and send back a short report, so only the report enters Patch's context (chapter 7).
Build a context yourself
The widget below puts you in charge of Patch's next call. Try three things:
Turn off the helper agent's report, then the tools, and watch how the answers change.
Drag the history slider down until the user's rule from step 14 is no longer in view.
Open "Try compaction" and compact with a careless summary. Then check the question about tests/fixtures/.
Interactive widget. Without JavaScript: sending everything fills 86% of the space Patch has for input, and compacting brings that down to 16%. A careful summary keeps the user's rule about test fixtures. A careless one loses it, and no tool can reach the session log that still has it. A cached call costs about 15% of a first call, with the same number of tokens.
Why context is the central problem
When Patch breaks a rule, read the exact context of the failing call before you blame the model. If the rule was not there, the model never saw it. Memory, plans, skills, and subagent reports all work by adding text to the context. Permissions, checkpoints, and scheduling work in your code, where the model never sees them. So most choices about what an agent can do are choices about what it sees.
Running example: Patch gets a context builder
Patch version 4 builds each context instead of sending every message. Stable parts go first, old tool outputs become placeholders, and a short task block with the user's rules goes last. Near the limit, it compacts with a fixed template, checks the size again, and stops with an error if the call still does not fit. Chapter 5 gives it memory that outlasts a session.
# Patch v3: everything, every call
- context = [system_prompt()] + state["messages"]
# Patch v4: a context builder
+ RESERVE = 16_000 # room for the reply
+ def build_context(state, window):
+ budget = window - RESERVE
+ prefix = [system_prompt(), tools(), agents_md()] # stable, so it can be cached
+ task = render_task(state) # goal, user rules, files changed, last test result
+ ctx = prefix + [state["summary"]] + clear_old_outputs(state, keep_last=6) + [task]
+ if size(ctx) > 0.8 * budget:
+ state["summary"] = compact(state) # template, user messages word for word
+ for keep in (6, 3, 1): # still too big? keep fewer recent steps
+ ctx = prefix + [state["summary"]] + recent(state, keep) + [task]
+ if size(ctx) <= budget:
+ return ctx
+ raise ContextTooLarge(size(ctx), budget) # never send a call that cannot fit
+ return ctx # the session log is untouched
render_task is a few lines of your own code, not something the user types. It copies the goal and the user's rules, word for word, from the user's own messages. Then it adds files_changed and last_test from state. The result is a short block of text. Your code rebuilds it from state on every call, so compaction can never drop it. And because it goes last, it is the newest thing the model reads.
Design checkpoint 1: the expense agent
This checkpoint closes Part 1. Write your answers, then compare them with one reasonable design.
An expense agent for a 300-person company
A 300-person company wants an expense agent. Employees forward receipts by email or upload a photo. The agent reads the receipt, checks it against the travel and expense policy, files the expense in the finance system, and routes it for manager approval. Employees can also ask it questions such as "why was my expense rejected?"
Is this an agent, a workflow, or both? Which steps need a model at all?
List the tools and give each an effect class (read-only, reversible write, external or irreversible).
What state does one expense need, and where does each piece live?
What does the model see when it checks one receipt against policy? What stays out of its context?
How does a run end, and what evidence says it succeeded?
Reveal one reasonable design
Mostly a workflow: receive, extract, check, file, route. A model reads messy receipts, interprets policy in unusual cases, and answers questions. Ordinary code converts currency, finds duplicates, applies limits, and routes approvals.
submit_expense: external, because it enters the finance system and starts approval.
message_employee: external.
Per expense: the receipt and its hash, the extracted fields, the policy results, the draft and submission ids, a status, and messages. It all lives in a database row, not only in a context window.
The model sees the instructions, the extracted fields, the policy sections for this category and region, and the employee's role. It does not see the full policy, other people's data, or finance credentials.
A run ends when the expense is submitted or a needs-info message goes out. The evidence is a submission id from the finance system and passing checks, not the model saying "done".
What goes wrong
Too much context
Why it happens
Loading everything feels safe, but the relevant lines get buried, and every call pays for all of it.
How to spot it
Log the size of each part on every call, and look for big parts that rarely matter.
What to do
Give each part a budget, send pointers instead of whole files, and put large tool catalogs behind a search tool.
A summary drops a rule
Why it happens
The model writing the summary cannot know what later steps will need.
How to spot it
Check each new summary against the user's rules and the open questions.
What to do
Use a fixed template, copy user messages word for word, and keep hard rules in state that goes into every call. Chapter 15 goes deeper.
Silent cache misses
Why it happens
A timestamp or a reordered tool list near the top changes the prefix on every call.
How to spot it
Watch the share of reused input tokens in the provider's usage reports.
What to do
Keep stable parts first and identical from call to call, and add changes at the end.
You decide, AI builds
Decide yourself
What the model sees at each step, and the budget for each part.
What must survive compaction, and where hard rules live.
When to clear, compact, or start a fresh session.
Which content is untrusted, and how it is labeled.
Delegate to AI
Token counting and budget checks.
Placeholders, saved outputs, and trimming that keeps each tool call paired with its result.
Cache markers for a specific provider.
The summary call and its template.
Check yourself
Patch's session log holds the user's rule, but the latest summary dropped it. Can the model follow the rule on the next call? (Q4)
Yes, because your code still has it
No, unless something puts it back into the context
Yes, if prompt caching kept the earlier prefix
Answer
No. The model sees only this call's context, and Patch's tools cannot reach the session log, which lives outside the repository. Caching reuses work on text that is still sent, so it never adds anything.
Why does context management matter, and what happens when the context gets too large? (Q17, Q18)
Answer
The model can use only what is in its context, and every call has a size limit and a cost. So what you send decides what the model knows. Past the window, the provider rejects the call, and before that, calls get slower and more expensive. Runtimes respond by clearing old tool outputs, summarizing old history, fetching on demand, and moving noisy work into subagents.
Patch replaced an old test log with a placeholder that names the saved file. Can the model still get the log's details?
No, the details are gone
Yes, with one tool call, because the placeholder points to the file
Only if the provider cached the log
Answer
Yes. The placeholder is a pointer, and read_file can fetch the saved file. Without the path, nothing would point to the file, so the model would have to think of searching for it.
Takeaways
The model works only from the context your code sends. Every fact it needs must be there or one tool call away.
Choose what goes in at every step by asking whether each part will change the model's next decision.
When the window fills, clear old tool outputs first. Then summarize with a template that keeps the user's rules word for word.