Long-running agents

Interactive agents and durable agents

Picture two crashes. A coding agent is fixing a bug while you watch, and your laptop restarts, so you start it again. A job that emails a report every night crashes at 2 a.m., right after sending the email but before saving that it did. Nobody is watching, so the restarted job sends the email again.

The first agent is interactive: a person watches it and handles its failures. The second must be durable, because it handles failures alone. Running for a long time is not the test. A durable agent keeps three promises after a failure:

• It keeps its progress after a crash.
• It does not repeat side effects, which are changes outside the agent, such as an email or a payment. If it is unsure whether one happened, it checks before trying again.
• It can wait for days with no program running.

Checkpoints

Take the nightly report job. It fetches data from three sources, asks the model for a summary, writes a report, and emails it. If it crashes during the third fetch, it should not repeat the first two. So it saves its progress after each step.

That saved progress is a checkpoint: a copy of the run's state, written to disk after each step. Here is the loop from chapter 0 with a load, a limit check, and a save added:

def run(task_id):
    state = store.load(task_id) or new_state(task_id)    # continue, or start fresh
    while state.status == "running":
        if state.cancelled or state.step >= state.max_steps:
            state.status = "stopped"                     # check the limits first
        else:
            reply = model(build_context(state))
            state.messages.append(reply)
            for call in reply.tool_calls:
                state.messages.append(execute(call))
            state.step += 1
            if not reply.tool_calls:                     # the model says it is done
                if check_success(state):
                    state.status = "done"
                else:
                    state.messages.append(note("The check failed. Keep going."))
        store.save(task_id, state)                       # the checkpoint, final status included

A crash between two saves means the unsaved step runs again. For a fetch, that wastes time, but for an email it can mean a second copy (see idempotency).

What to save

A new worker that takes over must know how far the old one got. Did it already try to send the email? So a durable agent saves these things outside the program:

• A run record: the task, its status, which worker has it, the attempt count, and the code and prompt version.
• The checkpoint.
• A record of each side effect and how far it got.
• Large outputs as files, referred to by path.
• The original request, so a rerun works on the same task.

The rule behind all of these is to save before you act. Write "about to send the email" before sending, and save the reply after. A crash in between then leaves a record saying the email may have gone out.

Queues and workers

Now the team wants reports for 40 customers every night. A scheduler puts 40 tasks in a queue, and a few workers take them one at a time. A worker is a process that takes a task, runs the agent loop for it, and records the result.

Each worker holds a lease, so that a dead worker's task is not lost. A lease is a claim that runs out unless renewed, such as "worker 3 owns task 17 until 02:05". A live worker renews it every minute with a short message called a heartbeat. When a worker dies, the heartbeats stop and the lease runs out. Then another worker takes over from the checkpoint.

while True:
    task = queue.claim(lease_seconds=300)     # other workers skip it for 5 minutes
    if task is None:
        sleep(5)
        continue
    with heartbeat(task, every=60):           # renew the lease while working
        try:
            run(task.id)                      # continues from its checkpoint
            queue.complete(task)
        except Exception as err:
            queue.fail(task, err)             # try again later, or give up after N tries

Leases have a price: a task can run twice. A worker might freeze without dying, lose its lease, and wake up after a second worker took over. Engineers call this at-least-once execution: each task runs at least once, and sometimes more.

A fencing token stops the frozen worker from damaging your database. Each new lease gets a higher number, and the database refuses saves that carry an older one. So when the frozen worker wakes up with lease 6 after lease 7 was issued, its next save fails and it stops. It may already have sent an email, though, so side effects still need the receiver's help.

Retries

The report job calls a data API, and the call times out. Should it try again? Only if the failure is temporary, such as a timeout or a rate limit. A permanent failure, such as a refused permission, goes back to the model as a tool result. Choosing the next move is the model's job (chapter 2).

def call_with_retry(fn, attempts=4, base=2.0):
    for n in range(attempts):
        try:
            return fn()
        except Temporary:                        # timeout, 429, 503
            if n == attempts - 1:
                raise                            # out of tries: fail the step
            sleep(base * 2**n * random.uniform(0.5, 1.5))   # backoff with jitter
    # permanent errors are not caught here; they go back to the model

Each retry waits about twice as long as the last, which is called backoff. It also adds a random delay, called jitter, so a thousand workers that failed together do not retry at the same moment.

A timeout means only that no answer came back, and the other side may have done the work. So retry reads freely, but give every send an idempotency key.

Schedules and events

The report job runs every night at 02:00, and a triage agent starts whenever a build fails. Each trigger should only add a task to the queue, so every run gets the same checkpoints and retries. Decide three rules for each trigger:

• Missed runs: if the system was down at 02:00, run once later, run every missed time, or skip?
• Overlap: if last night's run is still going, start another, skip, or wait?
• Repeats: event sources often deliver the same event twice, so check its id before adding a task.

Waiting works the same way. To check back on Monday, the agent saves a wake-up time instead of keeping a program asleep. A run that waits for approval saves itself and exits until the answer comes (chapter 11).

Resuming a run

A new worker loads the checkpoint and checks where the old worker stopped. Between steps, it starts the next step, and during a fetch, it fetches again. During the send, nobody knows whether the email went out.

Record and replay

Saving after each step still loses the step in progress. Durable execution goes further: it saves every model reply and tool result the moment it arrives. After a crash, your code reruns the steps, but each saved call returns its saved result at once. Only unsaved work makes real calls.

def step(state, n):
    reply = state.journal.get(("model", n))
    if reply is None:                                  # not saved yet: real work
        reply = model(build_context(state))
        state.journal.put(("model", n), reply)         # save the decision first
    state.messages.append(reply)
    for call in reply.tool_calls:
        result = state.journal.get(("tool", call.id))
        if result is None:
            result = execute(call)                     # side effects need keys, below
            state.journal.put(("tool", call.id), result)
        state.messages.append(result)

Why not call the model again? Because it may answer differently, and the new answer could contradict what already happened.

Idempotency: making repeats harmless

Back to 02:06: the worker sent the email, then crashed before saving the reply. The new worker finds no record of the send, so if it sends again, the team gets two reports.

An action is idempotent if doing it twice has the same effect as doing it once. Reading a file is idempotent, and so is "set the ticket to closed". Sending an email is not.

Queues, retries, and restarts all repeat work, so a non-idempotent action will eventually happen twice. You cannot prevent this on your side alone. There is always a moment when the email is sent but your record does not say so yet. The fix needs the receiving service's help, through an idempotency key: a unique id for one action, such as op-8c41.

1. Your code creates the key and saves it before the first attempt.
2. It sends the same key with every attempt, including after a restart.
3. The service remembers keys for a while, say 24 hours. A repeated key inside that window gets the first result back, and nothing happens again.

Your code also tracks each action through four states:

op = journal.get("send-report")
if op is None:
    op = journal.add("send-report", key=new_key(), state="intended")
if op.state == "confirmed":
    return op.result                          # it finished before the crash
if op.state == "attempted":                    # sent, but no reply was saved
    op = journal.update(op, state="outcome unknown")
if op.state == "outcome unknown" and not safe_to_retry(op):
    return reconcile(op)                       # ask the service or a person
if op.state == "intended":
    journal.update(op, state="attempted")      # saved before the first send
reply = email_service.send(report, idempotency_key=op.key)
return journal.update(op, state="confirmed", result=reply.id).result

When a retry is not safe, your code must reconcile, which means finding out what happened before acting. For example, it asks the email service whether it sent op-8c41, or asks a person to check the inbox.

A key made up at send time is lost in a crash. The retry then carries a new key and looks like a new email. Where you can, build the key from what the action means, like weekly-report:2026-10-19:team, so it survives even a lost checkpoint.

Crash it yourself

The widget runs the nightly report job. Start with the defaults: save nothing, and crash right after the email goes out. Step to the end, then change one setting at a time:

1. Save a checkpoint after each step. The fetches stop repeating, but the email still goes out twice.
2. Add the key saved before sending. When the new worker finds the email marked attempted, send again with the same key.
3. Under More settings, choose a service that sends every request, and send again. Then try asking the service instead.
4. Switch back, and start the new worker 30 hours later.

Why this is distributed systems

With a queue, workers, retries, a scheduler, and a remote email service, the report job is now a distributed system, with the usual problems:

Agents add two problems of their own. A model may answer differently each time, so a restart must replay saved replies. And the model knows nothing about crashes, leases, or keys, so your code and the services around it must provide every guarantee.

Running example: Patch works the night shift

Patch now runs at night: a worker claims each queued bug with a lease, checkpoints every step, and retries temporary errors. Opening a pull request still waits for a person (chapter 11), so the run saves itself and exits until morning. Each issue gets one branch, and Patch counts on the Git host to refuse a second open pull request for that branch. Check that your host really enforces this before you rely on it. If it does, two racing workers cannot open two.

# Patch v11: the whole run lives in one process
- run_agent(task, tools, max_steps=30)
# Patch v12: queued, checkpointed, safe to repeat
+ task = queue.claim(lease_seconds=600)
+ state = store.load(task.id) or new_state(task)
+ run_agent(state, tools, checkpoint=store.save, retries=4)   # exits while waiting
+ branch = f"patch/issue-{task.issue}"     # one branch per issue
+ pr = git_host.open_pr(branch=branch)    # the host must refuse a second open PR for this branch
+ # outcome unknown, such as a timeout? look the branch up before trying again

What goes wrong

You decide, AI builds

Check yourself

Takeaways

• Decide up front which failures each agent must survive: crashes, restarts, long waits, and repeated deliveries.
• Assume every task can run twice, because leases and retries allow it, and make each side effect safe to repeat.
• For every side effect, find out whether the receiving service ignores repeated keys, and for how long. If not, plan how to check what happened before retrying.

Going deeper

• Effective harnesses for long-running agents (Anthropic, 2025).
• What is idempotency? And why it matters for durable systems (Temporal, 2024).
• Scaling Managed Agents (Anthropic, 2026).