Practice project: Build a Complete Agent Harness (Capstone)

Lecture 22. A Retry Can Do the Harm Twice

An operations agent triages overnight alerts. At 02:10 it decides a database alert needs a human, calls create_ticket, and the request goes out. The network drops before the reply arrives. The harness sees a timeout, does what harnesses do, and retries. The first request had already landed. Two identical tickets now sit in the on-call queue, a second engineer is paged for the same fault, and the two of them spend twenty minutes finding out they are working the same problem.

Nothing crashed. The retry logic worked exactly as designed. The log even shows the whole story in order: request, timeout, request again. What the log cannot show is the one fact the harness needed at the moment it retried, which is whether the first request had already changed the world.

Decide now: when a call times out and you cannot tell whether it landed, is "try again" the safe default, or is it the thing that causes the damage? Choose before reading on.

This lecture gives you a side-effect ledger: every action that leaves your repository, its retry rule, and the stop reason you will record when it fails. It serves the Implement stage of the Plan → Implement → Verify → Evolve loop, because it fixes how actions are allowed to repeat, and it feeds Verify, because a run that ends for the wrong reason must not be read as a run that succeeded.

What is actually going on

An action that leaves your repository passes through four moments, and they do not fail together. The harness requests the action. The remote system executes it. The harness observes a result. The outside world commits the change. A crash, a dropped connection, or a timeout can sit between any two of them.

The failure in the opening lives between the second and third moments. The ticket system executed and committed. The harness never observed. From the harness's point of view, the request and the absence of an answer are all it knows. A durable event log helps here, because after a restart the harness can see what it intended to do. Anthropic's managed agents design is built on this idea: the session log lives outside the harness, so a crashed harness can be restarted and resume from the last recorded event. That solves the memory of intent. It does not tell you whether the outside system acted.

A line worth keeping: the log remembers what you asked, not what the world did.

This is an old problem in networked systems. The HTTP standard defines the property that matters. In RFC 9110, section 9.2.2, a request method is idempotent if the intended effect of several identical requests on the server is the same as the effect of a single one. PUT and DELETE are defined that way, and POST is not. The reason the standard cares is exactly the situation above: a client that lost the response needs to know whether repeating the request is safe. An action built on the idempotent shape can be repeated blindly. An action built on the other shape cannot, unless you add something to make it safe.

The picture below follows the opening scenario with the four moments marked. Look for the point where the two timelines disagree.

The log is accurate and the harness is blind in the same instant. That is why the retry rule cannot live in the log. It has to live with the action.

The rule and its condition

Retry is a policy of the action, not of the harness. Repeat an action blindly only if repeating it has the same effect as doing it once. Otherwise carry a key the other side uses to ignore the duplicate, or read first to find out whether it already happened, or stop and ask a human.

Some actions are naturally safe. Setting a ticket's status to a fixed value gives the same result however many times you do it. Reading a record changes nothing. Writing a file with the same bytes leaves the same file. Others are not. Creating a ticket, sending an email, and charging a card do something new on every execution. A common fix is to attach an idempotency key: a unique id you generate for the intended operation and send with the request, so the other system can recognize a repeat and return the first result instead of acting again. Other mechanisms serve the same purpose: unique operation ids, dedupe records, transactional writes, or a read-before-retry check.

The flowchart below puts the questions in the order that costs least. Ask about idempotence first, because it decides whether anything more is needed.

The last branch matters as much as the first. When no cheap safe path exists, stopping and handing over is correct, and the ledger has a column that records which stop that was.

Worked case

Situation: the alert-triage agent from the opening, now being fixed. It has four actions that leave the repository: it sets an alert's status, creates a ticket, posts a summary to a chat channel, and pages the on-call engineer. The team has the incident log and one afternoon.

Decide now: which of the four can the harness retry blindly, and which need something added?

Take them in order. Setting an alert's status to acknowledged is idempotent: the same call repeated leaves the same state, so retry with backoff. Creating a ticket is not, but the ticket system accepts a client-supplied reference. The team adds a key built from the alert id and the incident window, so a second create_ticket with the same key returns the first ticket instead of making a new one. Posting a summary to chat has no key and no way to list recent posts, so the harness records the intent and, on an ambiguous timeout, stops and flags it for a human rather than risk a duplicate. Paging is the dangerous one, because a duplicate page wakes someone twice. The pager service supports a deduplication key, so the team reuses the alert id.

Illustrative composite. Modeled numbers for an operations agent, not a measured audit: over a month of alert handling, 31 timeouts occurred on ticket creation. Before the key, 9 of those produced a duplicate ticket. After the key, the same 31 timeouts produced none, because each retry returned the ticket created by the first attempt.

The second half of the fix is what the run records when it cannot finish. A run that hits its turn, time, or cost limit has stopped, not succeeded, and neither has a run that crashed or was blocked by policy. The team writes down the end states and keeps them separate: completed, died, budget_stopped, policy_blocked, and needs_human. A dashboard that collapses them into one "done" will tell the on-call engineer a cheerful story on the night a run died halfway through creating tickets.

The state diagram below shows the end states and what moves a run into each. Notice that only one of them means the work is finished.

This matches the discipline in Lecture 9: a run that says it is finished is a claim, and the stop reason is the evidence that separates a finished run from one that merely ended.

Faded case

Skip to the open case if this is familiar. A different team runs a billing agent with three outside actions: it reads an invoice, applies a credit to an account, and emails a receipt. The billing system accepts a client-supplied idempotency key for credits. The email service offers none and cannot list sent mail.

Your decision: fill in the ledger row for each action. Name the retry rule, and name the stop reason you would record if the action keeps failing.

Open case

Illustrative composite. A deployment agent publishes a package, then updates a status page, then posts to a release channel. The publish step is rejected with an error if the version already exists. The status page accepts a fixed status string. The channel post has no key. A run crashes after the publish response is lost but before the status page step.

Write down, in three lines, whether the publish can be retried and why, what the status page step does on restart, and what you would record for the channel post. The first quiz question below checks your answer.

Where this breaks

Three limits apply before you build a ledger for everything.

A general retry framework for every shell command is too wide. Most commands in an agent's day change only the repository, and version control already gives you undo for those. Apply the ledger to actions that leave the repository: messages, payments, tickets, deployments, third-party records. Anything else is better served by the git history you already have.

Read-before-retry can race. If another writer can create the same record between your read and your retry, the read proves nothing and a key is the only safe answer. Likewise, an idempotency key only works if the other side supports it. A key you send to a service that ignores it is a comment, not a protection. Test the behavior once, on purpose, before you rely on it.

Stop reasons are only as honest as the code that writes them. Anthropic's long-running agents post found that an agent would mark a feature done without testing it end to end, so an agent's own report of completion is weak evidence, and the same applies to its stop reason. The component that ends the run, the launcher, should write the stop reason, not the agent, so a model that wants to look finished cannot label itself completed.

Key Takeaways

• Treat requested, executed, observed, and committed as four moments that can fail apart.
• The log remembers what you asked, not what the world did.
• Retry blindly only when repeating has the same effect as doing it once.
• Give creates, sends, and charges a key, a read, or a human, never a bare retry.
• Name end states separately: a limit, a crash, and a block are not success.
• Let the launcher write the stop reason, not the agent.

What You Have Now

A side-effect ledger: every action that leaves your repository, its retry rule, and the stop reason you will record when it fails. Save it as harness/side-effects.md in your repository, with the end-state list beside it as harness/stop-reasons.md; the lab's checker looks for both at those paths. It serves the Implement stage of the Plan → Implement → Verify → Evolve loop by fixing how actions may repeat, and it feeds Verify, because it makes a run that ended badly look different from one that finished.

Example Files

side-effects.md

One row per action that leaves your repository. If you cannot fill a column, that is the finding.

# Side Effects

| Action | Idempotent? | Dedupe key | Retry rule | Stop reason on failure |
|---|---|---|---|---|
| Set alert status to acknowledged | Yes | Not needed | Retry with backoff | The call's own error with the attempt count |
| Create a ticket | No | alert id plus incident window | Retry with the same key | needs_human with the key and last known state |
| Post a summary to the chat channel | No | None available | Do not retry on an ambiguous failure | needs_human, record the intended text |
| Page the on-call engineer | No | alert id | Retry with the same key | needs_human, record that a page may have gone out |

stop-reasons.md

The end states your launcher may write. Keep them distinct, and let the launcher, not the agent, write them.

# Stop Reasons

- completed: the goal was verified by evidence outside the agent's own claim.
- died: the process exited non-zero.
- budget_stopped: a turn, time, or cost limit ended the run.
- policy_blocked: a guardrail refused an action and the run stopped.
- needs_human: a retry was unsafe or an approval was missing.

A failure is never written with the prefix "completed".

Further Reading

• RFC 9110: HTTP Semantics, section 9.2.2 Idempotent Methods — the standard definition of idempotent and why it matters when a response is lost.
• Scaling Managed Agents: Decoupling the brain from the hands — a durable session log so a crashed harness resumes from the last event, and a failed tool call returned to the model as an error.
• Effective harnesses for long-running agents — why a completion claim needs evidence outside the model's own statement.
• Harness design for long-running application development — long autonomous runs and the structured handoffs that carry state between agents.

Exercises

1. Find your outside actions. Search your agent's tools and scripts for anything that sends, posts, charges, creates, or publishes. List each with the service it talks to. That list is the first column of your ledger.
2. Test one key on purpose. Pick one action that claims to accept a dedupe key. Send the same request twice with the same key against a test account, and record whether the second call created a second effect. Do this before you write "retry with the same key".
3. Write the stop reasons. Create harness/stop-reasons.md with the five end states. Then look at how your launcher or wrapper records the end of a run today, and note every case where a failure is written as success.

Check your understanding