Practice project: Build a Complete Agent Harness (Capstone)

Lecture 21. A Summary Is Not the Session

On Monday an agent spends four hours porting a test suite to a new runner. Early on it discovers that the tests only pass when they run one at a time, because two of them share a fixture database and corrupt each other in parallel. It writes the flag into a command, notes the reason in a scratch comment, and moves on. By mid-afternoon the conversation is long, so the harness compacts it. The summary reads well: runner configured, forty suites ported, six remaining, CI config updated.

On Tuesday the agent picks up from that summary. The flag is gone, and so is the reason for it. It runs the suite in parallel, sees seven flaky failures, and spends two hours "fixing" tests that were never broken. The fixture collision is rediscovered on Wednesday, by a different session, at the same price.

Decide now: when the agent sounded fully informed on Tuesday morning and was still missing the one fact that mattered, where did that fact live on Monday, and what should have kept it reachable?

This lecture gives you a context surface map: what lives in the durable log, what enters each model call, and what a summary is allowed to drop. It serves the Plan stage of the Plan → Implement → Verify → Evolve loop, because deciding what survives is a planning choice, and it feeds Evolve, because every dropped fact you rediscover is a rule your harness has not learned yet.

What is actually going on

Two different things get called "memory" in an agent, and confusing them is the root of this failure. One is the durable record: everything that happened, stored outside the model, in order, whether or not the model will ever see it again. The other is the active context: the specific set of tokens the model reads on the next call. The harness sits between them and decides, on every call, which slice of the record to show.

Neither contains the other. The record can hold a hundred thousand lines the next call never sees. The window can hold things that were never stored at all, such as the tool schemas loaded for this turn, a runtime note about the time, or the instruction you just typed. Anthropic's managed agents design keeps an append-only session log outside the model's window for exactly this reason: the harness can fetch slices of history and transform them before they reach the model, so losing something from the window does not mean losing it for good.

A line worth keeping: a summary is a bet on what you will not need.

Compaction is the usual way a window stays small. When a conversation nears its limit, the harness asks the model to summarize it and starts a new window from the summary. Anthropic's context engineering guide describes the approach used in Claude Code: it keeps architectural decisions, unresolved bugs, and implementation details, discards redundant tool output, and continues with the summary plus the five most recently accessed files. That is a good policy and it is still a transform. A transform that shortens cannot keep everything, so something that looked unimportant when it ran can be exactly the fact a later step needs.

Look at where the arrows go in the picture below. The record only grows, the window is rebuilt on every call, and the only way a fact crosses from one to the other is through the harness's slicing policy. Anything the policy does not select, and anything you did not write down, is gone from the next call.

The shape explains the Tuesday failure. The fixture fact was in the record on Monday, because everything is. It was dropped from the window by the summary, and nothing pointed the agent back to it.

The rule and its condition

Summarize only what you can lose, and keep a pointer to everything else. The condition is recovery: a summary is safe to take when the dropped detail can be found again by something the agent will actually do, such as open a named file or search an indexed log. If the detail is only in the old window, the summary has quietly deleted it.

Two tools get confused here, and the difference matters. Compaction shortens the conversation in place so the same agent keeps going. A reset clears the window completely and starts a fresh agent from a structured handoff. Anthropic's harness design write-up draws the line plainly: compaction preserves continuity but does not give a clean slate, so a model that tends to wrap up early near its window limit can keep doing so. A reset gives that clean slate, at the cost of needing a handoff file that holds enough state for the next agent to continue.

The same write-up reports that context resets became essential with one model that wrapped up early. Anthropic's managed agents post adds that when the same harness ran on a later model the behavior was gone, and the resets had become dead weight. Treat the reset as a bet about one model's behavior. The handoff file stays useful either way, because it is the thing that makes a restart cheap.

The flowchart below turns the table into an order of questions. Ask about recovery first, because it decides whether the summary is allowed to drop an item at all.

Notice that the last two branches end in a pointer, not a delete. The only items that go quietly are the ones that are recoverable and not needed verbatim.

Worked case

Situation: Anthropic published a post-mortem on recent Claude Code quality reports, in which the API and inference layer were unchanged while three product-layer changes made the agent feel worse. One of the three is a clean example of this lecture's failure, and it came from a change meant to save money.

The intent was reasonable. When a session had been idle for over an hour, the prompt cache would be cold anyway, so the harness could drop old reasoning from the next request to reduce the number of uncached tokens it had to send, then go back to sending full history. The bug was in the last step. Instead of clearing once, the flag stayed set, and every later request in that session kept only the most recent block of reasoning and discarded everything before it. As the post-mortem describes it, Claude would continue executing, but increasingly without memory of why it had chosen to do what it was doing. Users reported forgetfulness, repetition, and odd tool choices, and the constant cache misses were believed to be what drained usage limits faster.

Decide now: this change passed code review, unit tests, end-to-end tests, and internal dogfooding. What made it invisible to all of them, and what would you add to your own harness so the same shape of bug shows up on day one?

The reveal has three parts. First, the failure only appeared in a corner case, stale sessions, and two unrelated experiments hid it in most sessions the team used, so ordinary testing never exercised it. Per the post-mortem it took over a week to find and confirm the root cause. Second, the damage was not a crash. It was a slow loss of information that looks, from outside, like a model that got worse, which is why quality reports pointed at the model first.

Third, the fix pattern for your own system is small. Write an event every time the harness removes something from the window, with what was removed and why. Assert that a clearing that should happen once happens once per idle gap. Test the stale path on purpose, with a fixture that ages a session past the threshold and checks that turn five still has what turn one put in. And keep a pointer: if the window drops reasoning or tool output, the record still holds it, and the agent has a path back.

The sequence below shows the intended behavior against the bug. Look for the point where the flag stays set.

Illustrative composite. A team adds a rule to their own harness: when a session is idle for 45 minutes, drop tool output older than 10 turns from the next request. They add a counter, window_trim_events, per session. In a staged test with a session aged by hand, the counter reads 1 after the first resumed turn and 1 after the tenth. In a version with the same bug as above, the counter would read 10 after the tenth turn. The check costs one line in the test and catches the whole class.

Faded case

Skip to the open case if this is familiar. A different team runs a long-lived refactoring agent. It compacts at a fixed size and keeps a NOTES.md that the agent reads at the start of every session. The map they drew has five rows: the original task, decisions made, the list of changed files, test output, and a constraint the agent learned about the build system. They have to decide which of the five a summary may drop.

Your decision: mark each of the five as safe to summarize or not, and say where each one is recovered from. Pay attention to the constraint and the test output.

Open case

Illustrative composite. A deployment agent works across three sessions on a service migration. Session one learns that the staging database must be warmed before the health check passes, and mentions it in a message. Session two is a reset with a handoff file that lists steps completed and next steps. Session three runs the health check, sees it fail, and rolls back the deploy.

Write down, in three lines, what the handoff file was missing, where the warm-up fact should have lived, and which row of a context surface map would have named it. The first quiz question below checks your answer.

Where this breaks

Three limits apply before you build the map for everything.

Summarizing everything is cheaper, and for a short chat it is enough. The failure shows up on the second session, not the first, and only for facts a later step needs verbatim. If your agent runs one task in one window, a context surface map is more bookkeeping than the work deserves. The map earns its place when sessions span hours or days.

A pointer is only as good as what it points to. A log that cannot be searched, or a notes file that grows without structure, turns "recoverable" into a claim nobody can act on. Check the pointer by asking a fresh session to find the fact, with no help from you. If it cannot, the row says "safe to summarize" and the truth is the opposite.

A reset needs a handoff that can carry the whole job. The harness design write-up names the cost directly: a clean slate only helps if the handoff artifact has enough state for the next agent to pick up cleanly. A thin handoff is a dropped summary with extra steps. Lecture 12 covers how to write one, and Lecture 5 covers how to keep context alive between sessions; this lecture adds the question of what is allowed to be lost.

Key Takeaways

• Treat the durable record and the active window as two surfaces; neither contains the other.
• A summary is a bet on what you will not need.
• Write constraints, decisions, and flags to a file before any compaction runs.
• Choose compaction or a reset by how the model behaves near its limit, and revisit when the model changes.
• Log every time the harness trims the window, and test the stale-session path on purpose.
• Verify each recovery pointer by asking a fresh session to find the fact unaided.

What You Have Now

A context surface map: what lives in the durable log, what enters each model call, and what a summary is allowed to drop. Save it as harness/context-surfaces.md in your repository; the lab's checker looks for that file at that path. It serves the Plan stage of the Plan → Implement → Verify → Evolve loop by deciding what survives, and it feeds Evolve, because each fact you find was lost becomes a row you add.

Example Files

context-surfaces.md

One row per kind of thing your agent works with. Fill the last column with a path or a query a fresh session can actually run.

# Context Surfaces

| Item | Lives in | Enters the window when | Safe to summarize? | Where to recover it |
|---|---|---|---|---|
| Original task | NOTES.md | Start of every session | No, keep verbatim | NOTES.md section "Task" |
| Decisions and constraints | NOTES.md | Start of every session | No, write before any summary | NOTES.md section "Decisions" |
| Changed files | Version control | When the agent asks for the diff | Yes, pointer only | git diff against the base branch |
| Test and build output | Run log | When a failing command is rerun | Yes, if the command is recorded | The command in NOTES.md "How to run" |
| Tool schemas | Harness config | Every call | Not applicable, never stored in the record | Harness config file |

trim-event.json

One line the harness writes whenever it removes something from the window, so "once" and "every turn" look different.

{
  "event": "window_trim",
  "session": "s-2041",
  "removed": "reasoning blocks older than the latest",
  "reason": "idle_over_threshold",
  "count_this_session": 1
}

Further Reading

• Effective context engineering for AI agents — the compaction policy, structured note-taking, and sub-agents with clean windows.
• Harness design for long-running application development — the line between compaction and a reset, and why a reset needs a real handoff.
• An update on recent Claude Code quality reports — the idle-session bug told with its symptoms, why tests missed it, and the fixes.
• Effective harnesses for long-running agents — why compaction alone does not always pass clear instructions to the next session.

Exercises

1. Draw your two surfaces. For one agent you run, list what is stored outside the model and what enters the window on a typical call. Mark anything in the window that is not stored anywhere. That list is the first draft of your map.
2. Find a dropped fact. Look through your last few long sessions for a case where the agent rediscovered something it had already learned. Write the fact, where it lived when it was first learned, and the row that would have saved the second discovery.
3. Test a pointer cold. Start a fresh session with no history. Ask it to find one fact from your map using only the "Where to recover it" column. If it fails, fix the column, not the question.

Check your understanding