Practice project: Build a Complete Agent Harness (Capstone)

Lecture 23. Old Scaffolding Can Slow a New Model

Last spring a team built a harness around a model that tended to wrap up early. They added a context reset every forty minutes, a second agent that reviews each chunk of work, and a long system-prompt line telling the model to keep its replies short. Quality went up, and everyone was pleased. Then the model was upgraded. Nobody touched the harness. Runs got slower, bills got bigger, and the reviewer agent started rejecting work the new model had done correctly, because it was checking for mistakes the new model no longer made.

Three layers of machinery now sat between the model and the work, each one added to fix a problem that might not exist anymore. The team had no list of what each layer was for, what it cost, or when anyone last checked that it still helped. Removing a layer felt dangerous. Keeping them all was costing money every night.

Decide now: when the model changes underneath a harness that works, do you keep every layer because it was added for a reason, or strip layers because the model is better? Pick one before reading on.

This lecture gives you a scaffold ledger: each harness piece, the limit it bets on, what it costs, and the date you will test it again. It serves the Evolve stage of the Plan → Implement → Verify → Evolve loop, because it is how a harness gets smaller as well as bigger.

What is actually going on

Every piece of a harness is a bet. A context reset bets that the model gets worse as its window fills. A second reviewing agent bets that the first agent's own checking is not reliable enough. A brevity line in the prompt bets that shorter output costs nothing. Each bet was reasonable when it was placed, against one model on one set of tasks.

Bets go stale when the thing they were about changes. Anthropic's Scaling Managed Agents post states this plainly: a harness encodes assumptions about what the model cannot do on its own, and those assumptions need to be questioned often because they can go stale as models improve. Its example is a context reset added for a model that wrapped up work early as it sensed its context limit approaching. When the same harness ran on the next model, the behavior was gone and the resets had become dead weight.

A line worth keeping: a scaffold is a bet with an expiry date.

Staleness has a second cause that has nothing to do with the model. The harness itself changes: the default effort setting, the system prompt, what the product keeps in the window. Anthropic's post on recent Claude Code quality reports describes three such product-layer changes that made the agent feel worse while the API and inference layer stayed the same. One of them was a system-prompt line limiting how much text the agent wrote between tool calls. A broader ablation later showed a drop of 3% on one evaluation for two models, and the line was reverted. The takeaway for your own system is the discipline that followed: per-model evals for every prompt change, an ablation of each line, and gradual rollouts.

Look at the picture below. A piece of scaffolding does not go from "added" to "forever." It becomes suspect when its inputs change, and it has to be tested to get out of that state.

The loop through Suspect and Ablated is the work of this lecture. Most teams skip it because a piece that was proven once feels proven for good.

The rule and its condition

Keep a scaffold piece only while it earns both a quality lift and its cost on the model and tasks you actually serve, and re-test it whenever the model, the effort setting, the prompt, or the window policy changes. Test by removing one piece at a time.

One at a time matters. Anthropic's harness design write-up describes a first attempt to simplify that cut the harness back radically and could not match the original, and it became hard to tell which pieces were load-bearing. The author then moved to a methodical approach, removing one component at a time and reviewing the impact. The same write-up found that the extra evaluator agent was not a fixed yes-or-no: it was worth its cost when the task sat beyond what the current model does reliably on its own.

There is a measurement trap under all of this. Agent scores move with the machine as well as the harness. Anthropic's infrastructure noise study found a 6 percentage point gap between the most and least resourced setups on one benchmark, and recommends skepticism toward leaderboard gaps under 3 percentage points until the configuration is documented and matched. So an ablation must run on a fixed set of tasks, on the same machine settings, or you will measure the box and credit the change.

The flowchart below is the loop you run. It starts from a trigger, never from a feeling that the harness is bloated.

Removal ends with a proposal, not a deletion. A loop that is graded on the same harness it edits must not delete its own guardrails; that decision belongs to a person, as Loop Engineering's locked-scorer lecture argued.

Worked case

Situation: the team from the opening. After the model upgrade they have three pieces: a context reset every forty minutes, a reviewer agent on each chunk of work, and a brevity line in the system prompt. They have a fixed set of thirty tasks drawn from real work, the same machine settings for every run, and a rule that nothing is deleted without an owner decision.

Decide now: which piece do you test first, and what result would make you keep it?

Test the reset first. It is the piece whose bet is about the model, and the managed-agents post describes that exact bet going stale. Run the thirty tasks three ways on the new model: the baseline harness, the harness with the reset removed, and a repeat of the baseline to measure how much the score wobbles by itself. The repeat matters, because without a noise floor a one-point difference means nothing.

Illustrative composite. Modeled numbers for a team's scaffold test, not a measured audit: the baseline scores 22 of 30 tasks, and a repeat of the baseline scores 23, so the wobble is about one task. With the reset removed the harness scores 23, and the run takes 31 minutes instead of 44. The difference is inside the wobble and the run is faster, so the verdict is to remove the reset for the new model, keep it available behind a flag for the old model, and set a re-test date.

Now the reviewer agent. The evidence from the published write-up is that an extra evaluator pays when the task sits beyond what the model does reliably alone. So the test is not "does the reviewer help overall" but "on which of the thirty tasks does the model alone fail". If it fails on a handful, the reviewer earns its cost on those and not on the rest, and the shrink-or-gate verdict applies: run it only on tasks above a difficulty line. The cost framing matters too. The same write-up reports one task where a solo run took 20 minutes and cost $9 while the full harness took 6 hours and cost $200, over twenty times the price for a clearly better result, and the source says plainly that this is one task, not a law; see the harness design write-up.

The last piece is the brevity line, and here the lesson is about the order of operations. Do not judge it by how the output reads. Run the fixed tasks with and without the line and compare the score, because the published case shows a line that looked harmless costing 3% on one broader evaluation for two models, per the quality-report post. If you ship the change, roll it out gradually and keep the old behavior one flag away.

The ledger records each verdict with a next re-test date, so the question of whether a piece still pays is answered by a date and a number, not by memory.

Faded case

Skip to the open case if this is familiar. A team runs a refactoring agent with an extra planner agent in front of it, and a second verifier at the end. The task mix is mostly short, sequential edits: rename a function, update its callers, run the tests. They have the thirty-task set. A test shows the planner leaves the score within noise and adds time, and the verifier catches mistakes on four of the thirty tasks.

Your decision: what verdict do you write for the planner and for the verifier, and what do you say about adding more agents to this setup?

Open case

Illustrative composite. A team's harness has a retrieval step that pulls ten documents into every prompt, a long list of tool-use rules in the system prompt, and a retry wrapper around every call. A new model is released that handles long tool lists and long documents better. The team has no fixed task set and no record of why any of the three was added.

Write down, in three lines, what you would build before testing anything, which piece you would test first, and what you would record for each verdict. The first quiz question below checks your answer.

Where this breaks

Three limits apply before you build a ledger for everything.

Stripping a harness because a new model exists, with no before-and-after on your tasks, is the same mistake as keeping every layer forever. Both skip the measurement. A model that is better on a public benchmark may still be worse on your task shape, and a piece you remove on a hunch can bring back a failure you stopped seeing because the piece was hiding it.

A lift that is real on one task may not transfer. The published cost and quality numbers come from specific experiments, and the sources say so. Treat them as examples of how large the effects can be, not as constants to plan around. Multi-agent results show the same limit: Google Research's study of 180 configurations found centralized coordination improved a parallelizable finance task by 80.9% over a single agent, while every multi-agent variant degraded a sequential planning task by 39% to 70%, and independent agents with no cross-checking amplified errors 17.2x against 4.4x with a central orchestrator; see the Google Research write-up. Match the extra machinery to the shape of the work.

Ablation costs tokens and needs a fixed corpus, and the loop that is graded must not be the one that deletes its own guardrails. Run the test, write the verdict, and send removal to a person. That is also why the ledger has a column for the next re-test date: the cheapest ablation is the one scheduled in advance.

The diagram is a starting hypothesis, not a rule. Your fixed task set is what turns it into a verdict for your system.

Key Takeaways

• Treat every harness piece as a bet about a limit that can expire.
• A scaffold is a bet with an expiry date.
• Re-test when the model, effort, prompt, or window policy changes.
• Remove one piece at a time, on a fixed task set and the same machine settings.
• Repeat the baseline so you know how much the score wobbles by itself.
• Send removal to an owner decision; the graded loop never deletes its own guardrails.

What You Have Now

A scaffold ledger: each harness piece, the limit it bets on, what it costs, and the date you will test it again. Save it as harness/scaffold-ledger.md in your repository, with the fixed task list beside it as harness/ablation-tasks.md; the lab's checker looks for both at those paths. It serves the Evolve stage of the Plan → Implement → Verify → Evolve loop by making the harness shrink when the evidence says it should, and it reuses the incident-to-rule habit from Lecture 19 in the other direction: rules you add get an expiry check, too.

Example Files

scaffold-ledger.md

One row per piece of harness machinery. The Re-test when column is a date or a trigger, never "later".

# Scaffold Ledger

| Piece | Limit it bets on | Measured lift | Cost | Re-test when | Verdict |
|---|---|---|---|---|---|
| Context reset every 40 minutes | Model wraps up early near its window limit | None beyond noise on the new model | Extra orchestration, about 13 minutes per run | Next model change | Gate to the old model, propose removal |
| Reviewer agent on each chunk | Model's own checks miss errors | Catches errors on 4 of 30 fixed tasks | One extra agent run per chunk | Next model change or monthly | Keep, run only on tasks above the difficulty line |
| Brevity line in the system prompt | Shorter replies cost nothing | Unmeasured | None visible, quality risk | Before the next prompt change | Remove until a fixed-task run shows no loss |
| Retry wrapper on every call | Transient network errors | Fewer failed runs | Hides repeated failures | Next incident | Keep, but apply the Lecture 22 retry rule per action |

ablation-tasks.md

The fixed corpus. Draw it from real work, pin it, and run it the same way every time.

# Ablation Tasks

- Corpus size: 30 tasks drawn from the last two months of real work.
- Machine settings: same CPU, memory, and timeout for every run. Record them here.
- Runs per setting: baseline, one-piece-removed, and a repeat of the baseline.
- Noise floor: the spread between the two baseline runs, in tasks passed.
- Rule: a difference inside the noise floor is not a result.

Further Reading

• Scaling Managed Agents: Decoupling the brain from the hands — the source for harness assumptions going stale and a reset becoming dead weight on a newer model.
• Harness design for long-running application development — removing one component at a time, and the evaluator that pays only beyond the model's solo ability.
• An update on recent Claude Code quality reports — product-layer changes that degraded an agent while the API was unchanged, and the ablation discipline that followed.
• Quantifying infrastructure noise in agentic coding evals — why a benchmark score moves with the machine, and why small gaps need a matched setup.
• Towards a science of scaling agent systems — when extra agents help and when coordination cost dominates.

Exercises

1. List your pieces. Write down every piece of machinery between your model and your task: resets, extra agents, prompt rules, retries, retrieval steps. For each, write the limit it was added to cover. If you cannot name the limit, mark it Suspect.
2. Pin a task set. Pick 20 to 30 tasks from real work, record the machine settings, and run your current harness on them twice. The gap between the two runs is your noise floor; write it in harness/ablation-tasks.md before testing anything.
3. Run one ablation. Remove one piece from the ledger, rerun the pinned tasks, and write the verdict, the cost change, and the next re-test date. Do not remove a second piece until the first verdict is written.

Check your understanding