Practice project: Build a Complete Agent Harness (Capstone)

Lecture 20. A Working Tool Can Still Be the Wrong Tool

A support agent has six tools wired in: search, find, lookup, query, grep, and get_customer. Every one of them works. Every one passes its unit tests. A customer asks where a refund is, and the agent calls search with the order number, gets back a page of help-center articles, calls find, gets a different page of help-center articles, then calls lookup, which returns a billing record for the wrong account because two ids were buried in the middle of the reply. The final get_customer call dumps several hundred fields into the window, and the agent answers from whichever field it noticed first. The refund status it reports belongs to someone else.

Nothing is broken in the sense your test suite understands. The code did what the code says. The model made a reasonable choice among six overlapping names, then reasoned over a reply that buried the one fact it needed. Nobody can point at a failing assertion, and the customer still got the wrong answer.

Decide now: when this agent misroutes a request, do you change the model, rewrite the prompt, or look at the tools? Pick one before reading on.

This lecture gives you a tool contract sheet: every agent-facing tool scored on what it does, how it is called, what it returns, and what a failure tells the model. It serves the Plan stage of the Plan → Implement → Verify → Evolve loop, because what the agent is allowed to do is decided here, and it feeds Verify, because a tool whose reply is noise cannot be checked.

What is actually going on

A tool is not only code. Its name, its description, its argument list, and its result all end up in the model's context. That means a tool shapes the model's behavior twice: once before it is ever called, when the description is read during selection, and once after, when the result is read to choose the next step. Anthropic's engineering team makes this point directly in Writing effective tools for agents: tool descriptions and specs are loaded into the agent's context, so they steer how the agent calls tools, and refining the wording is one of the most effective ways to improve a tool.

A line worth keeping: a tool is read before it is run.

Because the model reads it, a tool has two separate costs. The first is the cost of choosing: how hard is it to pick this tool, with these arguments, over the others? The second is the cost of reading the answer: how much of the reply does the model have to carry and sift before it can act? A beautiful backend API can be a poor agent tool on both counts. If its description overlaps with another tool's, choosing costs more. If it returns every column in the table, reading costs more, and the extra fields sit in the window for every later turn, the same recurring tax Lecture 17 described for file dumps.

Four properties let you score any tool on both costs:

• Affordance: what the tool can actually do.
• Contract: how the model has to spell the request.
• Observation: what comes back.
• Feedback: whether what comes back makes the next correct step easier.

Watch where each hop in the picture below adds to the context. The description is paid for every turn, selection happens once per call, and the result is paid for again on every turn after it lands.

That is the whole mechanism: the tool sits in the window before it runs, and its answer sits in the window after. Everything else in this lecture is a way to make both sides smaller and clearer.

The rule and its condition

The rule is a condition: keep a tool only when it is distinct from the others, is chosen correctly when it should be, and returns what the next decision needs. Otherwise merge it, rewrite it, or trim it. Count tools by how often the right one is picked, not by how many exist.

More tools are not more capability. Each added tool widens the set of choices on every selection step. The same guidance says that too many tools, or tools that overlap in function, can distract an agent from efficient strategies, so it recommends being selective about which tools you build at all. It also suggests namespacing related tools under common prefixes so their boundaries are visible, and it reports that prefix versus suffix naming had non-trivial effects on its own tool-use evaluations, with results that vary by model, so you should choose a scheme by evaluating it on your own tasks. A tool that adds reach but also adds ambiguity can make the agent worse overall, so the only honest test is a run against realistic tasks.

When a tool fails the table, the fix depends on which row failed. Overlap calls for a merge, a vague name calls for a rewrite, and a bloated reply calls for trimming. The flowchart below puts those in order so you ask the cheapest question first.

Merging comes first because it is the only fix that removes a decision instead of improving one. Rewrites and trims come after, on the tools that survive.

Worked case

Situation: the support agent from the opening. It has six tools and answers refund questions. The team has a week and no appetite for a new model. They keep a log of tool calls with the user's question and the tool the agent picked.

Decide now: with the log in hand, which tool do you look at first, and what do you score it on?

Start with the log, not the code. Group the calls by question type and count how often each tool was the right pick. Three tools show up on refund questions: search, find, and lookup. The first two return help-center articles, so for an order question they are always the wrong pick. They exist because two different people added them at different times, and their descriptions both begin with the same sentence. That is an overlap row. Per the flowchart, merge them into one search_help tool with a description that says it is for policy and how-to questions only, and says outright that it is not for looking up an order.

Illustrative composite. Modeled numbers for a support agent's tool log, not a measured audit: of 200 refund questions, the agent picked a help-center tool first on 74 of them, and on 61 of those it then needed at least two more calls to reach the order. After merging the two help-center tools and adding one sentence to each description saying which kind of question it was not for, the first-pick rate for the order tool on refund questions rose to 181 of 200.

The next score is observation. get_customer returns the whole record. Trim it. The default reply carries customer_id, order_id, refund_status, and last_updated, and a detail: true option returns the full record for the rare case that needs it. The ids the next step needs are now at the top of a reply that is a few lines long, so a second call cannot silently use the wrong account.

The third score is failure. A lookup with an unknown order id used to return the string Error. Ask what the model would need in order to do something useful, and the answer is a short list: what happened, whether anything changed, whether a retry makes sense, and what to try instead. OpenAI makes the same point about its own tooling in Harness engineering: because the team writes its own lint rules, it writes the error messages to inject remediation instructions straight into agent context. A tool error is context in exactly the same way.

The sequence below shows the same failed call twice. The first reply gives the model nothing to act on. The second gives it a next move.

The failure shape that works is small and the same across every tool, so the model learns it once. The example file at the end of this lecture gives it as JSON. The cost of the rejected option, leaving the error as a bare string, is a model that guesses, and a guess at an order id on a refund question is how a customer receives someone else's status.

The finished contract sheet has a row per tool and a verdict on each. Two tools were merged into one, the order tool was trimmed, the customer tool got a detail option, and both returned a structured failure.

Faded case

Skip to the open case if this is familiar. A different team has a coding agent with a read_file tool, a search_code tool, and a run_shell tool. run_shell can do everything the other two can. The log shows the agent uses run_shell for about half of its file reads and spells the path wrong a noticeable share of the time. search_code is almost never called.

Your decision: do you merge, rename, or keep, and which tool do you change first? Name the row of the table that justifies it.

Open case

Illustrative composite. A billing assistant has a get_invoice tool that returns the full invoice object, including internal ledger ids, tax breakdowns, and line items. It also has a list_invoices tool and a search_invoices tool. The model picks list_invoices when asked about one invoice about a third of the time, then reads through a long list to find it. When get_invoice fails because an invoice id is mistyped, it returns the text Invalid.

Write down, in three lines, the verdict you would give each of the three tools and the one change you would make to the failure message. The first quiz question below checks your answer.

Where this breaks

Three limits are worth knowing before you trust a contract sheet.

A raw shell is sometimes the right tool. A narrow structured tool is easier to describe, but when the real task does not fit its schema, it gets in the way and the agent works around it. If the task is open ended and the model is strong at the shell, the broad tool can win. Neither shape is better in general. The only way to know is to run both against your tasks and count wrong picks, which is why the sheet begins with a log and not with an opinion.

A clean contract sheet does not make tool output safe. A tool can return text that came from somewhere you do not control, such as a web page, a file, or a connector. Anthropic's How we contain Claude across products notes that a trusted connector can still load poisoned content into the model's context, because auditing the connector is not the same as auditing the data it returns. Treat the Returns column as data, never as instructions, and let the harness, not the tool description, decide what the model may do with it.

Scores go stale. The sheet describes your tools against your model on your tasks today. A new model may pick differently, or handle a bigger reply without trouble. Lecture 23 covers how to retest a piece of the harness when the model changes; for tools, the cheapest trigger is the call log itself. If the wrong-pick rate moves, the sheet is out of date.

The policy check sits after the model, not inside the tool description. A well-written description makes a wrong pick less likely, but it does not stop a request the harness has not approved.

Key Takeaways

• Score every tool on affordance, contract, observation, and feedback before you blame the model.
• A tool is read before it is run.
• Merge overlapping tools first, because removing a choice beats explaining it.
• Return only the fields the next step needs, with the ids it must reuse.
• Write failures as observations: what happened, what changed, and whether a retry is safe.
• Treat tool output as data and let the harness decide what follows from it.

What You Have Now

A tool contract sheet: every agent-facing tool scored on what it does, how it is called, what it returns, and what a failure tells the model. Save it as harness/tool-contracts.md in your repository, with the failure shape beside it as harness/tool-error-shape.json; the lab's checker looks for both at those paths. It sits in the Plan stage of the Plan → Implement → Verify → Evolve loop, because it fixes what the agent may do and what it will be told, and it gives Verify a reply it can actually check.

Example Files

tool-contracts.md

One row per tool. Fill the last column only after you have a call log to count from.

# Tool Contracts

| Tool | What it does | Call shape | Returns | On failure | Keep or merge? |
|---|---|---|---|---|---|
| search_help | Answers policy and how-to questions from the help center | query: string | Top matches: title, one-line summary, article id | not_found, nothing changed, retry with different words | Keep (merged search and find) |
| get_order | Fetches one order by id | order_id: string, detail: boolean | order_id, customer_id, refund_status, last_updated; full record only when detail is true | not_found, nothing changed, similar ids listed | Keep (trimmed) |
| get_customer | Fetches one customer by id | customer_id: string, detail: boolean | customer_id, plan, open_orders; full record only when detail is true | not_found, nothing changed, retry is safe | Keep (trimmed) |

tool-error-shape.json

The same shape for every tool, so the model learns it once.

{
  "status": "failed",
  "reason": "not_found",
  "requested": "A-1042",
  "side_effects": "none",
  "retryable": true,
  "try_instead": ["A-1024", "A-1402"]
}

Further Reading

• Writing effective tools for agents — the source for treating tool descriptions as model input and for testing tools against realistic tasks.
• Effective context engineering for AI agents — why tools should be self-contained, robust to error, and low in overlap.
• Harness engineering: leveraging Codex in an agent-first world — custom lint messages that carry the fix, the same idea as an actionable tool error.
• How we contain Claude across products — why tool output stays an attack surface even when the tool is trusted.

Exercises

1. Count your wrong picks. Take one agent session log from your own project and list every tool call next to the task it was for. Mark each call right or wrong. The ratio per tool is the first column of your sheet, written before you change anything.
2. Build the sheet. Create harness/tool-contracts.md with the columns from the example file and a row for every tool your agent can reach. Fill the verdict column only for tools you have log evidence on, and write the reason in a sentence.
3. Give one tool a real failure. Pick the tool with the most wrong picks. Make its failure reply match harness/tool-error-shape.json, then rerun the task that failed and note whether the next call changed.

Check your understanding