Lesson 10. Evals Are the Contract

Harbor Mutual's claims-correspondence pilot went into its steering meeting with one slide: a quality score from the vendor's dashboard, green, comfortably above the target the legal team had asked for. Three weeks later the claims manager pulled the assistant from the motor team. Adjusters were rewriting most drafts before sending them. The letters were long, polite, and confidently cited policy clauses that did not apply to the claim in front of them. When the architect asked how the dashboard score had been produced, the answer was: a second model read each draft and rated it out of ten. Nobody had ever compared those ratings with what an adjuster would say about the same draft. The second model preferred long drafts, and the assistant had learned to write long. The pilot's exit criteria were now unmeasurable, and the contract had been written around the number. Decide now: before you accept any quality score into a contract, what is the one thing you demand to see next to it?

Illustrative composite. The dashboard read 92 out of 100 on the vendor's ten-point scale averaged across a week. In the same week adjusters rewrote 6 of every 10 drafts before sending. The judge model had never been run on a human-labelled sample, and it belonged to the same model family as the drafting model. The numbers are a teaching device, not a measurement from a real insurer.

The thing to demand is the judge's agreement with humans on a labelled sample of your own data. This lesson's deliverable is 07-eval-plan.md in your Dossier. An evaluation plan for your case: the golden set spec, the rubric, the judge validation protocol with its agreement threshold, and the CI gate. It is the first document of the Prove stage of Frame / Decide / Prove / Run: the evidence the client signs, in place of a model name.

What is actually going on

An evaluation is a measuring instrument with three parts, and each part fails in its own way. The sample is the set of inputs you measure on. The scorer turns an output into a number. The decision rule turns numbers into ship or block. Harbor Mutual's dashboard had a sample nobody had chosen (whatever traffic came through), a scorer nobody had checked, and no decision rule at all beyond "green is good".

The sample fails when it is written by the prompt author from the happy path. A set of thirty polite motor claims measures how well the assistant handles polite motor claims. It says nothing about the hostile customer, the missing policy document, the claim that straddles two policy versions, or the attachment that contains instructions addressed to the model. Anthropic's own guidance for test design says the same thing from the other side: make the cases task-specific, include the edge cases and the irrelevant or harmful inputs, and prefer many automatically graded cases over a few carefully hand-graded ones (develop tests, observed 2026-09-09).

The scorer fails when it is a language model with unexamined preferences. The MT-Bench work that popularised the idea of a model grading another model also catalogued the ways it goes wrong: position bias (the first answer in a pair wins more often), verbosity bias (longer answers win), self-enhancement bias (a judge prefers text in its own style), and limited reasoning on hard grading tasks (Zheng et al., 2023, observed 2026-09-09). Later work measured how much a judge over-rates its own family's answers and found the effect can be large in either direction and cannot be prompted away (self-preference study, observed 2026-09-09). Every one of these biases was operating on Harbor Mutual's dashboard.

The decision rule fails when it reads a single percentage as a fact. A pass rate on a sample is an estimate with an error bar, and the error bar on a small sample is wide enough to swallow most of the differences teams argue about. The statistics section of the worked case shows the arithmetic.

Two settings share these three parts. Offline evaluation runs the golden set on demand, before a release, and is the gate. Online evaluation samples live traffic, grades a slice with the validated judge and with implicit signals (edits, thumbs, escalations), and exists to catch the moment the live distribution drifts away from the golden set. The picture below shows how the two feed each other; read it left to right and look for the arrow that closes the loop.

The arrow from the online judge back into the traces is the one Harbor Mutual never drew: nothing that adjusters rejected ever became a labelled case, so the golden set (had there been one) would have frozen on launch day.

The rule and its condition

Use a code grader when the output has a checkable form (a label, a number, a schema, a citation that either resolves or does not). Use a model judge when the quality question needs reading, and only after the judge has been checked against human labels on your own data with a written agreement threshold. Use humans directly when the sample is small enough to afford and the stakes are high enough to demand it. Never gate a release on a judge that has not passed its own validation.

Pairwise and pointwise answer different questions. Pairwise says "B beats A on 58 of 100 pairs"; it is the right shape for prompt A/B tests and it neutralises position bias when you run both orders and keep only consistent winners. Pointwise says "84 of 100 outputs pass the rubric"; it is the only shape that can carry a contractual threshold, because a client cannot sign "better than last week". Preference questions get pairwise; gates get pointwise.

When there are more than two choices a tree helps. Follow the diamonds from the top; each exit names the grader and the one thing it still needs.

The tree ends in "label a slice" more often than teams like. That is the cost of the rule: a judge earns its place with an afternoon of expert labelling, not with a model name.

When the system is an agent

An agent produces a trajectory, not an answer: a sequence of tool calls, arguments and intermediate decisions ending in an outcome. Grading only the final text misses the unsafe tool call in the middle and the loop that took twelve steps where three would do. Trajectory evals add three families of metric: tool-call correctness (was the right tool called with the right arguments, checked by code against the expected call), step efficiency and plan adherence (did the path stay inside the budgets the agent spec from lesson 7 set), and outcome (did the environment end in the required state).

Agents are also unstable across runs, and a single pass rate hides it. The tau-bench authors introduced pass^k: the probability that all k independent trials of the same task succeed, which for a per-task success rate p is p to the power k. Their published retail-domain result for one frontier model fell from a pass^1 of 0.604 to a pass^4 of 0.383 (tau-bench paper, observed 2026-09-09). An agent that succeeds on six runs in ten looks fine on one run and unacceptable when a customer hits it four times. Run at least four trials per task and report pass^k; never report best-of-k as accuracy.

Public agent benchmarks are references for what "good" looks like in a domain, not substitutes for your set. SWE-bench Verified is a human-validated slice of 500 software tasks (swebench.com, observed 2026-09-09); the Berkeley Function Calling Leaderboard grades tool-call correctness across simple, parallel, multi-turn and agentic categories (BFCL, observed 2026-09-09); tau-bench and its successor simulate a user in retail, airline, telecom and banking domains. Read them to borrow their grading shapes (environment state checks, pass^k, simulated users). Your gate still runs on Harbor Mutual's letters, not on airline bookings.

Worked case

Harbor Mutual's assistant drafts claim correspondence grounded in policy wording and proposes adjuster visits. The NFR budget sheet from lesson 2 gives the eval its targets, and the legal team gave the constraint that matters most: a letter that cites a clause which does not apply is a regulatory incident, whatever else it gets right.

Illustrative composite. Budgets carried over from the Frame stage: hallucinated or misapplied policy clause at most 1 in 100 drafts; required disclosures present in 99 of 100; tone within the style guide in 95 of 100; adjuster-acceptable without rewrite in 80 of 100; cost per eval run under $50. Traffic: three claim types (motor, property, liability) in roughly a 55/30/15 split. All numbers are a teaching device.

The options on the table were the vendor dashboard (a same-family judge, ten-point scale, no labels), a human panel grading every release (three senior adjusters, a day per release), or a golden set with a validated judge and a small standing human slice. Decide now: which do you propose, and what do you have to produce before the first release can be gated?

The architect chose the third option and produced four things in order.

1. The golden set spec. Three hundred items, versioned. Two hundred and forty sampled from eighteen months of adjuster-written correspondence, stratified to match the claim-type split. Sixty constructed edge cases, twenty per family: missing or superseded policy document, ambiguous or hostile customer text, and attachments carrying instructions aimed at the model. Each item carries the input bundle (claim record, policy wording version, customer message), a reference letter written or approved by an adjuster, and the expected next action. Ownership: the claims quality lead owns the set; the architect owns the spec; items are added from online sampling monthly.

2. The rubric. Five binary criteria per draft, graded independently: the cited clause exists in the referenced policy version and applies to the claim; no factual statement about the claim is unsupported by the record; the required disclosures for the claim type are present; the tone matches the style guide; the proposed next action matches the reference. A draft passes when all five pass. Binary criteria, not a ten-point scale: a judge agrees with humans far more reliably on "is the disclosure present" than on "is this a seven or an eight".

3. The judge and its validation. The judge is a different model family from the drafter, pinned to a version, with the rubric and the reference letter in its prompt, and grades pointwise. Before it may gate anything it is validated on a 150-item labelled slice: two adjusters label independently, a third adjudicates disagreements, and the judge's verdicts are compared with the adjudicated labels. The picture below shows the protocol as a sequence; watch what happens when the threshold is missed.

The loop revises the rubric and the judge prompt; it never revises the labels to fit the judge. The threshold is written down before the run:

Agreement threshold (pre-registered)
  per criterion, judge vs adjudicated human label, n = 150
  lower bound of the 95% CI on raw agreement  >= 0.80
  Cohen's kappa                                >= 0.70
  false-pass rate (judge passes, humans fail)  <= 0.05
  all three must hold on every criterion; the worst criterion decides

Standard error of a proportion
  SE = sqrt( p (1 - p) / n )
  95% CI ~= p ± 1.96 SE

  n = 150, observed agreement p = 0.88
    SE = sqrt(0.88 × 0.12 / 150) = 0.0265
    CI = 0.88 ± 0.052  ->  0.828 to 0.932   (lower bound clears 0.80)

  n = 100, same p
    SE = sqrt(0.88 × 0.12 / 100) = 0.0325
    CI = 0.88 ± 0.064  ->  0.816 to 0.944   (still clears, barely)

  n = 300, same p
    SE = 0.0188 ; CI = 0.843 to 0.917

Cohen's kappa
  kappa = (p_o - p_e) / (1 - p_e)
  p_o = 0.88 (observed agreement)
  humans pass 0.70 of items, judge passes 0.72
  p_e = 0.70 × 0.72 + 0.30 × 0.28 = 0.588
  kappa = (0.88 - 0.588) / (1 - 0.588) = 0.709   (clears 0.70)

False-pass rate
  items humans failed = 45 ; judge passed 2 of them
  false-pass = 2 / 45 = 0.044   (clears 0.05)

Two things in that arithmetic decide the design. The confidence interval at n = 150 is about five points wide in each direction, so the threshold is stated on the lower bound: a judge that scores 0.88 on the sample has only shown it is above 0.83. And kappa cleared by less than a point; the citation criterion is where the judge and the adjusters disagreed most, and the rubric wording for it went through two revisions before the freeze.

4. The gate. The offline run and its decision rule, pre-registered in the release template:

Release gate (every prompt, model, retrieval or tool change)
  run the full 300-item set, four trials per item for the tool-calling path
  pass rate: lower 95% bound >= 0.80 on all-five-criteria pass
  clause criterion: zero failures on the 20-item superseded-policy slice
  disclosure criterion: lower bound >= 0.97
  tool-call correctness on the booking path: pass^4 >= 0.85
  paired comparison with the previous release: no criterion down by
    more than one standard error without a written waiver

  n = 300, observed pass p = 0.84
    SE = sqrt(0.84 × 0.16 / 300) = 0.0212
    lower bound = 0.84 - 1.96 × 0.0212 = 0.798   -> FAIL by two tenths of a point

  paired difference vs previous release, 300 shared items
    discordant items (one passes, other fails) = 60  ->  0.20
    SE_diff = sqrt(0.20 / 300) = 0.026
    a 2-point gain is under one SE: not a difference, do not celebrate it

The first real run failed the gate by two tenths of a point, which is the gate working: the team could either raise the pass rate or grow the set to tighten the interval, and both were honest moves. What they could not do was round 0.798 up.

Illustrative composite. Cost of one full gate run, at Anthropic's list prices: drafting 300 items on Sonnet 5 at about 6,000 input and 600 output tokens each is 1.8M input at $2 per million plus 180K output at $10 per million, about $5.40; judging 300 items on Opus 5 at about 4,000 input and 200 output tokens each is 1.2M at $5 plus 60K at $25, about $7.50; the four-trial booking path adds roughly $9. Under $25 a run, inside the $50 budget. Prices from the Anthropic pricing page, observed 2026-09-09; the token counts are modelled.

The rejected option had a price too. The vendor dashboard would have cost nothing to adopt and would have put a number in the contract that no one could reproduce; the first customer complaint would have been the first measurement. The full human panel would have cost three adjuster-days per release and would have stopped happening by the third release, at which point the team would have shipped without any gate at all.

Faded case

Skip to the open case if this is familiar.

Lumen sells an AI support agent to SaaS companies. One engineer, twenty percent monthly growth, and a prompt change every week. They have no golden set; they have thumbs-up data from tenants and a feeling that last week's prompt was better. The engineer wants to A/B the two prompts on live traffic and pick the winner, and asks whether they need "all that judge validation stuff" for a comparison.

The shape is different from Harbor Mutual in two ways. The question is a preference (A or B), so the judge grades pairwise, both orders, and only consistent winners count; position bias is neutralised by the design rather than by validation. And the traffic volume is large enough that a small weekly sample gives a narrow interval:

Illustrative composite. Two percent of traffic is 4,000 conversations a week. Under the null of no difference the win rate is 0.50, so SE = sqrt(0.25 / 4000) = 0.008. A five-point lift is more than six standard errors: detectable in one week. Judging on Haiku 4.5 at two orderings per pair, about 2,000 input and 50 output tokens each, is 16M input at $1 per million plus 0.4M output at $5 per million, about $18 a week, from the Anthropic pricing page, observed 2026-09-09.

What the pairwise result cannot tell Lumen is whether either prompt is good enough for a tenant to keep paying, and that is the number their sales deck quotes. One decision is left to you: what is the smallest validation that lets the same judge also produce an absolute number Lumen can put in front of a tenant?

Open case

A public-sector benefits helpdesk wants an assistant that answers eligibility questions in three languages, grounded in the benefit rules and the caller's declared circumstances. The agency's non-negotiable: the assistant may say "I cannot determine that, a caseworker will call you" as often as it likes, but a wrong statement of eligibility in either direction is a reportable error. Calls are recorded and transcribed; no labelled data exists; the caseworkers who could label are the same people whose backlog the assistant is meant to reduce. Design the eval plan: the golden set, the grader for the eligibility statement, how the judge gets validated with caseworkers who have no spare hours, and the gate. The first quiz item grades your answer.

Where this breaks

The golden set ages. Production drifts: a new benefit rule, a new policy wording version, a new customer behaviour. A set that is not refilled from online sampling measures last year's problem. Lesson 14 puts the refresh on a schedule and measures drift against the set; the eval plan states the cadence and the owner so that lesson has something to schedule.

Small samples cannot see rare failures. A failure mode that occurs once in two hundred inputs will appear in a 300-item random sample about once or twice, and its confidence interval will run from zero to several times its true rate. The fix is not a bigger random sample; it is a constructed slice that contains the failure mode on purpose, graded with a zero-tolerance rule, which is what the superseded-policy slice does.

Judges disagree with humans most where it matters. Validation agreement is an average across items, and the hard items (ambiguous clauses, borderline tone) are exactly where judge and human diverge. Report agreement per criterion and per stratum, gate on the worst, and keep a standing human slice on the hard stratum even after the judge validates.

Final-output grading hides unsafe trajectories. An agent can produce a correct letter after calling the booking tool twice with the wrong date. Trajectory metrics and pass^k catch this; a text rubric never will, which is why the booking path carries its own line in the gate.

Key Takeaways

• Demand a judge's agreement with human labels before you accept its score anywhere.
• Grade with code where the output has a form; validate a judge where it does not.
• Pre-register the gate on the lower confidence bound, and state the sample size that produced it.
• Stratify the golden set: random bulk plus a constructed slice per named failure mode.
• Grade agents on trajectories with pass^k over several trials, never best-of.
• Refill the set from online sampling; a frozen set measures launch day.

What You Have Now

An evaluation plan for your case: the golden set spec, the rubric, the judge validation protocol with its agreement threshold, and the CI gate. It lives at dossier/07-eval-plan.md, next to the agent spec from lesson 7 whose tool catalogue it grades and the NFR budget sheet from lesson 2 whose numbers it operationalises. It serves the Prove stage: the pilot cannot exit without a gate that passed, and the go/no-go memo in project 3 quotes this document's thresholds. Lesson 11 prices every run of it, lesson 12 adds the adversarial slice, and lesson 14 turns the refresh cadence into a schedule.

Example Files

The eval plan template. Copy it into the Dossier and replace every bracketed field.

# 07 · Evaluation plan — [system name]

## 1. Budgets this plan proves (from 01-nfr-budgets.md)
| Budget | Number | Measured by | Owner |
|---|---|---|---|
| [hallucinated clause] | [≤ 1 in 100] | [criterion C1, golden set] | [claims quality lead] |

## 2. Golden set
- Version: [v1.0] · Location: [path] · Owner: [name]
- Size: [300] = [240] sampled + [60] constructed
- Sampling frame: [source, period] · Strata: [claim type 55/30/15]
- Constructed slices (name, size, failure mode, tolerance):
  - [superseded-policy, 20, cites withdrawn clause, zero failures]
  - [hostile-customer, 20, tone breach, ≤ 1]
  - [instruction-bearing attachment, 20, follows injected text, zero]
- Item schema: input bundle, reference output, expected action, stratum, label provenance
- Refresh: [monthly], [N] items from online sampling, owner [name]

## 3. Rubric
| ID | Criterion (binary) | Grader | Reference needed |
|---|---|---|---|
| C1 | [cited clause exists in referenced version and applies] | [code lookup + judge] | [policy version] |
| C2 | [no unsupported factual claim about the claim record] | [judge] | [claim record] |
| C3 | [required disclosures present for claim type] | [code checklist] | [disclosure table] |
| C4 | [tone within style guide] | [judge] | [style guide] |
| C5 | [proposed next action matches reference] | [code] | [reference action] |
Pass = all criteria pass.

## 4. Judge
- Model/version: [family, pinned id] · different from generator: [yes]
- Mode: [pointwise, binary per criterion] · prompt hash: [sha]
- Validation slice: [150] items, labelled by [2 labelers + adjudicator]
- Threshold (pre-registered): lower 95% bound of agreement ≥ [0.80];
  kappa ≥ [0.70]; false-pass ≤ [0.05]; per criterion, worst decides
- Validation result: [table per criterion] · frozen on [date]
- Re-validate when: [judge model changes, rubric changes, quarterly]

## 5. Agent trajectory metrics (if tools are called)
- Tool-call correctness: [expected call per item, code-checked]
- Trials per item: [4] · Report: pass^4
- Budget adherence: [max steps, max tokens from 05-agent-spec.md]

## 6. Gate
- Trigger: [every prompt, model, retrieval, tool change]
- Rules: [lower bound pass ≥ 0.80; slice rules; pass^4 ≥ 0.85; paired regression rule]
- Waiver: [who may sign, what evidence]
- Runner: [tool] · Cost per run: [$] · Runtime: [min]

## 7. Online evaluation
- Sample: [2% of traffic] · Judge: [same pinned judge] · Human slice: [20/week on hard stratum]
- Implicit signals: [edit distance before send, escalation, thumbs]
- Feeds: [new items into §2 monthly]

The judge validation protocol as a runnable checklist:

judge_validation:
  slice:
    size: 150
    strata: { motor: 0.55, property: 0.30, liability: 0.15 }
    include_constructed: true          # hard items must be in the slice
  labelling:
    labelers: 2
    adjudicator: 1
    blind_to_model_output_source: true
    record: [label_per_criterion, labeler_id, adjudication_note]
  judge:
    model: "<different family, pinned version>"
    mode: pointwise_binary
    prompt_hash: "<sha256>"
    inputs: [item, rubric, reference_output]
  metrics_per_criterion:
    agreement_lower_95: ">= 0.80"
    cohen_kappa: ">= 0.70"
    false_pass_rate: "<= 0.05"
  on_fail:
    - revise_rubric_wording
    - revise_judge_prompt
    - rerun_same_labels          # never relabel to fit the judge
  on_pass:
    - freeze: [judge_model, prompt_hash, rubric_version]
    - revalidate_on: [judge_model_change, rubric_change, quarterly]

A CI gate definition in the declarative style the open-source runners accept (the field names are illustrative; keep the thresholds):

gate:
  suite: golden-set-v1.0
  trials_per_item: 4                 # tool-calling path only
  assertions:
    - metric: pass_rate_all_criteria
      lower_bound_95: ">= 0.80"
    - metric: slice.superseded_policy.failures
      equals: 0
    - metric: criterion.C3.lower_bound_95
      value: ">= 0.97"
    - metric: booking_path.pass_pow_k
      k: 4
      value: ">= 0.85"
    - metric: paired_delta_vs_previous
      rule: "no criterion below -1 SE without waiver"
  on_fail: block_merge
  artifacts: [report.json, per_item.jsonl, judge_prompt_hash]

Tooling, by where the gate runs. Licences as read on each project's own site on 2026-09-09; hosted tiers change often, so the plan links the pricing page and quotes no figure.

Further Reading

• Anthropic: define success and develop tests — the vendor's own guidance on task-specific cases, automated grading and a separate evaluator model.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — the paper that named position, verbosity and self-enhancement bias.
• Miller, Adding error bars to evals — standard errors, paired comparisons and clustering for eval results.
• tau-bench — the pass^k metric and simulated-user agent evaluation.
• Berkeley Function Calling Leaderboard — grading shapes for tool-call correctness.
• Inspect documentation — a clear reference for task, solver and scorer separation in agent evals.

Exercises

1. Take the NFR budget sheet from your own case and write one rubric criterion per budget, marking each as code-graded or judge-graded and naming the reference material the grader needs. Any budget that yields no criterion is not yet a budget.
2. Label a 40-item slice of your own outputs against that rubric with a colleague, independently, then compute raw agreement, kappa and the confidence interval by hand. Decide whether the interval is narrow enough to gate on and what sample size would be.
3. Write the gate for your case in the declarative form above, including the paired-regression rule and the waiver path, and compute the cost of one run at the prices on the Anthropic pricing page with today's date beside them.

Check your understanding