Lesson 04. Choose the Model by the Budget, Not the Benchmark

A claims-correspondence assistant goes live on the model that topped the public leaderboard the week the project started. The demo was flawless. Then the real adjusters arrive, each one waiting for a draft letter before they can move to the next file. Drafts take twice as long as the workflow tolerates, the first month's invoice lands on the CFO's desk at more than three times the ceiling agreed in discovery, and the team spends the following six weeks trimming prompts to claw back latency they never needed to lose. In week seven someone finally runs the golden set that had been sitting in a spreadsheet since discovery. A mid-tier model passes it. A small model with an escalation path passes it too, at a fraction of the cost, inside the latency budget. Nobody had checked, because the leaderboard had already "decided".

Illustrative composite. The scenario above is a teaching composite: p95 latency of 14 s against a 6 s budget, an invoice at 3.4x the agreed monthly ceiling, and a mid-tier model passing the same golden set at roughly a third of the cost per draft. The shape is common; the figures are modeled.

Decide now: if you had two weeks and the NFR budget sheet from discovery, which model would you have shipped, and what single piece of evidence would have justified it?

This lesson gives you a model-selection ADR for your case with the review condition that would reverse it, plus a routing sketch. It is the first Decide artefact in your Dossier: the charter and budgets from Frame become a choice you can defend and, when the condition fires, reverse.

What is actually going on

A leaderboard measures a distribution of tasks that is not yours. The top of any public benchmark is crowded: on saturated coding suites the leading scores sit within about a point of each other, so the ranking tells you almost nothing about which model will draft an insurance letter correctly. Worse, the budgets that end projects never appear on a leaderboard at all. Residency is a yes or no. Licence is a yes or no. Latency is a p95 you measure on your own traffic. Cost is a per-task figure that depends on how many tokens your context carries, not on the model's headline score.

Look at the price structure and the selection problem changes shape. Frontier vendors now sell a ladder of tiers: on Anthropic the input prices sit at $1, $2, $4 and $10 per million tokens for Haiku 4.5, Sonnet 5, Opus 5.5 and Fable 5.1, with output at five times input on every rung (pricing page, observed 2026-09-27). OpenAI and Google publish comparable ladders on their own pages (OpenAI pricing, Gemini pricing); read the numbers there rather than from a summary, because both move without notice. Across every vendor the same three multipliers dominate the bill: a cache read on a reused prefix costs 0.1x the base input price (0.025x on Fable 5.1, 0.05x on Opus 5.5), a batch job costs 0.5x, and US-only inference on Anthropic costs 1.1x (pricing page, observed 2026-09-27). Those multipliers are architecture, not model choice. A well-cached Opus 5.5 call can cost less than a badly built Sonnet 5 call.

The cost of a wrong choice is also asymmetric in time. Vendors publish retirement dates as "not before" commitments; as of 2026-09-27 Haiku 4.5's earliest retirement is still 2026-10-15, under three weeks out, and Fable 5.1's is 2027-09-01 (models overview). A choice with no review condition turns into a forced migration under a deadline, and a migration across tokeniser generations changes the bill: Claude 4.7 and later produce roughly 30% more tokens for the same text than Sonnet 4.6 and earlier, per the pricing page (observed 2026-09-27), so a price comparison that is not re-tokenised is wrong before it starts.

The selection, then, is a funnel, not a ranking. Watch which gates eliminate and which gates score.

The picture makes one thing visible: capability is a gate, not a score. A model either passes your eval at the budget or it does not; among those that pass, the ranking is by latency and then by cost. That is the reverse of how most teams pick, which is why most teams overpay.

The rule and its condition

Use the cheapest model that passes your golden set at the accuracy budget and inside the latency budget, after residency and licence have eliminated the rest; otherwise, when no single affordable model passes on every task, route: send the routine share to the cheap model and the hard share to a stronger one, and record both in one ADR with the condition that reverses it.

The ordering is the rule. Change it and the decision degrades in predictable ways.

The second decision inside the rule is hosted versus open weights, and it has its own condition.

Licence terms are the part teams skip. Meta's Llama 4 Community License requires a separate licence from Meta if the licensee had more than 700 million monthly active users in the month before the model's release, and it mandates "Built with Llama" attribution (licence text, observed 2026-09-14). Most clients are nowhere near that threshold; a consumer platform might be, and its counsel will ask. Some Mistral variants ship under research-only terms. Put the licence in the elimination gate, and put the check in the constraint register from Lesson 03 with an owner.

Routers, cascades and the effort dial

A router sends each request to one of several models before the call; a cascade calls the cheap model first and escalates when a confidence signal says the answer is weak. Both exist because the routine share of most workloads is large. The FrugalGPT cascade reported up to 98% lower cost with about 4 points higher accuracy on its benchmark tasks (paper, observed 2026-09-09). RouteLLM's learned router reported over 85% lower cost while retaining 95% of the strong model's quality on MT-Bench, with only about 14% of calls needing the frontier model (announcement, observed 2026-09-10; the paper states the method, the per-benchmark figures are in the announcement). Production teams usually see less than the paper figures because their traffic is less separable; treat these as upper bounds you must re-measure on your own eval.

Two cheaper levers come before a router, and a good ADR shows they were tried. The first is the effort setting. Anthropic's own measurements find that on work one model can handle alone, the same model at lower effort was cheaper every time, and they put numbers on the multi-model alternatives: an executor consulting a stronger advisor scored 3.5 points above the executor alone for slightly less money on a coding benchmark, but cost about 2.6 times as much on a task where it consulted the advisor almost every time; an orchestrator with cheaper workers cost 47 to 55% less on large-corpus work and scored 10 to 12 points lower (multi-model guidance, observed 2026-09-14). The second is batch: anything that is not interactive should not pay interactive prices, and most batches finish within an hour (batch processing, observed 2026-09-14).

The picture below places the tiers on the two axes that matter for a router: cost per task and pass rate on your eval. It is illustrative, because the y-axis is your golden set, which nobody else can plot for you.

Illustrative composite. The positions on the chart below are modeled for a drafting task like Harbor Mutual's, not measured on any public benchmark. Re-plot it with your own eval results before it goes into a Dossier.

What the chart shows: the mid tier alone clears the bar, the mid tier with an escalation route for the hard share clears it with margin at a small extra cost, and the frontier tier's extra pass rate sits above the bar where it earns nothing. A router that escalates everything is the frontier model with extra steps; the consult or escalation rate is therefore a number the ADR must commit to measure.

Worked case

Harbor Mutual, the regional property-and-casualty insurer, wants an assistant that drafts claim correspondence grounded in policy wording and can book adjuster visits. Discovery in Lesson 02 produced budgets; Lesson 03 produced a constraint register. The relevant rows:

Illustrative composite. Every figure in this case is modeled for teaching. Volume: 60,000 drafts a month across 1,900 adjusters and handlers. Budgets: clause-citation accuracy ≥ 97% on the golden set, hallucinated-clause rate ≤ 0.5%, p95 latency ≤ 6 s for an interactive draft, cost per draft ≤ $0.03 at p90, all inference and storage inside the EU. Constraint register: no data leaves the EU tenancy; no model whose licence needs a Meta or research-only agreement; text and image input (scanned letters); JSON output for the letter object.

The options on the table were the four Anthropic tiers reached through a regional endpoint in the client's own cloud tenancy, a frontier model from another vendor reached the same way, and a self-hosted open-weight model in the on-prem archive's data centre.

Decide now: write down your primary model and your escalation route before reading on.

Step one, eliminate. The EU-residency row removes every global-routed API endpoint. Anthropic's first-party residency option is US-only; the EU path is a regional endpoint on a cloud marketplace, which carries a 10% premium over global pricing (pricing page, observed 2026-09-14). The licence row removes nothing among the hosted candidates and removes Llama from the self-host list only if Harbor's counsel refuses the attribution clause, which they did not. Modality removes nothing: the candidates accept text and image. Structured output removes nothing. Five candidates survive.

Step two, run the golden set. The interim golden set from Lesson 02 has 300 claim scenarios with gold clauses, built by two senior handlers. Results, modeled:

Illustrative composite. Small tier: clause accuracy 91.2%, fails. Mid tier: 97.6% at default effort, passes. Large tier: 98.4%, passes. Frontier tier: 98.9%, passes. The other vendor's frontier model: 98.6%, passes. Dispute letters (about 12% of volume) are where the mid tier loses most of its misses.

Step three, latency. The interactive draft budget is 6 s at p95, measured end to end from the adjuster's click.

Illustrative composite. Mid tier p95 4.1 s. Large tier 7.8 s. Frontier tier 11.3 s. Other vendor's frontier 9.7 s. The small tier is fastest but already eliminated for drafting.

Only the mid tier survives the latency gate for the interactive route. The large and frontier tiers are not dead: they are candidates for a non-interactive escalation route, where a dispute letter can take 20 s because it goes into a review queue anyway.

Step four, cost per task at p90. Using the pricing page's verified rates and the regional premium:

Illustrative composite. A p90 draft carries a 3,000-token cached prefix, 4,000 tokens of fresh context (claim summary plus retrieved clauses) and 600 output tokens. Mid tier: 4,000 × $2/M + 3,000 × $0.20/M + 600 × $10/M = $0.0146, × 1.1 regional = $0.016. Large tier: $0.0365 × 1.1 = $0.040. Frontier tier: $0.0708 × 1.1 = $0.078. Monthly at 60,000 drafts: mid $960, large $2,400, frontier $4,680. Mid tier for all drafts plus 12% of volume escalated to the large tier: $960 + $288 = $1,248 a month, or $0.021 per draft blended, inside the $0.03 budget with margin for retries.

Self-hosting was costed and rejected on ops grounds: Harbor has no MLOps team, the volume is far below any GPU break-even you will compute in Lesson 11, and the residency requirement is satisfied by the regional endpoint. The rejected leaderboard option would have cost close to four times the cascade each month, failed the latency budget on every draft, and delivered about a point of accuracy above a bar that was already cleared.

The decision, recorded:

# ADR-0001: Model selection for the Harbor Mutual correspondence assistant

Status: accepted
Date: 2026-09-10
Owner: lead architect          Reviewers: Harbor CISO, Harbor finance partner, claims product owner

## Context
- Tasks: (a) classify inbound claim documents and route them; (b) draft claim
  correspondence grounded in policy wording; (c) draft dispute and complaint
  responses that go to a legal review queue.
- Budgets (01-nfr-budgets.md): clause-citation accuracy ≥ 97% on the 300-item
  golden set; hallucinated-clause rate ≤ 0.5%; p95 latency ≤ 6 s for (b);
  cost per draft ≤ $0.03 at p90; EU residency for inference and storage.
- Hard constraints (02-constraints.md): no data outside the EU tenancy; no
  licence requiring a separate vendor agreement; text+image input; JSON output.
- Eval: interim golden set v0.3, 300 items, owner = claims QA lead, run 2026-09-08.

## Decision
- Primary for (b): mid tier (Sonnet-class), reached through the regional
  endpoint inside Harbor's EU cloud tenancy, default effort.
- (a): small tier (Haiku-class) via the same endpoint, batch where the SLA allows.
- (c) and low-confidence (b): large tier (Opus-class) via the same endpoint,
  non-interactive route with a 20 s budget, output to the legal review queue.
- Escalation trigger: document class = dispute/complaint, OR the draft's
  self-reported confidence < 0.7, OR a citation the retriever cannot resolve.
- Pinned IDs recorded in 08-cost-model.md with earliest retirement dates from
  the vendor's models page; migration eval scheduled 60 days before each.

## Alternatives considered

| Option | Residency/licence | Golden-set pass | p95 latency | Cost per draft (p90) | Ops burden | Verdict |
|---|---|---|---|---|---|---|
| Small tier only | pass | 91.2% (fail) | 2.3 s | $0.008 | none | rejected: fails accuracy |
| Mid tier only | pass | 97.6% | 4.1 s | $0.016 | none | accepted as primary |
| Large tier only | pass | 98.4% | 7.8 s | $0.040 | none | rejected for (b): latency; accepted for (c) |
| Frontier tier only | pass | 98.9% | 11.3 s | $0.078 | none | rejected: latency and cost |
| Other vendor frontier | pass (regional) | 98.6% | 9.7 s | see vendor page | none | rejected: latency |
| Self-hosted open weights on-prem | pass | not run | not run | unknown until measured | high | deferred: no MLOps team; revisit at volume trigger |

## Consequences
- Gain: inside every budget with margin; one endpoint, one billing line, one
  audit trail; escalation route absorbs the mid tier's dispute-letter misses.
- Pay: 10% regional premium on every token; a router whose escalation rate
  must be measured (target 12%, alarm at 25%); two model retirements to track.
- Depend on: the regional endpoint's continued availability in the EU region;
  the vendor's "not before" retirement commitments; the golden set staying
  representative (re-sampled quarterly by claims QA).

## Review condition
Reverse or revisit when ANY of:
- Monthly golden-set pass rate for the primary route falls below 97%, or the
  hallucinated-clause rate exceeds 0.5%, on two consecutive runs.
- Measured escalation rate exceeds 25% for a month (the cascade is then costing
  more than the large tier alone would).
- Monthly drafts exceed 400,000 (re-run the self-host break-even from 08-cost-model.md).
- The vendor publishes a retirement notice for a pinned model, or the regional
  premium or list price changes by more than 20%.
Next scheduled review: 2026-10-01, ahead of the earliest pinned retirement date.

Every number in that ADR is either a budget from Lesson 02 or a measurement from the golden set, and the review section names the observation that would send the team back to this page. That is what makes it a decision record rather than a preference.

Faded case

Skip to the open case if this is familiar.

Lumen sells an AI support agent to SaaS companies: twelve people, one engineer owning everything, US-hosted, multi-tenant, revenue growing by a fifth every month, margin the number that matters this year. Its NFR sheet has no residency row yet (that arrives when the first European customer signs) and its accuracy budget is a resolution rate on a 400-conversation golden set built from real tickets. Latency is interactive: customers are typing.

Illustrative composite. Lumen's traffic: 1.5 million conversations a month, six turns each. Golden set results: small tier resolves 71%, mid tier 84%, large tier 87%; the budget is 82%. p95 per turn: small 1.1 s, mid 2.4 s, large 5.0 s against a 3 s budget. Around 18% of conversations involve billing disputes or account changes where the small tier's resolution rate collapses to 40%.

The elimination gates remove nothing: no residency row, hosted vendors only, text input, JSON output for the tool calls. The eval gate removes the small tier as a primary. The latency gate removes the large tier for interactive turns. That leaves the mid tier as primary, exactly as at Harbor, and the same escalation pattern for the hard share.

The one decision left to you: Lumen's engineer proposes the reverse cascade, small tier first on every turn and escalation to the mid tier when the small model's confidence is low, because the small tier is a fifth of the price. Write down whether you accept it and what number you need to see first.

Open case

An agency is building a retail catalogue assistant for a mid-size furniture chain: shoppers ask questions in natural language, the assistant answers from the catalogue and current stock, and it must return a structured product list the storefront renders. The chain operates in the UK and Ireland, has no in-house ML team, and is paying a fixed price for the build with a monthly run-rate cap. Peak traffic is Saturday afternoons and the two weeks before a bank holiday. The agency has an existing preference for one vendor's frontier model because "it demos best". Produce the elimination gates, the order of the remaining decisions, the routing sketch and the review condition. The first quiz item grades your answer.

Where this breaks

The rule assumes you have a golden set. If Lesson 02 produced budgets but no eval yet, you can still run the elimination gates, but the capability gate is a placeholder, and the ADR must say so with a date by which the golden set exists. Shipping on vendor benchmarks with a promise to "eval later" is the failure in the opening paragraph, delayed.

The rule assumes the budgets are honest. A cost-per-task budget set from a demo with a 2,000-token context is not the budget for production with retrieval, tool calls and retries. Price the p90 task, and re-price whenever the context specification (Lesson 05) or the tool catalogue (Lesson 07) grows. A model chosen on the median task fails on the tail.

Cascades break caches. Splitting traffic across two models means each keeps its own prefix cache, and the cheap model's warm prefix does the frontier model no good. On a workload where a cached prefix is most of the input, a single model at lower effort can beat a two-model cascade on cost. Lesson 05 gives you the arithmetic; run it before the routing sketch is final.

Deprecation windows are real but not symmetric. A "not before" date protects you from early retirement, not from price changes, regional availability changes or a quiet drop in quality after a model update. The review condition has to watch the golden set on a schedule, not the vendor's announcements page.

Key Takeaways

• Eliminate on residency, licence, modality and context first; score only the survivors.
• Capability is a gate on your own golden set, not a rank on a public leaderboard.
• Rank passing models by p95 latency per route, then by cost per task at p90.
• Try the effort setting and batch before a router; measure the escalation rate if you route.
• Record the choice as an ADR whose review condition names the observation that reverses it.
• Pin model IDs with their earliest retirement dates and schedule the migration eval.

What You Have Now

A model-selection ADR for your case with the review condition that would reverse it, plus a routing sketch. It lives at dossier/adr/0001-model.md, with the routing sketch either inside the ADR's decision section or beside it as dossier/adr/0001-routing.yaml; the lab's checker looks for both at those paths. Its inputs are the budgets in 01-nfr-budgets.md and the constraint register in 02-constraints.md; its outputs feed the context specification in Lesson 05 (which model sets the cache minimums), the cost model in Lesson 11 (which pinned prices and multipliers apply) and the rollout plan in Lesson 15 (which retirement dates force a migration eval). It serves the Decide stage: the first choice in the system that a client can read, question and, when the condition fires, reverse without an argument.

Example Files

The blank ADR, to copy into dossier/adr/0001-model.md:

# ADR-0001: Model selection for <system name>

Status: proposed | accepted | superseded by ADR-NNNN
Date: YYYY-MM-DD
Owner: <architect>          Reviewers: <security, finance, product owner>

## Context
- Tasks the model performs (one line each, with the route each belongs to):
- Budgets this decision must satisfy (copy the rows from 01-nfr-budgets.md):
  accuracy / hallucination tolerance, p95 latency per route, cost per task at
  p90, residency, risk class.
- Hard constraints (from 02-constraints.md): residency, licence, modality,
  largest single input after retrieval, structured-output guarantees.
- Eval used: name, size, owner, date last run, where it lives.

## Decision
- Primary model per route, and how it is reached (endpoint, region, tenancy):
- Routing or cascade, with the trigger for each escalation:
- Effort / thinking setting per route:
- Pinned model IDs, each with its earliest published retirement date:

## Alternatives considered

| Option | Residency/licence | Golden-set pass | p95 latency | Cost per task (p90) | Ops burden | Verdict |
|---|---|---|---|---|---|---|
| | | | | | | |

## Consequences
- What we gain:
- What we pay (premiums, a router to measure, retirements to track):
- What we now depend on (endpoints, vendor commitments, eval freshness):

## Review condition
Reverse or revisit this decision when ANY of the following is observed:
- <a golden-set threshold and how many consecutive runs>
- <an escalation or consult rate threshold>
- <a volume threshold that re-opens self-hosting>
- <a vendor event: retirement notice, price change, regional change>
Next scheduled review: YYYY-MM-DD (before the earliest pinned retirement date,
contract renewal or volume threshold, whichever comes first)

The routing sketch, to copy beside it:

# adr/0001-routing.yaml — one entry per route; the gateway (Lesson 08) reads this
routes:
  - name: classify_inbound
    trigger: every inbound document
    model: small-tier
    mode: batch            # not interactive; batch pricing applies
    effort: low
    budget_p95_latency_s: 60
  - name: draft_letter
    trigger: handler requests a draft
    model: mid-tier
    mode: interactive
    effort: default
    budget_p95_latency_s: 6
    escalate_when:
      - document_class in [dispute, complaint]
      - draft.confidence < 0.7
      - citation.unresolved == true
    escalate_to: review_draft
  - name: review_draft
    trigger: escalation from draft_letter
    model: large-tier
    mode: queued           # output lands in the legal review queue
    effort: high
    budget_p95_latency_s: 20
measure:
  escalation_rate: { target: 0.12, alarm: 0.25, window: 30d }
  golden_set: { schedule: monthly, threshold_pass: 0.97, threshold_halluc: 0.005 }

Further Reading

• Anthropic pricing: the one page with tiers, cache, batch and residency multipliers; check the date before you quote it.
• Optimizing for cost and intelligence: the effort dial and measured executor/advisor and orchestrator/worker configurations.
• FrugalGPT: the cascade pattern with confidence-based escalation and its reported cost reductions.
• RouteLLM: learned routers between a strong and a weak model, and why only a minority of queries need the strong one.
• Stanford AI Index 2025: why you budget on capability tier rather than today's list price.
• Llama 4 Community License: the clause your client's counsel will ask about.

Exercises

1. Run the funnel on your case. Take the constraint register from Lesson 03 and the budgets from Lesson 02. List every candidate model you would consider, then apply the four elimination gates in order and write one line per elimination naming the row of the register that did it.
2. Price the p90 task. For your primary candidate, write down the token counts of a p90 request (cached prefix, fresh input, output), look up the current prices on the vendor's page, apply the multipliers that apply to your topology, and compute cost per task. Repeat for the next tier up. Decide whether an escalation route is worth its measurement burden.
3. Write the ADR. Fill the template above for your case, including at least three review conditions, one of which is a vendor event with a date. Have someone who was not in the room read only the review section and say what would make them reopen it.

Check your understanding