Lesson 11. The Cost Model That Survives the CFO

Harbor Mutual's pilot budget was approved on a single line: two cents a letter. The vendor had multiplied one model call by one list price and the steering committee had multiplied that by the monthly letter count. The first invoice for the pilot month arrived at roughly three times the model spend the line implied, and when the finance analyst added the platform, the evaluation runs and the half-engineer who kept the thing alive, the fully loaded cost of a letter in month one came out above thirty times the number on the slide. Nothing had gone wrong technically. The assistant made three model calls per letter, not one; a third of letters triggered a booking loop with two more calls; retried calls paid their tokens twice; a judge sampled live traffic; and nobody had switched caching on. The CFO froze the second phase until someone could show the arithmetic. Decide now: when you price a task, which of those lines do you include, and which number (the average letter or the expensive one) do you put in front of the CFO?

Illustrative composite. The two-cent line was the draft call alone: about 6,000 input and 700 output tokens on Sonnet 5. Model spend with every call counted and no caching came to about six cents a letter; the fully loaded month-one figure, at 15,000 letters, was about sixty-nine cents. The numbers are a teaching device, not a measurement from a real insurer.

Include every line, and put the p90 letter in front of the CFO with the mean beside it. This lesson's deliverable is 08-cost-model.md in your Dossier. A cost model for your case: cost per task with the math shown, a twelve-month TCO under three scenarios, and the self-host break-even. It is the second document of the Prove stage of Frame / Decide / Prove / Run: the number the buyer signs, with the arithmetic that makes it auditable.

What is actually going on

A task is not a request. The unit the CFO pays for is a task (a letter drafted, a ticket resolved, a visit booked) and a task is a tree of calls: a routing call, a drafting call, zero or more tool-loop calls, a guard call, a sampled judge call, and the retries hiding inside each of them. Every call bills four kinds of token at four prices: uncached input, cache writes, cache reads, and output. Output is the expensive kind. On every row of the Anthropic price table, output costs five times input (Sonnet 5 at $2 in and $10 out; Opus 5 at $5 and $25; Haiku 4.5 at $1 and $5, per million tokens on the Anthropic pricing page, observed 2026-09-09). Agent loops are output-heavy and tool-result-heavy, so a per-request estimate built on one drafting call undercounts by construction.

Written out, the cost of one task is a sum, then three multipliers:

cost_per_task = Σ over calls of
    ( uncached_in × p_in
    + cache_write × p_write
    + cache_read  × p_read
    + out         × p_out )
  × (1 + retry_rate)
  × residency_uplift
  × speed_tier

The picture below is that formula drawn as a flow. Read it left to right and count the boxes that a "one call" estimate never sees.

Five boxes feed the sum where the slide had one, and two multipliers sit after them. The tool-loop box is the one that breaks averages: a letter with a booking costs almost twice a letter without, so the distribution of task cost has a long right tail.

That tail is why the CFO number is the p90, not the mean. The mean times the volume is the expected bill and belongs in the TCO. The unit price you commit to in a contract, or compare with a vendor's per-task quote, is the p90 task, because that is the number that decides whether an unlucky month still fits the budget. Quote both, label both.

The rule and its condition

Use the three levers in a fixed order, cheapest to implement first, and model twelve months under three scenarios before you quote any number. Use prompt caching whenever a prefix above the model's minimum cacheable size repeats within the TTL; otherwise reorder the prompt until one does. Use routing to a smaller model only after you have tried a lower effort setting on the same model, because a cascade fragments the cache. Use batch for anything nobody is waiting on. Multipliers below are Anthropic's, read from the pricing page on 2026-09-09; OpenAI and Google publish their own on their pricing pages and this lesson quotes none of their figures.

Two facts govern the order. Caching is free money with a break-even of one hit for a 5-minute write and two hits for a 1-hour write, and steady-state saving of 90% on the reused prefix (97.5% on Fable 5.1), from the same pricing page. Routing has larger headline numbers in published work (RouteLLM reported over 85% cost reduction on MT-Bench at 95% of frontier quality with about 14% of calls going to the large model, in its own announcement, observed 2026-09-10), but those are benchmark numbers on someone else's traffic; production teams report 40 to 70% (routing overview, observed 2026-09-09), and every routed call is a call that no longer shares the main model's cache.

Worked case

Harbor Mutual's assistant drafts claim correspondence grounded in policy wording and proposes adjuster visits through a booking tool. The context spec from lesson 5 fixed a 6,000-token cacheable prefix on the drafting call; the agent spec from lesson 7 fixed the tool loop; the topology ADR from lesson 9 placed inference on a regional endpoint in the EU tenancy, which carries the +10% regional uplift listed on the Anthropic pricing page, observed 2026-09-09. The CFO wants three things: the cost of a letter with the arithmetic, a twelve-month total under a base, a high-adoption and an adverse scenario, and the volume at which running their own models would be cheaper.

Illustrative composite. Inputs to the model: 15,000 letters a month for the three pilot months, 30,000 for the next three, 45,000 at steady state; a booking loop on 30% of letters; a 4% retry rate; 5% of live letters sampled by an Opus 5 judge; cache hit rate 90% at the 5-minute TTL (about two letters a minute in business hours keeps the prefix warm). Fixed monthly lines: 20 offline gate runs at about $25 each from lesson 10 ($500); gateway, tracing and vector-store hosting in the EU tenancy ($2,100); two adjuster-days of labelling ($1,200); half an engineer for operations and evals ($6,000). Every figure is modelled; the prices are list prices from the Anthropic pricing page, observed 2026-09-09.

The options were the vendor's two-cent line, a "just measure it in production" plan, or a model built line by line from the specs already in the Dossier. Decide now: what is the cost of one letter, and what does the twelve-month total look like when caching fails?

The architect built the model in four steps.

1. The cost-per-task worksheet. One row per call in the task tree, tokens split by price class, calls per letter as an expectation. Cached tokens are priced as 0.9 × read plus 0.1 × write at the 5-minute rate.

The drafting row alone:

draft call, Sonnet 5, 5-minute cache, 90% hit
  prefix reads   0.9 × 6,000 × $0.20 / 1M  = $0.00108
  prefix writes  0.1 × 6,000 × $2.50 / 1M  = $0.00150
  dynamic input        6,000 × $2.00 / 1M  = $0.01200
  output                 700 × $10.0 / 1M  = $0.00700
  subtotal                                  = $0.02158
  × 1.04 retries                            = $0.02244   -> $22.44 per 1,000 letters

same call with no caching
  input               12,000 × $2.00 / 1M  = $0.02400
  output                                    = $0.00700
  × 1.04                                    = $0.03224   -> $32.24 per 1,000

mean letter (all rows, uplift applied)      = $0.0352
plain letter (no booking, not judged)       = $0.0276
p90 letter (booked, judged)                 = $0.0758   -> 2.2 × the mean

The p90 letter is the number that goes in the contract's unit-price clause, at about seven and a half cents; the mean, three and a half cents, multiplies the volume in the TCO. Without caching the mean would have been six cents, which is the figure the pilot invoice had shown.

2. Three scenarios, twelve months. Base as above. High adoption: volume reaches 60,000 by month six and the booking rate rises to four letters in ten. Adverse: the volume stays base, but a request ID lands in the prompt and the hit rate drops to six in ten, provider capacity errors push retries to one call in ten, and legal asks for a judge sample twice the size.

The picture below plots the three model-spend lines by month. Look at the gap between base and adverse in the steady-state months; that gap is what the three levers are worth when they stop working.

Illustrative composite. Every value in the chart is a modelled monthly figure from the table above, in US dollars, at Anthropic list prices observed 2026-09-09.

The chart says two things the CFO needed to hear. The adverse scenario costs about a third more in model spend than base, so the levers are worth protecting with a gate on cost per task in CI (the worksheet's own rule: fail a prompt change that raises cost per task by more than a tenth). And model spend is a tenth of the twelve-month total; the fixed lines, the half-engineer above all, decide the TCO at this volume. A cost model that only prices tokens flatters the project.

3. The self-host break-even. Stated as a formula, then filled in with badged numbers, because GPU prices move monthly and the throughput figures available were secondary sources.

C_self   = N_gpu × r_gpu × 730 h + C_ops + C_platform        (per month)
capacity = N_gpu × tokens_per_s_per_gpu × 3,600 × 730 × u      (u = achieved utilisation)
p_api    = blended API $ per token for the SAME quality (must pass the lesson-10 gate)
V*       = C_self / p_api                                      (break-even tokens per month)
self-host wins only if V* <= capacity at the u you can actually sustain

Illustrative composite. Harbor's steady state is about 22,000 tokens per letter, 45,000 letters, so about 1.0 billion tokens a month at a blended API price near $1.60 per million: about $1,600 a month. Two H100-class GPUs at an assumed $4 an hour each come to $5,840 a month before people; add a quarter of an engineer at $3,000. V* = $8,840 / $1.60 per million = 5.5 billion tokens a month, five and a half times the steady-state volume, and that assumes an open-weight model passes the lesson-10 gate at Harbor's quality bar. GPU hourly rates and the tokens-per-second figures behind them were secondary-source numbers on 2026-09-09; put the vendor's own price page and date in your Dossier before you use any of them.

Break-even for Harbor is not a cost question at all; if it ever self-hosts it will be because the residency constraint tightens, and the topology ADR from lesson 9 already records that condition. The model states the volume trigger (5.5 billion tokens a month) so the review has a number to watch.

4. Latency, capacity and controls. The cost model carries a latency budget because caching changes it: a cached prefix shortens prefill, and prefill is what time-to-first-token is made of. The SLO is on p95 time-to-first-token and p95 end-to-end separately, streaming on from the first token.

Illustrative composite. Budget to first visible text: gateway and auth 50 ms, retrieval with rerank 250 ms, input guard 120 ms, model time-to-first-token with the cached 6,000-token prefix 500 to 800 ms (measure it; the prefix length dominates), then streaming. At about 60 tokens a second a 700-token letter takes twelve seconds to finish, which adjusters accepted once the first paragraph appeared inside 1.2 s.

Capacity is checked against the provider's limits, which are per model and per organisation across requests per minute, input tokens per minute and output tokens per minute; cache reads are exempt from the input-token limit on most models, and a 429 (your limit) is a different failure from a 529 (the provider's capacity), per the rate-limits documentation, observed 2026-09-09. Harbor's business-hours peak of about thirteen letters a minute is roughly 290,000 tokens a minute, of which about half are cache reads; the model records that figure next to the tier's published limit so the runbook in lesson 14 has a threshold.

Controls, all at the gateway from lesson 8: a hard monthly cap per tenant and per API key, alerts at 50, 80 and 100% of the monthly budget, and attribution tags on every call (tenant, feature, prompt version, model, cache hit). The unit metrics the FinOps Foundation recommends for AI spend (cost per query, per user per month, per workflow completion) are the ones the dashboard reports, per FinOps for AI, observed 2026-09-09.

The rejected options had prices. The two-cent line would have put a number in the contract that the first invoice contradicted, which is what happened. "Measure it in production" would have produced a number three months in, after the pilot budget had been spent, with no way to say which lever had failed when it rose.

Faded case

Skip to the open case if this is familiar.

Illustrative composite. Lumen is fictional; its growth rate and cost lines are a teaching device.

Lumen sells an AI support agent to SaaS companies, multi-tenant, growing twenty percent month on month, with one engineer and margins that decide whether the next hire happens. Their per-seat price is fixed; their cost per conversation is not. Twelve months of 20% growth is a factor of 1.2 to the twelfth power, about 8.9, so a cost line that is comfortable today is a margin problem by spring unless cost per conversation falls faster than volume rises.

Illustrative composite. Today: 400,000 conversations a month, three calls each, a 6,000-token policy prefix, 2,500 tokens of ticket and history, 600 tokens out, all on Sonnet 5, no caching. Prefix cost 7.2 billion tokens at $2 per million is $14,400 a month; with caching at a 90% hit rate it falls to about $3,100. Dynamic input adds $6,000 and output $7,200, for about $16,300 a month, or four cents a conversation. Prices from the Anthropic pricing page, observed 2026-09-09.

The shape differs from Harbor in two ways. Fixed costs are a small share (one engineer, no adjusters, no regional uplift), so the levers are the whole story, and growth means the order you pull them in matters more than which ones you pull. The engineer has a week and three candidates: switch caching on and verify it, move the first classify-intent call to Haiku 4.5, or put a semantic cache in front of the gateway on the theory that support questions repeat. One decision is left to you: which goes first, which goes second, and what has to be measured before the third is allowed?

Open case

An agency is building a retail catalogue assistant for a fashion retailer: shoppers ask about fit, materials and availability across 40,000 products, the assistant calls a stock-lookup tool and a size-recommendation tool, and traffic is seasonal, with a November peak the retailer says is eight times a quiet month. The retailer wants a fixed monthly fee for the assistant and a guarantee that the agency will not come back for more money in December. The agency's proposal prices it at the average conversation cost of a September test week, on the main model, no caching, one call per conversation, and adds a fifteen percent margin. Write the cost model: the task tree, the levers in order, the three scenarios with December in them, the number you would put in the fixed fee, and the clause that protects the agency. The first quiz item grades your answer.

Where this breaks

Prices and tokenisers move under the model. The Stanford AI Index reports the cost of running a model at a fixed capability level fell more than 280-fold between late 2022 and late 2024 (2025 AI Index, observed 2026-09-09), and the same pricing page above notes that Anthropic's 4.7-and-later tokeniser produces about 30% more tokens for the same text than earlier models. Model on capability tier, keep the router so a cheaper model can drop in, and re-count tokens with the provider's counter on every model migration. A cost model without a dated price sheet attached is fiction within a quarter.

Utilisation assumptions hide in self-host numbers. Headline dollars per token for a self-hosted model assume near-peak batching. At low or bursty utilisation the cost per token multiplies, and the ops headcount is real money every month whether or not traffic arrives. The formula above puts utilisation in the capacity line on purpose; a break-even that assumes full utilisation is not a break-even.

Cascades fight caches. Every routed call that escalates pays uncached input on a second model. A cost model that adds the routing saving and the caching saving as if they were independent overstates both. Model the escalated share explicitly.

The mean hides the loop. An agent that usually takes three steps and sometimes takes twelve has a mean that looks fine and a p90 that does not. Without the loop and token budgets from the agent spec, the tail is unbounded and so is the bill; the cap at the gateway is the last line, not the first.

Key Takeaways

• Price the task tree, never the request: route, draft, tools, guard, judge, retries, uplift.
• Quote the p90 task as the unit price and the mean times volume as the bill.
• Pull the levers in order: cache (verify reads), route (after lower effort), batch (nobody waiting).
• Model twelve months under base, high and adverse; the adverse case is your own design with a lever broken.
• Write self-host break-even as a formula with utilisation in it; badge every GPU number with a date.
• Gate cost per task in CI and cap spend per tenant at the gateway.

What You Have Now

A cost model for your case: cost per task with the math shown, a twelve-month TCO under three scenarios, and the self-host break-even. It lives at dossier/08-cost-model.md, beside the context spec from lesson 5 that fixes what is cacheable, the agent spec from lesson 7 that fixes the tool loop, and the topology ADR from lesson 9 that fixes the uplift. It serves the Prove stage: the go/no-go memo in project 3 quotes its p90 and its twelve-month total, the runbook in lesson 14 inherits its capacity thresholds, and the rollout plan in lesson 15 uses the break-even volume as one of its review conditions.

Example Files

The cost-per-task worksheet, as a table you fill in and the formulas that produce each column. Copy both into 08-cost-model.md.

# 08 · Cost model — [system name]

Price sheet: [provider pricing page URL], observed [date]. Tokeniser: [provider count_tokens on date].

## 1. Cost per task
| Call | Model | Cacheable prefix | Dynamic input | Output | Calls per task | $ per 1,000 tasks |
|---|---|---|---|---|---|---|
| [route] | [small model] | [tokens] | [tokens] | [tokens] | [1.0] | [ ] |
| [main] | [main model] | [tokens] | [tokens] | [tokens] | [1.0] | [ ] |
| [tool loop] | [model] | [tokens] | [tokens] | [tokens] | [steps × share of tasks] | [ ] |
| [guard] | [small model] | [tokens] | [tokens] | [tokens] | [1.0] | [ ] |
| [online judge] | [judge model] | [tokens] | [tokens] | [tokens] | [sample rate] | [ ] |
| Sum × (1 + retry rate [0.04]) | | | | | | [ ] |
| × residency uplift [1.10] | | | | | | **[ ]** |

Mean task: $[ ]  ·  Plain task: $[ ]  ·  p90 task (worst common path): $[ ]  ·  Unit price quoted: p90.

## 2. Scenarios, twelve months
| Month | Volume base / high / adverse | Model spend base / high / adverse | Fixed | Total base |
|---|---|---|---|---|
| 1..12 | [ ] | [ ] | [ ] | [ ] |
Adverse definition: [which lever fails, retry rate, judge rate].
Fixed lines: [gate runs], [platform], [labelling], [people].

## 3. Self-host break-even
C_self = [N] × [$/h, source, date] × 730 + [ops] + [platform]  ;  p_api = [$/M blended]
V* = C_self / p_api = [tokens/month]  ;  capacity at u = [ ]  ;  verdict: [ ]  ;  review trigger: [volume or constraint]

## 4. Latency budget (p95): gateway [ ] + retrieval [ ] + guard [ ] + TTFT [ ] ; stream from first token
## 5. Capacity: peak tokens/minute [ ] (cache reads [ ]) vs tier limit [ ] on [rate-limits page, date]
## 6. Controls: caps per [tenant/key] · alerts at [50/80/100%] · tags [tenant, feature, prompt version, model, cache hit]
## 7. CI rule: fail any change that raises cost per task by more than [10%] against this sheet

Formulas behind the columns

per-call cost
  cached_prefix_cost = hit × prefix × p_read + (1 − hit) × prefix × p_write
  call_cost          = cached_prefix_cost + dynamic_in × p_in + out × p_out
  row_cost           = call_cost × calls_per_task

task
  mean_task = (Σ row_cost) × (1 + retry_rate) × residency × speed
  p90_task  = same sum over the worst common path (tool loop taken, judged)

month
  model_spend  = volume × mean_task
  total        = model_spend + fixed
  TCO_12       = Σ months

cache break-even (5-minute TTL)
  one hit repays one write: write 1.25× vs read 0.1×  ->  saves 0.9 × prefix × p_in per hit after the first

CI cost gate
  Δ = (mean_task_new − mean_task_sheet) / mean_task_sheet ; fail if Δ > 0.10

Further Reading

• Anthropic pricing — the price sheet, cache and batch multipliers, uplifts and the tokeniser note this lesson's arithmetic uses.
• Anthropic prompt caching — TTLs, minimum cacheable prefix per model, and the response fields that prove a hit.
• Anthropic rate limits — the three limits per model, what cache reads count against, and 429 versus 529.
• FinOps Foundation: token economics — the unit metrics finance teams expect for AI spend.
• Stanford AI Index 2025 — the inference-cost decline that argues for modelling on capability tier.
• OpenAI pricing and Gemini API pricing — read the current figures there; this lesson quotes none.

Exercises

1. Draw the task tree for your own case from the agent spec and fill the worksheet's first table with real token counts from the provider's counter. Compute the mean and the p90 task and write one sentence on why they differ.
2. Define your adverse scenario as your own design with one lever broken (name the lever and the mechanism), then compute twelve months under base, high and adverse. State what the adverse gap is worth as a CI cost gate.
3. Write the self-host break-even for your case as the formula with every number dated and sourced, and state the review trigger (a volume or a constraint) that would reopen the topology ADR.

Check your understanding