Lesson 09. Where It Runs Decides What It Costs

A Nordic bank built a document assistant for its onboarding team: upload a customer's proof-of-address, ask what it shows, get a structured answer. The team used a first-party model API because it was fastest to integrate and the vendor's terms said no training on customer data, which satisfied the engineer who read them. Six weeks before launch, the bank's data-protection officer read the processor addendum the bank had actually signed. It required processing of customer identity documents inside the European Economic Area, and the API in use processed in the United States with no in-region option the bank qualified for. The gateway had to move, the model had to move to a private endpoint in an EU region, the logging pipeline had to be rebuilt to stay in-region, and the eval suite had to be rerun on a model that was one release behind. Decide now: which single line in the constraint register from Lesson 03, had it been read before the first commit, would have prevented all of it?

Illustrative composite. The rebuild took the team a quarter and cost more than the original build, because the residency constraint touched the model, the logs, the index and the fallback path at once. The bank is fictional; the shape of the failure is common.

This lesson gives you a topology ADR and a tenancy model for your case, with the residency and isolation requirements each satisfies. It is the sixth and last decision of the Decide stage in the Frame / Decide / Prove / Run loop, filed as adr/0003-topology.md, and it fixes where the gateway from Lesson 08 and the endpoints it fronts actually live.

What is actually going on

Topology is three questions that teams answer as one. Where do the model weights run? Where do prompts and outputs go, and under whose terms? Where do the logs, the index and the cache land? The Nordic bank answered the first (a vendor's servers), assumed the second (the terms looked fine), and never asked the third. Residency clauses bind all three, and so do data-classification policies, so a topology chosen by the model's capability alone will be rewritten by whichever constraint was read last.

There are four rungs. A hosted API, where the vendor runs the weights under its commercial terms and residency options. A private endpoint, where a hyperscaler runs the same or similar weights inside your cloud tenancy and region (Bedrock, Azure AI Foundry, Vertex AI). Self-hosted open weights on GPUs you rent or own. Edge or air-gapped, where the model runs on the device or inside a network that reaches nothing. Each rung up buys control over data and residency and pays in model choice, time to value and operations. The industry's own reference frameworks agree on the shape; the AWS Well-Architected Generative AI Lens, released in April 2025, organises the same trade-offs into six pillars across six lifecycle phases (AWS announcement, observed 2026-09-09).

The tree below is entered from the constraint register, not from a model leaderboard. Each diamond is a question the register answers, and the rung you reach is the lowest one the register permits.

Two things in the picture deserve a second look. Rung three has two entrances: the register can force it (no acceptable cloud region) or the cost model can invite it (steady volume past break-even), and the second entrance is only open if a platform team exists to walk through it. And there is no arrow from rung four back down: an air-gapped deployment is a commitment to a model that ages until someone carries a new one in.

The rule and its condition

Choose the lowest rung the constraint register permits, before the first line of code, and record in the ADR the constraint that forbids each lower rung; climb later only when a constraint or a measured break-even forces it. The condition is legible in your own Dossier: if the ADR cannot name the register line that forbids rung one, you belong on rung one.

Most regulated enterprises land on rung two with rung one kept for features the private endpoint does not yet carry, and route between them by data classification: public or anonymised content to the cheapest capable API, confidential content to the in-region private endpoint, restricted content to whatever the register demands. That routing is a gateway rule from Lesson 08, and it has a residency guard on the fallback edge, because the commonest violation is a failover that silently crosses a geography during an incident.

The terms, dated

Vendor data terms are contract facts with expiry dates, and the ADR carries them with the date you checked. The table below is what the three first-party providers published as of the observation date; every cell moves, so re-verify at contract time and write the new date into the ADR.

Sources, all observed 2026-09-27: Anthropic's commercial terms and pricing page, the latter fetched directly, which lists the inference_geo: "us" option at a 1.1x multiplier on all token categories for recent models and a 10% uplift for regional endpoints on the hyperscalers; a report on the covered-model retention exception, plus the current Covered Models list (four models: Fable 5, Fable 5.1, Mythos 5, Mythos 5.1); OpenAI's enterprise privacy page and data-residency announcement; Google's Vertex AI data-governance page. The default retention window at Anthropic and OpenAI was thirty days on the observation date; treat that as a value to re-read, not a constant.

The hyperscaler options have caveats of their own. Azure OpenAI Data Zones keep processing inside a geography (a US zone, an EU zone) rather than a single region (Azure announcement, observed 2026-09-09). Bedrock's geographic cross-region inference keeps traffic inside a geography but may move prompts and outputs between regions within it (AWS documentation, observed 2026-09-09). Whether "inside the EU" satisfies a clause that says "inside Ireland" is a question for counsel, and the ADR records the answer with the date.

For a review, the six Well-Architected pillars translate into six questions you can ask any topology: can we evaluate and version prompts in this environment (operational excellence); are injection, tenant isolation and secrets handled here (security); what is the fallback and the quota when this endpoint fails (reliability); what is time-to-first-token with our cached prefix here (performance); what does a task cost here at our utilisation (cost); and are we running the smallest model that passes the eval (sustainability). Lesson 14's runbook and Lesson 15's review checklist reuse the same six lines.

Tenancy is a design, not a flag

Multi-tenancy is where topology meets the tool catalogue. A tenant is any party whose data must not reach another party through your system: a customer of a SaaS, a business unit of a bank, a broker inside an insurer. Isolation has to hold at every layer that touches the model, and the layers are more numerous than most designs admit.

Whatever the tier, the tenant claim comes from a signed token verified at the gateway and the retrieval layer, never from the prompt, which Lesson 08's rule already implied. Prompts are tenant-scoped, so a tenant's custom instructions cannot leak into another's context. Logs are partitioned by tenant, so a support engineer reading one tenant's trace cannot see another's. Memory, if the agent has any, is tenant-scoped grounding data. And the cache deserves its own paragraph.

A prefix cache shared across tenants is a side channel. If two tenants' requests can hit the same cached prefix, one tenant can detect from response timing whether another tenant sent a request with that prefix, which turns the cache into an oracle for other people's prompts; researchers demonstrated timing-based cache detection against production APIs in early 2025 (Gu et al., Auditing prompt caching in language model APIs, observed 2026-09-09). Vendor caches are scoped to your organisation, which is enough when your organisation is the only tenant; when it is not, put the tenant identifier inside the cached prefix or use per-tenant cache scoping, and on a self-hosted engine with a shared prefix cache, treat cross-tenant sharing as a decision that needs a threat model, not a default.

Per-tenant quotas and cost attribution come from the gateway, and they are the reason tier one is still legitimate for small tenants: a virtual key per tenant with a budget, alerts at a threshold, and tags on every call give you a showback report the finance team can read and a kill switch per tenant when a loop runs away. Read the diagram for the one place a request is turned into a tenant, and the three things that then diverge.

If a request can reach an index, a prefix or a cache without passing through the verified claim, the diagram for your system has an edge this one does not, and that edge is the breach.

What self-hosting actually costs

The self-hosting argument is always made with a headline price per million tokens, and the headline assumes the GPUs are busy. Cost per token is the hourly rate of the hardware divided by the tokens it served that hour, so a GPU that idles most of the day charges several times its headline rate for the tokens it did serve. Add the people: capacity planning per model, cache monitoring, engine upgrades, licence tracking for the weights, and a security posture for the model files. A break-even sketch belongs in the topology ADR, and Lesson 11 builds the full version; the sketch below shows the shape.

Illustrative composite. Take a steady document-extraction workload of three billion input and three hundred million output tokens a month, acceptable from a small hosted model. On the Anthropic sheet fetched 2026-09-09 (pricing), Haiku 4.5 through the batch endpoint is $0.50 per million input and $2.50 per million output, so the API bill is about $2,250 a month. For the self-hosted side, GPU hourly rates from the aggregators observed on the same date are unverified and ranged from about $2 to $12 per H100-class GPU-hour depending on provider and commitment, so use your own quote; at a mid-range $4 per hour, two GPUs run around $5,800 a month before any person touches them, and a quarter of an engineer adds several thousand more. At 40% utilisation the effective cost per token is 2.5 times the busy figure; at 10% it is ten times. The workload stays on the API until volume grows by an order of magnitude, or until the register forbids the cloud, and the ADR says which of those two would reopen the decision.

The sketch teaches the two variables that matter: which API model you are replacing, and how steady the traffic is. Break-even claims that quote a single monthly token volume without those two are not usable.

Worked case: Harbor Mutual

Harbor Mutual's constraint register carries the lines that decide topology. R-07: EU policyholder data is processed inside the EU; UK policyholder data may be processed in the EU or the UK under the group's transfer assessment, dated and owned by counsel. R-08: claim documents are classified confidential and may not be sent to a vendor under default retention. R-09: the group has an existing cloud tenancy with a platform team of four, none with GPU operations experience. R-10: the policy archive is on-premises and stays there for the pilot. The NFR budgets add a monthly volume that is modest for a pilot and grows if the rollout in Lesson 15 succeeds. Legal has asked for the answer to "where does a claimant's letter go" in one sentence.

Three topologies were costed. Option A: the first-party API with the US inference option. Option B: a private endpoint in the tenancy's EU region, with geographic inference kept inside the EU, the gateway and all logs and indexes in the same region, and the UK handled under R-07's transfer assessment. Option C: self-hosted open weights on GPUs next to the on-premises archive. Decide now, and name the register line that eliminates each option you reject.

Option A fails R-07 on its face: the residency option available at the time was US-only, and EU policyholder data cannot go there. It also fails R-08 unless zero retention is approved, and the model family Harbor Mutual wanted was among those the vendor excluded from zero retention on the observation date. Option C satisfies R-07 and R-08 and fails R-09: four platform engineers with no GPU experience would be operating a serving stack, a model upgrade process and weights security for a pilot whose volume does not come close to keeping a GPU busy. It also lags the model choice the eval in Lesson 10 will need. Option B satisfies R-07 and R-08 by construction: prompts and outputs stay in the tenancy's EU geography under the hyperscaler's terms, the gateway and logs live in the same region, and the retrieval index for the archive is built in-region from the on-premises source under R-10. The compromises are recorded: geographic inference may move a prompt between EU regions, which counsel confirmed is inside R-07; the private endpoint lags first-party features, so the risk register gains a line and the fallback chain in Lesson 14 stays in-geography only.

Illustrative composite. At pilot volume the private endpoint's regional uplift added roughly a tenth to a token bill that was small to begin with; the self-hosted option would have cost more than ten times that in GPU rental alone at the pilot's utilisation, before the platform team learned to run it. The rejected option's real cost was the quarter the Nordic bank spent: retrofitting residency after launch. Figures are a teaching device.

The ADR the architect filed follows; the blank skeleton is under Example Files.

# ADR 0003 · Deployment topology and tenancy — Harbor Mutual claims correspondence
Status: accepted · Date: 2026-09-09 · Owner: solutions architect · Reviewers: counsel, platform lead

## Context (register lines this answers)
R-07 EU residency for EU policyholders; UK under transfer assessment (counsel, dated).
R-08 Claim documents confidential; no vendor default retention.
R-09 Platform team of four, no GPU operations. R-10 Archive stays on-premises for the pilot.

## Decision
Rung 2: private model endpoint in the existing cloud tenancy's EU region, geographic
inference constrained to the EU. Gateway, logs, traces, retrieval index and cache in the
same region. Fallback in-geography only. UK traffic routed to the EU endpoint under R-07.

## Rejected
Rung 1 (first-party API): residency option US-only on the observation date; fails R-07.
Zero retention not available for the intended model family; fails R-08.
Rung 3 (self-hosted): fails R-09; pilot volume leaves GPUs idle; model choice lags the eval.

## Tenancy model
Single tenant (Harbor Mutual). Two internal populations, adjusters and brokers, are
separated by document-level ACLs (04-grounding-data.md) and per-feature virtual keys,
not by tenancy. Cache scope: organisation; prefix includes the population identifier.

## Vendor terms recorded (observed 2026-09-09)
Hyperscaler data-processing terms for the endpoint; regional uplift on the price sheet;
geographic inference may move prompts between EU regions (counsel: inside R-07).

## Review conditions (any one reopens this ADR)
- Volume passes the break-even in 08-cost-model.md for three consecutive months.
- A required model feature is unavailable on the private endpoint for more than one quarter.
- Counsel revises the UK transfer assessment.
- The vendor's residency or retention terms change (re-verify at each contract renewal).

## Evidence to collect (Prove stage)
Residency configuration exported as infrastructure code; endpoint region in every trace;
a failover drill showing no cross-geography fallback.

Faded case: Lumen

Skip to the open case if this is familiar.

Lumen runs on a hosted API in the US, with one namespace per tenant in a single vector store, one virtual key per tenant at the gateway, and the tenant identifier inside every cached prefix. Its first regulated prospect, a healthcare SaaS, has sent a security questionnaire that asks for a business associate agreement, zero retention at the model vendor, an index that no other tenant shares, and evidence that logs are partitioned. The prospect is worth a fifth of Lumen's revenue. The engineer's instinct is to self-host "so the data never leaves." The register lines are all US, no residency clause, and one engineer.

Open case

An internal platform team at a bank is building a shared LLM service for forty product teams across three countries with different residency rules. Some teams handle public marketing text; some handle customer transaction data; one handles regulated model-risk documentation that may not leave the country it is produced in. The bank has cloud tenancies in all three countries and a platform team of twelve, two of whom have run GPU clusters. Volume is large and steady in one country, small elsewhere. Choose the rung per data class and country, the tenancy tier per team type, and the review conditions. The first quiz question grades your answer.

Where this breaks

Private endpoints lag first-party APIs in features by weeks to months, and the gap is uneven: batch endpoints, file handling and newer capabilities arrive on different schedules per platform. Write the feature you depend on into the risk register with the platform's availability on the day you checked, and design the eval so it can run on the endpoint you will actually ship.

Cross-region fallback is the residency violation nobody plans. The gateway's fallback chain must carry the same geography constraint as the primary route, and the failover drill in the ADR's evidence list exists to prove it. Degraded mode for a regulated workload is a queue for a human, not a request that crosses a border.

Air-gapped deployments freeze the model. Without a plan and a budget for carrying new weights in and re-running the eval, the enclave runs a model that ages while the threat model and the workload move on.

Dedicated tenancy does not scale in people. Tier three for every tenant is a fleet of small deployments each needing upgrades, monitoring and budget alerts. Reserve it for tenants whose contracts demand it and charge accordingly; the rest live in tier two, where isolation is structural and operations are shared.

The terms move. Every cell in the vendor table has changed at least once in the year before the observation date. The ADR's date is part of the decision, and the review condition "terms change at renewal" is not a formality.

Key Takeaways

• Enter the topology tree from the constraint register; choose the lowest rung it permits and record what forbids each lower one.
• Answer three questions, not one: where the weights run, where prompts go under whose terms, where logs and indexes land.
• Date every vendor term in the ADR and re-verify at contract renewal; retention and residency options move.
• Isolate tenants at every layer: verified claim, prompt prefix, index, cache scope, key, budget, logs.
• Treat a shared prefix cache as a side channel; scope caches per tenant when more than one tenant exists.
• Cost self-hosting at your real utilisation with people included, and against the API model you would replace.

What You Have Now

A topology ADR and a tenancy model for your case, with the residency and isolation requirements each satisfies. It lives at dossier/adr/0003-topology.md, the third decision record in the pack that Project 02 assembles alongside the model and augmentation ADRs and the reference-architecture diagrams. It closes the Decide stage: the model, the context, the grounding data, the pattern, the boundaries and now the place they run are each recorded with the condition that would reverse them. Lesson 11 turns the break-even sketch into the full cost model, Lesson 13 turns the vendor terms and the residency configuration into compliance evidence, and Lesson 14 turns the in-geography fallback into a tested chain.

Example Files

# ADR 0003 · Deployment topology and tenancy — [system name]
Status: proposed | accepted | superseded · Date: [YYYY-MM-DD] · Owner: [name] · Reviewers: [counsel, platform, security]

## Context (register lines this answers)
[R-xx residency] [R-xx data classification] [R-xx operations capacity] [R-xx data location]

## Decision
Rung [1-4]: [endpoint, region or geography, deployment type]. Gateway at [place].
Logs, traces, index and cache at [place]. Fallback constrained to [geography].
Routing by data class: public → [rung]; confidential → [rung]; restricted → [rung].

## Rejected
Rung [n]: fails [R-xx] because [observable fact, dated].

## Tenancy model
Tenants: [who counts as a tenant]. Tier per tenant class: [1 | 2 | 3].
Claim source: [signed token, verified at gateway and retrieval].
Isolation per layer: prompt prefix [ ] · index [ ] · cache scope [ ] · key and budget [ ] · logs [ ] · memory [ ].

## Vendor terms recorded (observed [date])
Training: [ ] · Retention: [ ] · Zero retention: [status] · Residency option: [ ] · Price uplift: [ ]

## Review conditions (any one reopens this ADR)
- [volume past break-even for N months]
- [required feature unavailable on the endpoint for more than one quarter]
- [terms or residency options change at renewal]
- [a new tenant class with a requirement the current tier cannot meet]

## Evidence to collect (Prove stage)
[residency configuration as infrastructure code; region in every trace; failover drill result; tenant-isolation test]

| Tenant class | Tier | Index | Cache scope | Key and budget | Logs | Region |
|---|---|---|---|---|---|---|
| [standard] | 2 | namespace per tenant | per tenant | per tenant | partitioned | [ ] |
| [regulated] | 3 | dedicated, in required region | per tenant | per tenant, alerting | partitioned | [ ] |

Further Reading

• AWS Well-Architected Generative AI Lens: the six pillars applied to generative workloads; the review questions in this lesson follow its structure.
• Google Cloud generative AI architecture overview: the second hyperscaler's reference shapes, useful for comparing private-endpoint designs.
• Bedrock geographic cross-region inference: the caveat that "inside the geography" is not "inside the region," in the vendor's own words.
• Auditing prompt caching in language model APIs: the timing side channel behind the per-tenant cache-scope rule.
• Vertex AI generative AI data governance: one vendor's retention and caching terms as a model for what to record in the ADR.

Exercises

1. Walk your case's constraint register through the topology tree and write the ADR's Rejected section first: one line per lower rung naming the register line that forbids it. If you cannot write a line for rung one, your decision is rung one.
2. List every layer in your system that touches the model (prompt prefix, index, cache, key, logs, memory) and write, for each, where the tenant claim is verified before that layer is reached. Any layer without a verification point is your tenancy model's open edge.
3. Build the break-even sketch for your case in an illustrative blockquote: the API model you would replace at its current sheet price, your real monthly volume, your utilisation pattern, a dated GPU quote and the fraction of an engineer. State the two conditions that would reopen the decision.

Check your understanding