Lesson 05. Context Is an Architecture, Not a Prompt

The correspondence assistant's system prompt grows the way every system prompt grows: a paragraph for tone, a paragraph for each letter type, the full style guide pasted in after legal reviewed it, forty exemplar letters, the glossary, the escalation rules, the list of things it must never say. By the end of the pilot it is forty thousand tokens long, assembled fresh on every turn with the adjuster's name and today's date at the top. Cost triples between the first and third month with no change in volume. Then a quieter symptom: the model starts ignoring rules that live in the middle of the prompt. The "never quote a superseded wording" instruction sits at token 19,000, surrounded by exemplars, and the drafts begin to quote superseded wordings. The team's response is to move the rule to the top, which works, and to add a second copy at the bottom, which also works, and nobody asks why a rule needs to be in two places to be obeyed.

Illustrative composite. The 40,000-token prompt, the tripled cost and the rule at token 19,000 are modeled for teaching. The mechanism, a prefix rebuilt per turn with volatile content at the top and instructions buried in the middle, is the common one.

Decide now: which three lines in that prompt would you move, and where, before touching a single word of the instructions?

This lesson gives you a context specification for your case: layers with token budgets, a cache layout with TTL choice, and the output schemas at each boundary. It is the second Decide artefact in your Dossier, and it is the one that turns the model choice from Lesson 04 into a bill you can predict.

What is actually going on

Two mechanisms explain both symptoms. The first is the cache. Every major vendor now caches a prompt prefix and charges a fraction of the base input price when a later request repeats it. On Anthropic the prefix is matched in a fixed order, tools then system then messages, and a change at any level invalidates that level and everything after it; the price of a cache read is 0.1x the base input price (0.025x on Fable 5.1), a 5-minute cache write costs 1.25x and a 1-hour write costs 2x (prompt caching, observed 2026-09-09). Put a date or a user's name at the top of the prefix and no request ever matches the last one. The team in the opening paragraph paid the full input price on 40,000 tokens per turn for content that changed by a dozen tokens per turn. OpenAI and Google cache differently (OpenAI on prefixes with its own minimums and retention; Gemini with implicit and explicit modes and a storage charge); read the mechanics on their pages before you design for them (OpenAI prompt caching, Gemini context caching).

The second mechanism is attention. Models do not read a long context evenly. The "lost in the middle" study found retrieval-style performance highest when the relevant passage sits at the start or end of the context and lowest when it sits in the middle, a U-shape rather than a plateau (Liu et al., observed 2026-09-09). The RULER benchmark, which tests retrieval, multi-hop tracing, aggregation and QA at increasing lengths, found almost every model falling below its quality threshold well before its claimed context length, with only half of the tested models effective at 32K (RULER, observed 2026-09-09). NoLiMa, which removes literal word overlap between question and passage, found that at 32K tokens 10 of 12 models claiming 128K or more dropped below half of their short-context score, and one strong model fell from 99.3% to 69.7% (NoLiMa, observed 2026-09-09). Anthropic's own documentation calls the degradation "context rot" and says curation matters as much as capacity (context windows, observed 2026-09-09). A rule at token 19,000 of a 40,000-token prompt is exactly where the U-shape is lowest.

So the context is not one string. It is a stack of layers with different lifetimes, and the layer order is the cache layout. Look for which layers change on every request and which never change.

The picture shows the discipline: anything volatile lives below the breakpoint. The adjuster's name and today's date belong in L4, not at the top. The instructions the model must obey on every turn belong in L1, short, near the start or end of the stable block, never buried under forty exemplars.

The rule and its condition

Design the context as layers with token budgets: a stable prefix cached for as long as its reuse interval justifies, a per-session layer cached briefly, and a per-turn layer never cached; put a validated schema at every boundary where a model's output becomes another component's input; and choose long context over retrieval only when the corpus is small, stable and reused across turns, otherwise retrieve.

The cache TTL choice follows from the write multipliers. A 5-minute write costs 1.25x and repays itself after one read; a 1-hour write costs 2x and repays itself after two reads; a 1-hour cache that is read only once costs 2.1x the uncached price, which is worse than not caching (prompt caching, observed 2026-09-09; the arithmetic follows from the multipliers). So: expected reuse within five minutes, use the 5-minute cache; reuse spaced more than five minutes apart but at least twice in the hour, use the 1-hour cache; a prefix used once, do not cache it. The minimum cacheable prefix depends on the model, from 512 tokens on the newest models to 4,096 on Haiku 4.5 and the older Opus generations (prompt caching, observed 2026-09-09), which is one reason the model ADR from Lesson 04 comes before this specification.

Structured output is the other half of the rule, and it is constrained decoding, not a polite request for JSON. Anthropic exposes a JSON-schema output format plus a strict mode on tool definitions, supported on current models, with limits you must design around: no recursive schemas, no numeric range or string length constraints, and a compiled grammar that is cached for 24 hours so a schema change carries a small one-off cost (structured outputs, observed 2026-09-09). OpenAI's strict structured outputs are the equivalent; check the current page for its constraints (OpenAI structured outputs). Validate the output anyway on your side of the boundary. The schema is the contract between the model and the code that consumes it, and the code is where the validation belongs.

Budgets, tools and compaction

A layer without a budget grows until it hits the window, and the window is far larger than the effective context the evidence above supports. Give every layer a token ceiling and an owner, and measure the actual counts in production; the vendor's token-counting endpoint exists for this, and the tokeniser changed between model generations by about 30% on Anthropic's 4.7 and later models (pricing page, observed 2026-09-09), so counts from one model are not counts for another.

Tool definitions are the layer teams forget to budget. Anthropic reports that a typical five-server MCP setup consumes about 55K tokens of tool definitions before any work starts, that selection accuracy degrades once a model faces more than 30 to 50 tools, and that deferring definitions behind a tool-search tool cuts definition tokens by more than 85% (tool search, observed 2026-09-09). Lesson 07 designs the tool catalogue; this specification decides how much of it is loaded eagerly into L0.

Long sessions need a compaction rule, and it belongs in the specification, not in an engineer's head. Anthropic's server-side compaction summarises older turns when input tokens reach a threshold, 150,000 by default with a 50,000 minimum, and bills the summarisation pass as an extra iteration (compaction, observed 2026-09-09); context editing can clear old tool results at a configurable trigger, and clearing invalidates the cached prefix below the cleared block (context editing, observed 2026-09-09). Write down the threshold, what the summary must preserve, and the fact that the pass costs money. Thinking tokens count toward the context and are billed as output; a long agent loop that keeps prior thinking blocks fills the window silently.

Worked case

Harbor Mutual's correspondence assistant has a model (Lesson 04: mid tier primary, large tier on a queued escalation route, small tier for classification). Adjusters work in bursts through the day; a draft session opens a claim file, asks for a first draft, then two or three revisions, all within a few minutes. Every draft cites policy wording and must return a letter object the case-management system renders and files.

Illustrative composite. Traffic: about 3,000 draft calls a day on the primary route, in sessions of three to four calls spaced under two minutes apart. Layer sizes after redesign: L0 tools and schemas 2,500 tokens; L1 identity, policy and hard rules 1,800; L2 correspondence style guide, six exemplars and glossary 6,000; L3 claim-file summary plus retrieved clauses 4,000 per session; L4 the request itself, 500. Stable prefix (L0 to L2): 10,300 tokens.

Decide now: which layers get which TTL, and where does the breakpoint go?

Step one, place the breakpoint. L0 to L2 are identical for every adjuster on every claim. They get one breakpoint at the end of L2. L3 changes per claim but is reused across the three or four calls of a session, so it gets a second breakpoint at the end of L3. L4 is never cached.

Step two, choose TTLs. L0 to L2 are read continuously across the working day, thousands of times an hour, so the 5-minute cache would stay warm on its own; the 1-hour cache costs one double-priced write an hour and removes the risk of a quiet ten minutes evicting the prefix, and the difference is a handful of writes a day. Harbor picks the 1-hour cache for L0 to L2. L3's reuse happens within a session of a few minutes, so the 5-minute cache is right; a 1-hour write on a claim file that is read three times would cost more than it saves.

Step three, price it. Using the verified mid-tier rates and the regional premium from Lesson 04:

Illustrative composite. Uncached, the 10,300-token stable prefix on 3,000 calls costs 30.9M tokens × $2/M = $61.80 a day before the regional premium. Cached: 24 one-hour writes × 10,300 × $4/M = $0.99, plus 3,000 reads × 10,300 × $0.20/M = $6.18, so $7.17 a day, an 88% reduction on the prefix. L3 at 4,000 tokens: one 5-minute write per session ($0.01) and three reads ($0.0024) instead of four uncached passes ($0.032), roughly a 60% reduction per session. Per draft, the prefix now costs $0.0024 instead of $0.0206; the fresh L3 plus L4 input and the 600-token output are unchanged. The regional 1.1x multiplier applies to every line.

Step four, the cross-lesson effect. The routing sketch from Lesson 04 sends the hard share of drafts, about one in eight, to the large tier on the queued route. That route keeps its own cache. Its traffic is thinner, a few hundred calls a day spread across the working hours, so a 1-hour cache on the same 10,300-token prefix at the large tier's write price costs more per write and is read fewer times per write.

Illustrative composite. Large-tier route: 24 writes × 10,300 × $10/M = $2.47 a day plus 360 reads × 10,300 × $0.50/M = $1.85, so the prefix costs $0.012 per escalated draft against $0.0024 on the primary route. Still far below the uncached $0.057 per call, but five times the primary route's prefix cost. Blended cost per draft moves from $0.021 to about $0.019 once both caches are counted, and the cascade still beats routing everything to the large tier with its own warm cache ($0.040 per draft before the cache, about $0.035 after).

The rejected alternative was the pilot's layout: one long prompt with the adjuster's name at the top, never cached because the name broke every match, and with its rules sitting in the middle of the U-shape.

Illustrative composite. At the pilot's 40,000-token prefix and 3,000 calls a day, the uncached prefix alone would have cost 3,000 × 40,000 × $2/M, about $240 a day at the verified mid-tier rate, more than thirty times the cached design's $7.17.

Step five, the schemas. Every boundary gets one: the classifier's output into the router, the retriever's output into L3, and the draft into the case-management system. The draft schema:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "ClaimLetterDraft",
  "type": "object",
  "additionalProperties": false,
  "required": ["letter_type", "salutation", "body_paragraphs", "cited_clauses", "confidence", "needs_human_review", "review_reasons"],
  "properties": {
    "letter_type": { "type": "string", "enum": ["acknowledgement", "request_for_information", "settlement_offer", "partial_decline", "decline", "dispute_response"] },
    "salutation": { "type": "string" },
    "body_paragraphs": { "type": "array", "items": { "type": "string" } },
    "cited_clauses": {
      "type": "array",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["clause_id", "policy_version", "quoted_text"],
        "properties": {
          "clause_id": { "type": "string" },
          "policy_version": { "type": "string" },
          "quoted_text": { "type": "string" }
        }
      }
    },
    "confidence": { "type": "string", "enum": ["high", "medium", "low"] },
    "needs_human_review": { "type": "boolean" },
    "review_reasons": { "type": "array", "items": { "type": "string", "enum": ["dispute", "unresolved_citation", "low_confidence", "amount_over_authority", "tone_flag"] } }
  }
}

Note what the schema does not do: it does not enforce that policy_version is the current one. That is a check the code makes against the grounding-data index (Lesson 06) after validation, because the model cannot know which version is current and the schema cannot express it. The schema guarantees shape; the code guarantees truth.

Faded case

Skip to the open case if this is familiar.

Lumen's support agent serves many tenants, each with its own product documentation, tone and escalation rules. One engineer, one prompt template, one cache. The current template puts the tenant's name, product docs and the customer's ticket into a single system prompt, in that order, and re-sends it every turn of a six-turn conversation.

Illustrative composite. Layer sizes: tools 3,000 tokens (shared); Lumen's identity and policy 1,200 (shared); per-tenant pack 5,000 to 15,000 (docs, tone, rules); per-conversation ticket and history 2,000 growing to 6,000; the customer's turn 200. Around 1.5 million conversations a month across 300 tenants, heavily skewed: the top 20 tenants generate most of the traffic, the long tail sees a conversation every hour or two.

The layers are the same as Harbor's; the decision that differs is the TTL on the per-tenant pack, because reuse is not uniform. For the top tenants, the pack is read continuously; for the tail, a pack may be read twice an hour. A second consideration is isolation: a cached prefix is keyed by its content, and Lumen's tenants must never share one, which the layout handles as long as the tenant pack sits above the breakpoint and the tenant ID is inside it, not below it.

One decision left to you: does Lumen use one TTL policy for every tenant, or two? Write down your answer and the number you would use to decide per tenant.

Open case

An internal platform team at a bank is building a shared assistant gateway for forty internal applications: a code assistant for developers, a policy Q&A tool for operations, and a document-summarisation service for compliance, all on one model contract. Each application brings its own instructions and tools; some run thousands of calls an hour, some run a dozen a day; the compliance service processes 200-page documents that are rarely reused; the bank's policy is that no application may see another's prompt, cache or output. Produce the layer specification, the cache layout with a TTL rule per application class, the budget for tool definitions, and the schema policy at each boundary. The first quiz item grades your answer.

Where this breaks

Caching saves money, not attention. A 100K-token stable prefix at a tenth of the input price is cheap and is still 100K tokens the model must attend over; the RULER and NoLiMa evidence applies to the whole window, cached or not. If the eval shows rules being missed, the fix is a smaller stable block and retrieval into L3, not a warmer cache.

Toggles above the breakpoint invalidate silently. Changing the tool list, the tool_choice, the effort setting or a web-search flag between turns changes the cached level and everything below it (prompt caching, observed 2026-09-09). The specification should list what may vary between turns and what may not, and the gateway (Lesson 08) should enforce it. Watch the cache-read token count in production; a layout that looks right on paper and reads zero cached tokens is wrong somewhere.

Schemas guarantee shape, not truth. A validated policy_version can still be superseded; a well-formed cited_clauses array can still cite a clause that does not say what the letter claims. The RAG triad in Lesson 06 and the judge in Lesson 10 measure that. Do not let a passing schema validation stand in for an eval.

Vendor semantics differ enough to matter. Minimum cacheable sizes, TTL options, storage charges and whether a cache is implicit or explicit all vary between providers and change without notice. A specification that names one vendor's mechanics must carry the date it was checked, and the portability section of the rollout plan (Lesson 15) must say what changes if the model ADR is reversed.

Key Takeaways

• Order layers by lifetime: tools, identity, tenant pack, session, turn; nothing volatile above the breakpoint.
• Choose TTL by measured reuse: 5-minute for continuous reads, 1-hour for spaced reads, none for single use.
• Give every layer a token budget and an owner; measure real counts, re-count on model migration.
• Put a validated schema at every boundary; validate in code; remember schemas guarantee shape, not truth.
• Keep rules short and at the edges of the stable block; the middle of a long context is where they get lost.
• Prefer retrieval when the corpus is large, changing or associative; long context when small, stable and reused.

What You Have Now

A context specification for your case: layers with token budgets, a cache layout with TTL choice, and the output schemas at each boundary. It lives at dossier/03-context-spec.md, with the schemas beside it under dossier/schemas/ — for Harbor Mutual's letter-drafting boundary, that's dossier/schemas/claim-letter-draft.json; the lab's checker looks for both at those paths. It reads the model ADR from Lesson 04 (which sets the cache minimums and the write multipliers), and it is read by the grounding-data design in Lesson 06 (which fills L3), the agent specification in Lesson 07 (which decides what goes into L0), the integration specification in Lesson 08 (which enforces the invariants at the gateway) and the cost model in Lesson 11 (which prices every layer). It serves the Decide stage: the bill and the behaviour of the system are both decided here, before a line of prompt text is written.

Example Files

The specification, to copy into dossier/03-context-spec.md:

# 03 Context specification: <system name>

Model (from adr/0001-model.md): <tier / ID>; minimum cacheable prefix: <n> tokens; tokeniser checked: YYYY-MM-DD

## 1. Layers and budgets

| Layer | Contents | Budget (tokens) | Owner | Changes when | Cache |
|---|---|---|---|---|---|
| L0 Tools and schemas | eager tool definitions, output schemas | | | tool catalogue release | 1h |
| L1 Identity and policy | role, hard rules, output contract | | | policy change (versioned) | 1h |
| L2 Tenant / domain pack | glossary, exemplars, style | | | tenant onboarding, quarterly review | 1h or 5m |
| L3 Session state | retrieved docs, file summary, compaction summary | | | per session | 5m |
| L4 Turn | user message, tool results | | | per request | none |

Total stable prefix (L0-L2): <n> tokens. Largest expected request: <n> tokens (p90) / <n> (max).

## 2. Cache layout
- Breakpoint 1 after: <layer>. Breakpoint 2 after: <layer>. (Max 4 on Anthropic.)
- TTL per breakpoint and the measured reuse interval that justifies it:
- Invariants above the breakpoints (must not change between turns): tool list, tool_choice, effort, search flags, <...>
- Things that must live below the breakpoints: user identity, timestamps, request IDs, tenant-specific volatile data
- Isolation: tenant/application identifier is inside the cached prefix, never inferred from below it
- Production check: cache_read tokens > 0 on the second call of a session; alert if zero for <n> minutes

## 3. Compaction and editing
- Trigger threshold: <n> input tokens; what the summary must preserve; who owns the summary prompt
- Tool-result clearing: keep last <n>; note that clearing invalidates the prefix below it
- Thinking blocks: cleared after <n> turns on models that retain them

## 4. Output schemas at each boundary

| Boundary (from -> to) | Schema file | Strict mode | Validated in code by | Truth checks the schema cannot express |
|---|---|---|---|---|
| classifier -> router | schemas/inbound-class.json | yes | gateway | |
| retriever -> L3 | schemas/retrieved-chunk.json | n/a | ingestion service | ACL, freshness |
| model -> consumer | schemas/<output>.json | yes | consumer service | version currency, amounts vs authority |

## 5. Long context vs retrieval (per corpus)

| Corpus | Size (tokens) | Change rate | Question type | Decision | Evidence |
|---|---|---|---|---|---|

## 6. Budget review
- Re-count all layers on: model migration, tool catalogue release, tenant pack change
- Eval item that watches the middle of the prefix: <golden-set item IDs>
- Last reviewed: YYYY-MM-DD by <owner>

Further Reading

• Prompt caching: prefix order, breakpoints, minimums, TTLs and the write and read multipliers used in this lesson.
• Structured outputs: the JSON-schema output format, strict tools and their constraints.
• Context windows: the vendor's own account of context rot and the 1M window's pricing.
• Lost in the Middle: the U-shaped position effect.
• RULER: how to measure a model's effective context length on your own tasks.
• NoLiMa: why associative questions degrade faster than literal-match ones as context grows.

Exercises

1. Audit a prompt you have. Take any system prompt you or your team runs today. Split it into the five layers, count tokens per layer with the vendor's counting endpoint, and mark every line that changes between requests. Draw where the breakpoint goes and estimate the cache-read share you would get.
2. Choose the TTL with data. For one route in your case, measure or estimate the interval between requests that share a prefix. Apply the break-even rule (one read repays a 5-minute write, two reads repay a 1-hour write) and write the TTL decision with the interval that justifies it.
3. Write the specification. Fill the template for your case, including at least one boundary schema written in full and one "truth check the schema cannot express" per boundary. Then write the golden-set item that would catch a rule being lost in the middle of your stable block.

Check your understanding