Lesson 06. Grounding Data Is a Governed Asset

A customer receives a letter declining part of a water-damage claim. The letter quotes the exclusion clause word for word, with the clause number, in the assistant's careful tone. The clause is from the policy wording that was retired eleven months earlier; the customer's policy was renewed onto the new wording, which narrowed the exclusion. The customer's broker notices, because brokers read wordings, and complains. The investigation finds the cause quickly: the index holds every wording ever published, nothing marks which one is current for a given policy, and the retired clause matched the query slightly better than the new one. The investigation also finds something worse. While tracing the retrieval log, an engineer sees that a query from one brokerage returned a chunk of a claim file belonging to another brokerage. The index had no document-level access rule, only the assumption that the application's front end would never ask for the wrong thing.

Illustrative composite. The eleven-month-old wording and the cross-broker chunk are modeled for teaching. Both failures come from one cause: the corpus was loaded as a pile of text, not managed as data with an owner, a freshness rule and an access rule.

Decide now: before you touch the retriever, which two fields would you add to every chunk in that index, and who in the company gets to set their values?

This lesson gives you a grounding-data design document and an augmentation ADR: ingestion, chunking, index, access rules, freshness, and the retrieval metrics that will prove it. It is the third Decide artefact in your Dossier and the one most often skipped, because the demo works on a folder of PDFs and nobody asks who owns the folder.

What is actually going on

Retrieval-augmented generation is an information-retrieval system with a language model at the end, and most of its failures are retrieval failures: the wrong chunk, the stale chunk, the chunk the caller was not entitled to, or no chunk at all because the query and the passage do not share words. The model then does what it does with whatever it is given, and does it fluently. A hallucinated clause and a correctly quoted superseded clause look identical from the outside, and only the second one can be prevented by data governance.

The pipeline that produces the index is where governance lives, because every property the retriever later needs is either attached at ingestion or is not there. Watch which stages attach which properties.

The diagram shows the two things the opening failure lacked. Metadata is a stage of its own, populated from the source system, not inferred from the text: owner, doc_version, valid_from, valid_to, acl_groups, tenant_id. And the change feed is a loop back into the pipeline, not a nightly job someone remembers to run. Parsing matters more than teams expect; scanned multi-column policy documents and dense tables are where open-source layout parsers and hosted document-intelligence APIs diverge most, and the sensible pattern is to route clean documents to the free parser and escalate scans and tables to a hosted one by a classifier, keeping page and section provenance so a citation can point at a source span rather than a chunk.

Chunking is measurable, not folklore. Chroma's chunking evaluation reported recall differences of several points between strategies on the same corpus, with semantic and cluster-based chunkers around 0.91 to 0.92 recall at 400 tokens against the mid-to-high 80s for plain recursive character splitting (Chroma research, observed 2026-09-09; figures as reported by the authors). Anthropic's contextual-retrieval experiments reported the top-20 retrieval failure rate falling from 5.7% to 2.9% when each chunk was prefixed with a short model-written context before embedding and lexical indexing, and to 1.9% with a reranker added (contextual retrieval, observed 2026-09-09; figures as reported by the authors). Those are the two cheapest quality levers in the pipeline, and both are decided at ingestion.

The rule and its condition

Treat grounding data as a governed asset: every document in the index carries an owner, a freshness rule, an access rule and a chunk strategy chosen by the query type it must answer; retrieve hybrid (lexical plus dense) with a reranker and the access filter applied inside the search; and choose RAG over fine-tuning or long context when the facts change, otherwise use the tree below.

The augmentation choice has more than two branches, so it is a tree. Facts that change belong in retrieval, not in weights: Ovadia and colleagues found retrieval consistently outperformed unsupervised fine-tuning for injecting both known and new knowledge, and that a model needs many paraphrased exposures to absorb a fact at all (Ovadia et al., observed 2026-09-09). Fine-tuning changes behaviour: format, tone, tool-call discipline, jargon. Long context with a cache wins when the corpus is small, stable and reused, per the evidence in Lesson 05. GraphRAG answers the "global" question, themes across a corpus, that chunk retrieval cannot, at an indexing cost its own maintainers warn about (GraphRAG docs, observed 2026-09-09).

The tree encodes fine-tuning's real conditions, which are stricter than most proposals admit. Tune only when the problem is behaviour that prompting and schemas could not fix, when you hold thousands of representative labelled traces with a frozen held-out split, when the target is stable enough that the tune will not be obsolete before it pays back, when the payback computed from volume times the per-task saving is under six months, and when a golden-set gain of at least two points over the prompted baseline is expected with a regression suite guarding capabilities you did not train. Parameter-efficient methods make the training itself cheap: LoRA trains low-rank adapters over frozen weights (LoRA), and QLoRA fits a 65B-parameter model onto a single 48 GB GPU by quantising the base to four bits (QLoRA, observed 2026-09-09). The data and the eval are the cost, not the GPU hours. Note also what is on offer: Anthropic publishes no first-party fine-tuning for current Claude models (absence observed on its platform docs, 2026-09-09), and hosted tuning of other vendors' flagships has been narrowing, so a tuning plan should assume open weights and a serving stack you or a hosting provider run.

Retrieval at query time

The query side has its own shape, and the order of operations is the access control. Watch where the filter runs.

Three details in that sequence are decisions. Hybrid search is the default, not an optimisation: exact identifiers such as clause numbers, part numbers and statute IDs never match well in embedding space, and the lexical index catches them. The reranker sits after fusion and looks at a bounded candidate set; reranking hundreds of candidates per query makes the reranker the latency floor. And the as_of date travels with the query, so a question about an old policy can legitimately reach a superseded wording while the default excludes it.

Embedding choice follows the same discipline as model choice in Lesson 04. The MTEB leaderboard is the standard yardstick and covers retrieval, classification, clustering and reranking tasks in many languages (MTEB, observed 2026-09-09), but its leaders are large models with high dimensions and the spread in cost and latency between candidates is an order of magnitude. Pick by your own retrieval eval, your languages, and the dimension your store can index; on pgvector, HNSW indexes on the standard vector type are limited to 2,000 dimensions and higher-dimensional embeddings need the half-precision type (pgvector index guide, observed 2026-09-09). Version the embedding model on every vector, because an upgrade without a re-embed produces a mixed vector space that ranks nonsense.

The store is chosen by where the data already lives and how the access filter is expressed, not by raw nearest-neighbour speed.

Freshness is a data-engineering job. Treat vectors as derived data: a change-data-capture feed from the source system (Debezium streams row-level changes from the common databases with sub-minute latency, per its project blog, observed 2026-09-09) drives re-chunk and re-embed of the changed documents only, keyed on a content hash; deletions write tombstones so a removed document can never be returned; a periodic full re-index is a consistency check, not the freshness mechanism. Write the propagation target down and measure it.

The metrics that prove all of this come in two groups. Retrieval metrics are code-graded: recall@k, the share of gold passages present in the top k, and MRR, the mean of one over the rank of the first relevant hit. Generation metrics are the RAG triad: context relevance (are the retrieved chunks about the question), groundedness or faithfulness (is every claim in the answer supported by the chunks), and answer relevance (does the answer address the question), popularised by the TruLens project (RAG triad, observed 2026-09-09) and implemented with variations in RAGAS (RAGAS docs) and, with statistical confidence intervals from a small human-labelled set, in ARES (ARES, observed 2026-09-09). Grade generation with a different model than the generator; Lesson 10 explains why.

Worked case

Harbor Mutual's assistant grounds every letter in policy wording and the claim file. The sources: about 400 policy wordings across product lines, each with several versions in force for different renewal dates, owned by underwriting; claim files (correspondence, adjuster notes, photos) owned by claims operations and visible only to the handling team and the customer's own broker; correspondence templates owned by legal; regulatory guidance owned by compliance and updated monthly. The archive is on-prem, mostly PDF, a third of it scanned.

Illustrative composite. Sizes: policy wordings about 60,000 pages, around 2 million chunks at clause level; claim files 1.2 million policies' worth, growing daily; templates a few hundred pages; guidance a few thousand pages. Budgets from Lesson 02: recall@10 ≥ 0.90 on a 300-question golden set, faithfulness ≥ 0.95, a superseded-wording citation rate of zero on the golden set, and a freshness target of 15 minutes for a new wording version.

Decide now: which source gets which chunk strategy, and which source is the one you would not put in the shared index at all?

Step one, ownership and access. Each source gets an owner who approves ingestion and signs the freshness rule. Wordings carry product, doc_version, valid_from, valid_to from the policy-admin system. Claim files carry claim_id, broker_id, handling_team as acl_groups. Templates and guidance are readable by every internal caller.

Step two, chunk strategy by query type. Wordings are looked up by clause: chunk at clause boundaries using the document structure, prepend a short model-written context naming the product and section, and index both lexically (clause numbers are exact identifiers) and densely. Guidance is read for synthesis: 400-token recursive chunks with overlap and the same contextual prefix. Templates are retrieved whole by letter type; they are small enough to be a single chunk each.

Step three, the claim file. This is the source Harbor keeps out of the shared corpus index. A single claim file fits in a few tens of thousands of tokens, is reused across the three or four calls of a drafting session, and is exactly the "small, stable, reused" case from Lesson 05. It goes into the session layer L3 under the 5-minute cache, pulled directly from the claims system by claim_id under the caller's own entitlement. Nothing from another brokerage can appear, because nothing from another claim is ever retrieved. The cross-broker leak becomes structurally impossible on this route, and the corpus index carries no claim data at all.

Step four, retrieval and store. Harbor already runs Postgres for the policy-admin data, and the ACL is a join against the same tables, so pgvector with a lexical index in the same database is the store; the vector count is within its comfortable range and the filter is a where clause on product, validity window and caller groups, applied inside both searches. Reciprocal rank fusion, then a cross-encoder reranker over 50 fused candidates down to 8. The as_of date is the policy's renewal date, so a dispute about a 2024 renewal legitimately reaches the 2024 wording.

Step five, freshness. The policy-admin system emits a version event when a wording is published or retired; a change-data-capture consumer re-parses, re-chunks, re-embeds only the affected wording and flips valid_to on the superseded version within the target. Guidance is re-ingested on a monthly schedule by compliance. A nightly count comparison between the policy-admin system and the index raises an alert on drift.

Illustrative composite. Measured in the pilot: p95 propagation for a new wording version 4 minutes against the 15-minute target; recall@10 of 0.93 with hybrid plus reranker against 0.84 dense-only, which would have failed the budget; faithfulness 0.96; superseded-wording citations on the golden set: zero after the validity filter, eleven before it. The leak test, a caller with no groups running the whole golden set, returned zero chunks.

The rejected alternative was the pilot's design: everything, claim files included, in one dense-only index with no metadata, refreshed by a weekly full reload. It failed three budgets at once and, per the opening, it leaked. The augmentation ADR that records this:

# ADR-0002: Augmentation strategy for the Harbor Mutual correspondence assistant

Status: accepted          Date: 2026-09-10          Owner: lead architect

## Context
- Facts the assistant must get right: current policy wording per policy and
  renewal date; the claim file; regulatory guidance. All change; wordings are versioned.
- Behaviour the assistant must show: letter format and tone (handled by
  schema and exemplars in 03-context-spec.md).
- Budgets: recall@10 ≥ 0.90; faithfulness ≥ 0.95; superseded citations = 0;
  freshness ≤ 15 min for wordings.

## Decision
- Wordings and guidance: hybrid retrieval (clause-level chunks with context
  prefix, BM25 + dense, RRF, cross-encoder rerank to 8), validity and ACL
  filters inside the search, `as_of` = policy renewal date.
- Claim file: long context in the session layer under the 5-minute cache,
  fetched by claim_id under the caller's entitlement; never in the shared index.
- No fine-tuning: the problems are facts (retrieval) and format (schema).
- No GraphRAG: no global "themes across the corpus" question exists in the charter.

## Alternatives considered

| Option | Facts current? | Leak risk | Cost | Verdict |
|---|---|---|---|---|
| Single dense index, no metadata (pilot) | no | high | low | rejected: failed recall, leaked |
| Fine-tune on wordings | no (stale on publish) | n/a | data + eval | rejected: facts branch |
| Whole wording library in long context | no (effective context) | low | high per call | rejected: corpus too large, changes |
| Hybrid + rerank + validity/ACL filters, claim file in L3 | yes | structural none on claim data | moderate | accepted |

## Consequences
- Gain: current facts by construction; leak test passes structurally on the claim route.
- Pay: a CDC consumer to operate; a reranker on the latency path (~budgeted);
  a golden set that underwriting must keep current when wordings change.

## Review condition
Revisit when recall@10 falls below 0.90 or faithfulness below 0.95 on two
consecutive weekly runs; when p95 propagation exceeds 15 minutes for a week;
when a charter change introduces a corpus-wide synthesis question; or when
the embedding model is upgraded (full re-embed required first).
Next review: 2026-12-01.

Faded case

Skip to the open case if this is familiar.

Lumen's support agent answers from each tenant's product documentation: help-centre articles, release notes and internal runbooks the tenant chooses to share. Tenants update documentation weekly; some publish release notes daily. One engineer, one index, three hundred tenants. Questions are mostly lexical ("how do I rotate an API key") but a growing share are associative ("why did my invoice change"). Lumen has no per-tenant ACL requirement inside a tenant today, only strict isolation between tenants.

Illustrative composite. Corpus: 300 tenants, from 50 to 4,000 pages each, about 1.8 million chunks total; retrieval budget recall@10 ≥ 0.85 on a 400-question set drawn from real tickets; freshness target one hour from a tenant's publish event.

The ownership rule is straightforward: each tenant owns its own corpus and its publish event is the freshness trigger. Chunking is 400-token recursive with a contextual prefix, because the documentation has no reliable clause structure. Hybrid retrieval with a reranker, because the associative share is growing. Isolation by a per-tenant namespace or collection with tenant_id from the signed token, never from the prompt.

One decision left to you: Lumen's engineer proposes one shared index with a tenant_id metadata filter, because it is simpler to operate than three hundred collections. Write down whether you accept it, and the condition under which you would change the answer.

Open case

A scale-up adding "ask your data" to its business-intelligence product wants users to type questions and receive answers grounded in the customer's own warehouse tables, dashboard definitions and the analysts' saved queries. The corpus per customer is mostly structured (schemas, column descriptions, metric definitions) plus a few hundred pages of prose documentation, and it changes every time an analyst edits a metric. Different users in the same customer see different tables under row-level and column-level security enforced by the warehouse. Produce the grounding-data design: what is retrieved versus what is queried live, the chunk strategy per source, the access rule and where it is enforced, the freshness rule, and the two metrics you would put in the golden set. The first quiz item grades your answer.

Where this breaks

Governance has a cost and a bottleneck. An owner per source who approves ingestion is also an owner who can be on holiday. The design needs a deputy per source and a rule for what happens to the freshness target when approval stalls, or the index quietly returns to "pile of text" with better paperwork.

Metadata mirrored from a source system is only as good as the source system. If the policy-admin system records valid_to a week late, the index is a week stale with perfect propagation. The freshness measurement must run end to end, from the business event to the retrievable chunk, not from the change feed to the index.

Contextual prefixes and rerankers cost money and latency at scale. Writing a short context for every chunk is one cheap-model call per chunk, affordable on a stable corpus and painful on one that churns; a reranker on every query is a fixed latency the SLO in Lesson 14 has to absorb. Both are worth it when the recall eval says so, and the eval is what tells you, not the pattern's reputation.

Filters inside the search assume the store can filter well. Some approximate-nearest-neighbour indexes degrade badly under selective filters, returning fewer than k results or scanning far more than they should. Test filtered recall on your store with your real selectivity before the design is final; the vector benchmark suites include filtered cases for this reason.

Key Takeaways

• Give every source an owner, a freshness rule, an access rule and a chunk strategy chosen by query type.
• Attach doc_version, valid_from, valid_to, tenant_id and acl_groups at ingestion, from the source system.
• Retrieve hybrid with a reranker; apply the access and validity filters inside the search, never after top-k.
• Send changing facts to retrieval, small stable reused corpora to long context, behaviour problems to prompting first.
• Fine-tune only on behaviour, with thousands of labelled traces, a regression suite and a payback under six months.
• Prove it with recall@k and MRR on retrieval, the RAG triad on generation, and a leak test in CI.

What You Have Now

A grounding-data design document and an augmentation ADR: ingestion, chunking, index, access rules, freshness, and the retrieval metrics that will prove it. They live at dossier/04-grounding-data.md and dossier/adr/0002-augmentation.md. They read the budgets from Lesson 02, the constraint register from Lesson 03 and the context specification from Lesson 05 (which decides what the retriever fills into the session layer), and they are read by the agent specification in Lesson 07 (retrieval becomes a tool with a risk rating), the topology ADR in Lesson 09 (where the index lives and how tenants are isolated), the eval plan in Lesson 10 (the golden set and metrics defined here) and the threat model in Lesson 12 (the access rule is a control). They serve the Decide stage.

Example Files

The design document, to copy into dossier/04-grounding-data.md:

# 04 Grounding-data design: <system name>

## 1. Sources

| Source | Owner (approver) | Deputy | System of record | Volume | Format | Sensitivity |
|---|---|---|---|---|---|---|

## 2. Per-source rules

| Source | Query type it answers | Chunk strategy | Context prefix? | Freshness rule | Propagation target | Access rule (fields) |
|---|---|---|---|---|---|---|

## 3. Ingestion pipeline
- Parse: <parser per document class; escalation rule for scans and tables; provenance kept: page, span>
- Chunk: <strategy per source, size, overlap, structure boundaries>
- Contextualise: <model, prompt, cached document prefix, batch>
- Embed: <model and version recorded on every vector; dimensions; store type>
- Index: <vector index type; lexical index; metadata fields with types>
- Change feed: <CDC source, hash key, tombstones, periodic consistency check>

## 4. Metadata schema (every chunk)
tenant_id, source, doc_id, doc_version, valid_from, valid_to, acl_groups[], owner,
page, span_start, span_end, embedding_model_version, ingested_at, content_hash

## 5. Retrieval
- Entitlement resolver: <where tenant_id and acl_groups come from; signed token>
- Query: <rewrite? hybrid k per index; fusion; reranker candidates -> top n>
- Filters applied inside the search: tenant, groups, validity window with as_of
- Provenance returned: page and span for citations
- Latency budget: <ms for retrieval + rerank, from 01-nfr-budgets.md>

## 6. What is NOT in the index (and where it goes instead)
- <e.g. per-record files fetched live under the caller's entitlement into L3>

## 7. Metrics and tests
- Golden set: <n questions with gold passages; owner; refresh rule>
- Retrieval: recall@k ≥ <x>, MRR ≥ <y>; run on every index change
- Generation (RAG triad): context relevance, faithfulness ≥ <x>, answer relevance; judge model ≠ generator
- Leak test: zero-group caller and cross-tenant caller return zero chunks; in CI
- Freshness: end-to-end propagation p95 ≤ <target>; drift alert threshold
- Filtered recall on the store at production selectivity: measured on YYYY-MM-DD

## 8. Review
- Re-embed required on: embedding model change
- Revisit chunking when: recall eval shows gaps by source
- Last reviewed: YYYY-MM-DD by <owner>

The augmentation ADR skeleton, to copy into dossier/adr/0002-augmentation.md:

# ADR-0002: Augmentation strategy for <system name>

Status: proposed | accepted | superseded by ADR-NNNN
Date: YYYY-MM-DD          Owner: <architect>          Reviewers: <data owner, security, product>

## Context
- Facts the system must get right, and how often each changes:
- Behaviours the system must show (format, tone, tool discipline):
- Budgets this decision must meet (recall@k, faithfulness, freshness, cost per query):
- Corpus size in tokens and chunks; question types (lexical / associative / global):

## Decision
- Per corpus: retrieval | long context + cache | GraphRAG | fine-tune, with the tree branch that led there
- What lives in the session layer instead of the index, and why
- Fine-tuning: not now | conditions met (traces: <n>, held-out split frozen: YYYY-MM-DD, payback: <months>, expected gain: <points>, regression suite: <name>)

## Alternatives considered

| Option | Facts current? | Leak risk | Cost | Verdict |
|---|---|---|---|---|

## Consequences
- Gain:
- Pay (operations, latency, eval upkeep):
- Depend on (source-system metadata quality, embedding model, reranker vendor):

## Review condition
Revisit when: <metric threshold and window>; <freshness breach>; <charter change introducing a new question type>; <embedding or model change>
Next review: YYYY-MM-DD

Further Reading

• Contextual retrieval: the chunk-context technique and the reported failure-rate reductions with and without reranking.
• Evaluating chunking strategies: how chunkers were compared on recall and precision, and why size and boundary choice both matter.
• Fine-tuning or retrieval?: the study behind "facts go to retrieval".
• The RAG triad: context relevance, groundedness and answer relevance defined.
• ARES: RAG evaluation with confidence intervals from a small human-labelled set.
• GraphRAG documentation: when community summaries answer global questions, and what indexing costs.

Exercises

1. Inventory your sources. For your case, fill section 1 and section 2 of the design document for every source. If you cannot name an owner for a source, mark it and decide whether it enters the index at all.
2. Write the leak test. Describe, concretely for your store, the two queries (zero-group caller; cross-tenant caller) and the assertion that would fail a CI run. Then describe the filtered-recall test at your real selectivity.
3. Take the augmentation decision. Walk each corpus through the tree and record the branch in the ADR. For any corpus where fine-tuning is proposed, write the payback arithmetic (volume × per-task saving against labelling, training and eval cost) and the regression suite that would guard it.

Check your understanding