Lesson 01. The Architect Decides What Not to Build

A regional insurer runs a pilot: an assistant that drafts claim letters for its handlers. The vendor team builds it well. Retrieval over the policy archive works, the drafts read cleanly, handlers who try it say they like it. Month three, the steering committee asks for "more coverage" and the team adds subrogation letters. Month five, someone asks whether it could book adjuster visits, so a calendar integration goes in. Month seven, the security team notices that the calendar integration acts as a service account and opens a review. Month nine, the CFO asks what the pilot has saved. Nobody can answer, because nobody measured how long a letter took before the pilot began, and nobody wrote down what result would have ended it. The sponsor cancels. The engineers are proud of the system; the buyer remembers only the invoice. Decide now: what single document, written in week one, would have changed the ending?

Illustrative composite. The nine-month timeline and the sequence of scope additions are a composite of engagements this course draws on, not a single company's record. The shape is common; the specifics are invented.

This lesson's deliverable is the first file of your Architecture Dossier. An engagement charter for one real or course case: problem, users, non-goals, pilot exit evidence, milestones. It serves the Frame stage of the Frame → Decide → Prove → Run loop, the stage where a wish becomes a set of constraints.

What is actually going on

The pilot above did not fail on engineering. It failed because nobody held the job this course teaches. A solutions architect is the person who decides what the system is, what it is not, and what evidence will let it move from one stage of its life to the next. The engineers decide how. The sponsor decides whether to pay. The architect decides what, which mostly means deciding what not to build, and writing that decision where every later argument can find it.

Look at where the pilot went wrong and you find the same missing decision each time. Subrogation letters were added because nobody had written "correspondence for first-party property claims only" as the scope. The calendar integration went in because nobody had written "no actions that change external systems" as a non-goal. The CFO's question had no answer because nobody had written "handler minutes per letter, measured for two weeks before the pilot, compared against the same measure in week eight" as the evidence that would end the pilot. Each addition was reasonable on its own. Together they turned a bounded pilot into an unbounded project, and unbounded projects have no ending except cancellation.

An engagement has a lifecycle with named states and, more importantly, named exits. Trace the diagram below and notice that almost every transition is a decision to stop, park or decline, and each one names the trigger that fires it.

The diagram shows three things the dead pilot never had: a charter state before any building, a written trigger on every forward arrow, and an exit from every state that is not a failure but a decision. "Parked with a revisit date" is a legitimate outcome. Cancellation after nine months is not; it is the absence of an outcome.

The loop and the Dossier

The engagement lifecycle is the client's view. Your own working rhythm is smaller and repeats inside every state. You Frame a wish into constraints. You Decide, recording each choice with the condition that would reverse it. You Prove the decisions with evidence before anything reaches production. You Run what you shipped and let the review feed the next Frame. Read the diagram left to right and notice which files it produces at each stage.

The loop shows why the course's artifact is a folder and not a slide deck. Every stage ends in a file that a client can read, a reviewer can check and an engineering team can build from. Together those files are the Architecture Dossier, and you build it one file per lesson:

dossier/
  00-charter.md            this lesson: problem, users, non-goals, exit evidence, milestones
  01-nfr-budgets.md        NFR budget sheet and use-case qualification scorecard
  02-constraints.md        client-segment profile and constraint register
  adr/0001-model.md        model selection with its review condition
  03-context-spec.md       context layers, prompt spec, cache layout
  adr/0002-augmentation.md RAG versus fine-tune versus long context
  04-grounding-data.md     ingestion, chunking, index, access rules, freshness
  05-agent-spec.md         pattern choice, loop exits, tool catalogue with risk ratings
  06-integration-spec.md   tool inventory, identities, consent flows, gateway placement
  adr/0003-topology.md     deployment topology and tenancy model
  diagrams/*.mmd           context, container and sequence diagrams
  07-eval-plan.md          golden set, rubric, judge validation, CI gate
  08-cost-model.md         cost per task, twelve-month TCO, self-host break-even
  09-threat-model.md       trust-boundary DFD, control matrix, red-team plan
  10-compliance-annex.md   obligation to design element to evidence
  11-runbook.md            SLOs, fallback chain, trace schema, incident playbook
  12-rollout-plan.md       stage gates, HITL policy per stage, adoption, exit options
  README.md                one-page executive summary for the buyer

Every file after 00-charter.md reads the charter. The budgets in file 01 are the numeric form of the charter's exit evidence. The constraint register in file 02 starts from the charter's users and non-goals. If the charter is wrong, every later file inherits the error, which is why this lesson is the front door of the course and not a warm-up.

Why pilots die

You will be asked, in your first week on any engagement, why so many AI pilots fail. Answer with sourced figures and their limits, because the sponsor has already read the headlines.

The most quoted figure comes from the MIT NANDA report "The GenAI Divide: State of AI in Business 2025", which found that about 95% of enterprise generative-AI pilots showed no measurable profit-and-loss impact, on a base of 52 executive interviews, 153 surveyed leaders and 300 public deployments, against an estimated $30 to 40 billion of enterprise investment (report PDF, observed 2026-09-09). Read the critiques before you repeat the number: the metric counts documented P&L impact inside roughly six months, and most surveyed pilots never measured a baseline at all, so "no measurable impact" often means "nobody measured" (one critique, observed 2026-09-09). The criticism attacks the precision, not the direction, and the direction is the lesson: a pilot with no baseline cannot show impact even when it has some.

The same report is widely quoted as finding that bought or partner-built solutions reached production about twice as often as internal builds. That split is unverified as of 2026-09-09: it circulates in secondary coverage and could not be checked against the report itself, so quote it as a claim under review, not as a fact, until you have read the page it came from.

Gartner's forecast is the second headline. Its press release of 25 June 2025 predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing cost, unclear business value and inadequate risk controls, and estimates that only about 130 of the thousands of vendors selling "agents" offer real agentic capability (Gartner press release, observed 2026-09-09). The same release forecasts that 15% of day-to-day work decisions will be made autonomously by 2028 and that a third of enterprise applications will embed agents by then, up from under one percent in 2024, so the forecast is about badly framed projects, not about the technology's ceiling (same release).

McKinsey's State of AI 2025 survey of roughly 2,000 respondents across 105 countries, fielded in June and July 2025, reports that 88% of organisations use AI in at least one function, 62% are experimenting with agents and 23% are scaling agents somewhere, yet no single business function has more than 10% of companies scaling agents, only 39% report any effect on earnings, and about 6% qualify as high performers with more than 5% of earnings attributable to AI (McKinsey State of AI, observed 2026-09-09).

Put the three together and the pattern is not "AI does not work." It is: most organisations start, few measure, fewer scale, and the ones that scale are the ones that could show a number. The charter is where the number gets named. Adoption is the other half: coverage of the MIT report argues that pilots stall because organisations avoid the friction of changing how people work (Forbes commentary, observed 2026-09-09), which is why the charter names the users and the decision they own today, not just the system.

The rule and its condition

Write the charter before any design, and refuse to start building until all five of its fields hold a specific answer: the problem, the users, the non-goals, the pilot exit evidence and the milestones. Use a charter when the engagement will consume more than a few days of anyone's time or will touch a customer, a regulator or a P&L line; otherwise, for a one-day spike that answers a technical question, write the question and the answer and skip the charter. If a field cannot be filled, the engagement is not ready: the fix is more discovery, a baseline measurement, or a decline, never a kickoff.

The screen below runs before the charter. It answers whether the wish is an LLM problem at all, and most of its exits say "not yet."

Each exit on the left is a legitimate answer to the client, and the second one matters most for regulated work: "a human decides, the model may only draft" is not a rejection of the wish, it is the shape of most systems that reach production in insurance, health and public service. Only the bottom node earns a charter.

Worked case

Harbor Mutual is a regional property-and-casualty insurer with operations in the EU and the UK, an on-premises document archive of policy wordings and claim files, and a cloud tenancy its analytics team already uses. The chief operating officer wants "an assistant that drafts claim correspondence grounded in policy wording and can book adjuster visits." She has a nervous legal team, a procurement process that takes a quarter, data-residency and audit obligations, and a CFO who wants a number before the next board meeting. She also has a dead pilot behind her: the one in the opening of this lesson was hers.

Illustrative composite. Harbor Mutual is fictional. The figures in this case are modeled for teaching: about 1.2 million policies in force, 380 claims handlers, roughly 4,100 outgoing letters a week, and a first pilot that ran nine months and cost around €410,000 before cancellation. Treat them as a worked example, not a benchmark.

You have three ways to charter this. Option A takes the wish whole: drafting plus booking, for handlers and, once trusted, for customers directly. Option B charters drafting only, for handlers, with booking and any customer-facing output named as non-goals for this engagement. Option C declines to charter until a baseline exists and comes back in a month. Decide now, and write down which field of the charter drove your choice.

Here is the reasoning, field by field. The problem: handlers spend time assembling letters from policy wording, and the time is uneven, so the problem statement must name which letters, not "correspondence." The users: claims handlers, who own the decision of what a letter says today, so the assistant drafts and the handler signs. The non-goals: booking adjuster visits changes an external calendar, which is an irreversible action against a system the pilot does not own, and customer-facing output puts generated text in front of a regulated audience without a handler in the loop; both belong in a later engagement with their own charter. The exit evidence: minutes per letter for the complex letter types, measured by the handlers' own case system for two weeks before the pilot and again in the pilot's eighth week, plus a count of drafts a handler rejected for citing the wrong wording. The milestones: baseline measurement, thirty-example test, pilot with twelve handlers, exit decision.

Option B is the charter. Option A repeats the dead pilot with a signature on it: the booking feature alone would pull the security team into the engagement before drafting has proved anything, and the customer-facing path would pull legal in before a single letter has been measured. Option C wastes the sponsor's window; the baseline is a milestone inside the charter, not a precondition for writing it, because the charter can name the measurement method now and fill the number in two weeks. The cost of the rejected option is concrete: under Option A the first quarter goes to a security review and a legal review of features that produce no evidence about the problem, which is the same quarter the sponsor has left before the board asks its question again.

The filled charter for Harbor Mutual, as it would sit in dossier/00-charter.md:

# 00 · Engagement charter · Harbor Mutual · Claims correspondence assistant

| Field | Value |
|---|---|
| Sponsor | Chief Operating Officer (owns the claims cycle-time target) |
| Architect | You |
| Date / version | 2026-09-14 · v0.2 (v0.1 replaced after the discovery workshop) |
| Segment | Regulated enterprise (insurance, EU and UK) |

## 1. The problem
Claims handlers assemble letters for complex first-party property claims by hand,
citing policy wording from an archive with several versions of some wordings.
Complex letters take far longer than templated ones, and the time is where the
backlog comes from. The problem is the time and the citation errors on complex
letters, not correspondence in general.

## 2. Users and the decision they own today
Claims handlers (internal). They decide what a letter says and sign it. That
decision stays with them for the whole engagement.

## 3. Baseline
Handler minutes per complex letter and the citation-error rate on a sample of
letters, measured from the case system over the two weeks before the proof of
concept starts. Method: case-system timestamps plus a fifty-letter audit by the
quality team. Number: to be filled by 2026-09-28.

## 4. Non-goals for this engagement
- Booking adjuster visits (changes an external calendar; separate charter later)
- Any customer-facing output without a handler's sign-off
- Subrogation, liability and third-party correspondence
- Replacing the handlers' case system or search

## 5. Pilot exit evidence
The pilot ends, either way, when both are measured in week eight:
- Minutes per complex letter versus baseline, on the same letter types
- Drafts rejected for citing the wrong wording, per hundred drafts, versus the
  baseline citation-error rate
The numbers that count as success are set in 01-nfr-budgets.md; the charter
only fixes what is measured and when.

## 6. Milestones
1. Baseline measured (two weeks)
2. Thirty-example test on real letters (one week)
3. Pilot with twelve handlers in one region (eight weeks)
4. Exit decision meeting with sponsor, head of claims, CISO (one meeting)

## 7. Kill criteria and revisit conditions
- Thirty-example test: fewer than twenty-four drafts accepted by a senior
  handler with minor edits → park, revisit when the wording archive is
  versioned
- Pilot: any generated citation reaches a customer without sign-off → stop

## 8. Open questions for discovery
- Which archive versions are authoritative, and who owns that decision?
- Does the EU/UK residency obligation cover model inference or only storage?
- What did the previous pilot's users say about why they stopped using it?

Notice what the charter refuses to contain: no model name, no vendor, no architecture. Those are Decide-stage choices and they wait for the budgets and constraints in the next two lessons.

Faded case

Skip to the open case if this is familiar.

Lumen is a twelve-person startup selling an AI support agent to software companies. The system is multi-tenant, hosted in the United States, and growing fast; one engineer owns everything from the prompt to the database. The founder's wish is "autonomous ticket resolution, so we can close the enterprise deal we are negotiating." She wants a charter by Friday.

Illustrative composite. Lumen is fictional. Assume about 20% month-on-month growth in ticket volume and a single enterprise prospect worth more than the rest of the customer base combined.

The problem, users and milestones fall out of the wish: the problem is unresolved tier-one tickets, the users are the customers' support teams, who decide whether a resolution is right today, and the milestones are a thirty-ticket test, a two-week shadow run on one tenant and an exit meeting. Two fields are yours to write. Which non-goal does the charter need, given what a support agent can do to an account, and what is the pilot exit evidence, given that the founder's real goal is the deal?

Open case

A national agency runs a benefits helpdesk. Callers ask whether they qualify for a payment, why a payment changed, and what to send. The agency's director wants "a chatbot that answers any benefits question" to cut call volume, and has promised the minister a launch date in the spring. Callers are often in financial distress. The agency has no labelled data, its caseworkers have no spare hours, every software component must pass a procurement authorisation before it touches citizen data, and the agency is subject to a transparency obligation that requires citizens to be told when they are talking to a machine. Write the charter's five fields. The quiz's first item grades your non-goals and exit evidence.

Where this breaks

The charter assumes a sponsor who will name a kill criterion. Some will not, because naming one admits the pilot might end. When a sponsor refuses, write the criterion yourself in the charter's open questions and make its acceptance the first milestone; if it is still unaccepted at the thirty-example test, the engagement has told you what it is.

The charter also assumes the problem is stable for the length of the pilot. A research spike, where the client does not yet know what they want to learn, is not an engagement and does not get a charter; it gets a question, a time box of days and a written answer. Chartering exploration produces a document that is wrong by the second week and then ignored.

Finally, a charter rots. Every later file in the Dossier reads it, so a scope change that is not written back into the charter makes the Dossier lie. The fix is procedural: the charter carries a version, the exit-decision meeting reads it aloud, and a change to users, non-goals or exit evidence produces a new version, never an edit in place. The course's last lesson closes this loop when the pilot's exit evidence is compared against what the charter promised.

Key Takeaways

• Decide what the system is not before anyone decides how to build it.
• Write the charter's five fields; an unfillable field means discovery, not kickoff.
• Name the exit evidence as a measurement, a method and a date.
• Run the LLM screen first; most exits say "not yet," and that is an answer.
• Quote pilot-failure figures with their sources and their critiques.
• Treat every scope change as a new charter version, never an edit.

What You Have Now

An engagement charter for one real or course case: problem, users, non-goals, pilot exit evidence, milestones. It lives at dossier/00-charter.md, the first file of your Architecture Dossier, and every later file reads it: the budgets in the next lesson turn its exit evidence into numbers, the constraint register turns its users and non-goals into design limits, and the rollout plan at the end of the course compares the pilot's result against it. It serves the Frame stage of the Frame → Decide → Prove → Run loop. If you wrote it for the Harbor Mutual case, keep it; the project at the end of this module rebuilds it from a discovery transcript and you will want to compare the two.

Example Files

The blank charter template. Copy it to dossier/00-charter.md and fill every field with a specific answer or the words "cannot be filled yet: <what is missing>".

# 00 · Engagement charter · <client> · <system in five words>

| Field | Value |
|---|---|
| Sponsor | <name, title> (owns the P&L line or target the pilot moves) |
| Architect | <you> |
| Date / version | YYYY-MM-DD · v0.1 |
| Segment | <early startup / scale-up SaaS / regulated enterprise / public sector / agency / internal platform> |

## 1. The problem
<One paragraph. The task, who does it today, what it costs in time, money or
errors, and why now. No solution words: no "assistant", "agent", "model".>

## 2. Users and the decision they own today
<Who uses the system, and which decision stays with them. If the system
decides, say so and say who is accountable.>

## 3. Baseline
<The measure, the method, the date it was or will be taken, and the number
once known. "Cannot be filled yet" is allowed; a milestone must then fill it.>

## 4. Non-goals for this engagement
- <Adjacent capability the sponsor mentioned, and why it waits>
- <Irreversible actions the system will not take>
- <Audiences it will not face>

## 5. Pilot exit evidence
<The measurements taken at the end of the pilot, either way, with the method
and the week they are taken. The thresholds live in 01-nfr-budgets.md.>

## 6. Milestones
1. <Baseline measured>
2. <Thirty-example test on real inputs>
3. <Pilot with named users, named duration>
4. <Exit decision meeting, named attendees>

## 7. Kill criteria and revisit conditions
- <Result at a milestone that parks the engagement, and what would revive it>
- <Event during the pilot that stops it outright>

## 8. Open questions for discovery
- <Every fact the charter needed and nobody in the room had>

Further Reading

• MIT NANDA, The GenAI Divide: State of AI in Business 2025: the primary source behind the "95% of pilots" headline; read the method section before quoting it.
• Gartner, June 2025 press release on agentic AI project cancellations: the cancellation forecast and the "agent washing" claim in the analyst's own words.
• McKinsey, The State of AI 2025: adoption, agent experimentation and the small group reporting earnings impact.
• AWS Prescriptive Guidance, architecting a generative-AI proof of concept: a vendor's own version of hypothesis, success criteria and evidence before a build.
• Michael Nygard's original architecture decision record format: the decision-record shape the rest of this course extends for model, retrieval and topology choices.

Exercises

1. Take the opening story and rewrite it as the charter that would have prevented it. Fill all eight sections. Mark every field you had to invent; those are the discovery questions the original team never asked.
2. Run the LLM screen on three wishes from your own organisation or a client. For each, name the exit that fires and write the one-sentence answer you would give the person who asked.
3. Write the charter for a real engagement you are on or expect. If any field reads "cannot be filled yet," write the milestone that fills it and the date. Keep this file; it is dossier/00-charter.md for the rest of the course.

Check your understanding