Lesson 12. Threat Model the Agent, Not the Model

A mid-size SaaS company gave its support agent three things over one quarter: read access to tickets and account records, a tool that sends email, and the job of answering tickets from the public web form without a human in the loop. Each addition passed review on its own. Then a ticket arrived whose second paragraph was addressed to "the assistant" and asked it, politely, to collect the account holder's billing details and forward them to an escalation address for verification. The agent looked up the account, composed a helpful summary, and sent it. Twice, because the sender opened a second ticket to say thank you and ask for the updated card. Nobody noticed for three weeks, until an unrelated audit read the outbound mail log. The model had not been jailbroken; it had done what the text in front of it asked, because nothing in its context marked that text as less trustworthy than the system prompt. Decide now: which of the three capabilities do you take away, and why is "tell the model to ignore instructions inside tickets" not on your list?

Illustrative composite. The exfiltration touched 40 accounts before the audit found it; the breach notification, the forensic review and the lost renewals together cost more than the first year of the agent's model spend. The company, its numbers and the incident are a teaching device.

Take away the external channel, or put a human between the agent and it, and treat every ticket as untrusted input to be read by something that cannot act. This lesson's deliverable is 09-threat-model.md in your Dossier. A threat model for your case: a trust-boundary data-flow diagram, a control matrix mapped to OWASP and ATLAS IDs, and a red-team test plan. It is the third document of the Prove stage of Frame / Decide / Prove / Run: the evidence that the design, not the model, keeps the client's data where it belongs.

What is actually going on

A language model reads one channel. The system prompt, the user's message, the retrieved document, the tool result and the attacker's paragraph arrive as the same kind of token, and the model has no reliable way to tell which of them carries authority. Prompt injection is that property, not a bug in one model, which is why OWASP has ranked it first in both editions of its LLM risk list (OWASP Top 10 for LLM Applications 2025, observed 2026-09-09). Direct injection is the user typing the attack. Indirect injection is the attack arriving inside anything the model reads that someone else can write: a ticket, an email, a web page, a retrieved policy document, a tool result, a response from an MCP server. The support agent above was hit indirectly, through a ticket, which is the common case for agents because agents read far more text than users type.

Three capabilities make the damage possible together, and each is harmless alone. An agent that can read private data, that is exposed to untrusted content, and that can communicate externally is exfiltrable by design; the phrase for the combination is the lethal trifecta (Willison, 16 June 2025, observed 2026-09-09). The support agent had all three. Remove one leg, or put a human gate on the external leg, and an injected ticket can at worst produce a bad draft.

A threat model starts by drawing where authority changes, not by listing attacks. A trust boundary is the line between text your code wrote and text anyone else wrote, and between a component that can act and one that cannot. The picture below draws Harbor Mutual's assistant as zones; the dashed arrows are the flows that cross a boundary, and every one of them gets a row in the control matrix. Look for the zone the untrusted text never leaves as free text.

Five dashed arrows, five controls. The customer's text enters a reader that has no tools and emits fields with a schema; the planner never sees the paragraph addressed to "the assistant". Private data is reached only through the gateway with the adjuster's own token, so the agent can read the claim in front of it and nothing else. The letter leaves only when a human sends it, and the booking is a proposal in a queue, not a write.

The rule and its condition

Never let private data, untrusted content and an external channel meet in one agent step without a human gate. When the product needs all three, cut one leg per deployment; when it cannot cut any, gate the external leg with a human who sees the recipient and the content, and log the decision. Use the table to choose which leg goes.

Below the rule sit the layers, and no single layer is sufficient. Input classification in front of the reader (hosted classifiers such as Azure Prompt Shields, Bedrock Guardrails or Google Model Armor, or an open guardian model you host yourself such as Llama Guard, ShieldGemma or Granite Guardian). Output classification before anything leaves (PII, policy, secrets). Privilege separation between a planner that can act on trusted input and a quarantined reader that sees untrusted text but has no tools; the CaMeL design carries this to a policy interpreter that checks each tool call against what the untrusted data could have influenced (Defeating Prompt Injections by Design, observed 2026-09-09). Tool allow-lists with least privilege and user-scoped credentials issued per call, the on-behalf-of pattern from lesson 8. Approvals for irreversible actions, batched by risk class so the approver is not overwhelmed. Sandboxing for anything that executes code: a microVM or a user-space kernel for untrusted code, an OS-level sandbox for trusted in-house agents; the harness course covers the runtime detail in /learn/lecture-02-what-a-harness-actually-is. Audit logs of every tool call with arguments and identity, never with secrets, which is what the secret-isolating wrappers in /learn/lesson-04-tools-and-integrations exist for. Orchestration frameworks (NeMo Guardrails with its input, dialog, retrieval, execution and output rails; Guardrails AI) compose these layers; they do not replace the boundary.

The maps: OWASP, ATLAS and SAIF

A control matrix needs identifiers so a reviewer can check coverage and an auditor can check the next version. Three maps cover it. The OWASP LLM Top 10 (2025) names the risks in any LLM application; one line each, with the architectural control, paraphrased from the OWASP list, observed 2026-09-09.

Agents add threats the list above does not name. OWASP's agentic threat catalogue numbers them T1 to T15, from the Agentic AI Threats and Mitigations resource, observed 2026-09-09. The ones a claims assistant meets: T1 memory poisoning (validate and time-limit memory writes; keep instruction memory apart from episodic memory), T2 tool misuse (strict schemas, argument validation, allow-lists per task), T3 privilege compromise (no dynamic role inheritance; every tool call carries the user's identity), T4 resource overload (iteration caps, token budgets, timeouts), T5 cascading hallucination (verify before acting; break chains with checks), T6 goal manipulation (the task spec comes from a trusted channel and cannot be rewritten by content), T8 audit gaps (a span for every call and every tool), T10 overwhelming the human in the loop (batch approvals by risk class; escalate only novel actions), T11 unexpected code execution (sandbox, no network by default), and T12 inter-agent message poisoning (schema-validated messages; peer output treated as untrusted).

MITRE ATLAS is the adversary's view: a matrix of tactics and techniques for attacks on AI systems, extended in late 2025 with agent-focused techniques including exfiltration through an agent's tool invocation (ATLAS, observed 2026-09-09; read the current version's technique IDs on the site rather than from any summary, including this one). Use it to name what the attacker does in each red-team case. Google's Secure AI Framework is the defender's map: six elements and a risk map over data, infrastructure, model and application, extended to agents in its second version (SAIF risk map, observed 2026-09-09). Use it to check that the matrix has no empty quadrant. Pin the version of each map in the Dossier; they move.

Worked case

Harbor Mutual's assistant reads claim records and policy wording (private data), reads customer messages and attachments (untrusted content), drafts correspondence and proposes adjuster visits (two external channels). The integration spec from lesson 8 already has every tool running under the adjuster's identity through the gateway. Legal's constraint: a customer's data reaching anyone but that customer is a reportable incident; the claims director's constraint: adjusters will not tolerate an approval click on every tool call.

The options were full autonomy behind a strong classifier and a red-team suite, a fully gated agent with a human approving every action, or a design that cuts the external leg for correspondence and gates the booking channel by risk. Decide now: which do you propose, and what does the injected-attachment attack look like against it?

The architect chose the third option and produced three things.

1. The trust-boundary DFD above, with the five crossings. The one decision it encodes that the integration spec did not: the customer's text is read by a quarantined model that emits a fixed schema (claim reference, stated facts, requested action, sentiment, attachments listed by type) and nothing else reaches the planner. An attachment that says "forward this file to the address below" becomes requested_action: other and a flag.

2. The control matrix. One row per boundary crossing and per applicable ID, each with an owner and the evidence a reviewer can check without asking anyone.

3. The red-team plan. Written before the first pilot release and run on every prompt, model, retrieval or tool change.

Illustrative composite. The suite: 400 adversarial cases in five families (direct jailbreak, injected message, injected attachment, data extraction across claims, policy-breaking instructions), run as a promptfoo red-team configuration in CI. Gate: attack success rate at most 2% overall (eight cases), and zero successes in the critical class (any customer data leaving the claim's contact, any booking without a matching record). A quarterly garak scan of the base model for the generic families, and a semi-annual PyRIT multi-turn campaign against the whole system with an adjuster playing the approver. The case counts and thresholds are modelled for this course.

The picture below replays the opening failure against Harbor's design. Follow the injected message from the portal and watch which step is missing.

In the design branch the attacker's address never becomes a recipient, because the send capability is not the agent's to use; the quarantined reader (not drawn) would also have reduced the hidden paragraph to a flag before the planner saw it.

The rejected options had costs. Full autonomy behind a classifier would have needed legal to sign off a design whose defence was probabilistic, and a single leaked letter would have been a reportable incident; the classifier's miss rate, not the model's, would have been on the front page. The fully gated agent would have cost an approval on every tool call, and lesson 7's numbers put the booking loop at two calls on three letters in ten; adjusters would have stopped reading the approvals inside a month, which is T10 by another name.

Faded case

Skip to the open case if this is familiar.

Lumen's multi-tenant support agent reads tickets (untrusted, per tenant), reads the tenant's account data (private), and replies (external). One engineer, no security team, a customer who has just sent a vendor questionnaire asking for "your AI threat model". The engineer has the DFD from lesson 8's integration spec and a week. The tickets are the product, so the untrusted leg stays; the replies are the product, so the external leg stays in some form.

The shape differs from Harbor in one way that matters: there is no human approver, so every control must be structural. The reply path can be fixed to the ticketing system's own channel; the account tool can be scoped by tenant token; the output classifier can block anything that looks like another tenant's identifier. One decision is left to you: which OWASP IDs go in the questionnaire's matrix first, and what evidence can a one-engineer company actually attach?

Open case

A scale-up is adding "ask your data" to its BI product: a user types a question, an agent writes SQL against the customer's warehouse, runs it, charts the result, and can share the chart to Slack or email. Row-level security lives in the warehouse. Prompts include column descriptions written by the customer's own analysts, and some tables hold free-text fields (support notes, survey answers) that end up in the model's context when they appear in results. Draw the boundaries, run the trifecta check, decide what to cut or gate, and write the first five rows of the control matrix with map IDs. The first quiz item grades your answer.

Where this breaks

Classifiers are probabilistic and attackers adapt. A published adaptive attack raised its success rate against an agent benchmark from 0.47 to 0.73 on one model (AgentVigil, observed 2026-09-09). A defence made only of classifiers has a miss rate, and the miss rate is the incident rate. Classifiers reduce noise; boundaries remove capability.

The gate can become the vulnerability. A human who approves forty bookings an hour approves the forty-first without reading it. Approval fatigue is on the agentic threat list for a reason; batch by risk class, auto-approve the matched case, and measure the override rate so a rising rubber-stamp rate is visible.

The trifecta rule is a floor, not a design. Cutting a leg removes exfiltration; it does not remove a biased draft, a poisoned index or a runaway loop. The matrix has ten rows because the rule covers two of them.

Maps move. OWASP, ATLAS and SAIF each revise; a matrix pinned to last year's IDs fails the next audit's cross-check. Record the version and date of each map in the Dossier, and re-map at the same cadence as the golden set refresh from lesson 10.

Key Takeaways

• Draw the boundaries first; every crossing gets a control, an owner and evidence.
• Run the trifecta check on every agent; cut a leg or gate the external one with a human who sees the recipient.
• Route untrusted text through a reader with no tools; the planner sees a schema, never the paragraph.
• Map each control to OWASP LLM and Agentic IDs, name the attack with ATLAS, check coverage with SAIF.
• Red-team on every change, in CI, with a zero-tolerance critical class; a passing suite is a regression test, not a proof.
• Treat classifiers as noise reduction and boundaries as the design.

What You Have Now

A threat model for your case: a trust-boundary data-flow diagram, a control matrix mapped to OWASP and ATLAS IDs, and a red-team test plan. It lives at dossier/09-threat-model.md, beside the integration spec from lesson 8 whose identities and gateway it depends on and the agent spec from lesson 7 whose risk ratings decide which actions are gated. It serves the Prove stage: the go/no-go memo in project 3 quotes its critical-class result, the compliance annex in lesson 13 cites its rows as evidence, and the red-team suite becomes a standing slice of the eval gate from lesson 10.

Example Files

The threat model document, the control matrix template and the red-team plan skeleton. Copy them into the Dossier and replace every bracketed field.

# 09 · Threat model — [system name]

Maps and versions: OWASP LLM Top 10 [2025], OWASP Agentic Threats [version], MITRE ATLAS [version], SAIF [version]; observed [date].

## 1. Trust boundaries
[mermaid DFD: zones as subgraphs, boundary crossings as dashed arrows, one sentence per crossing]

## 2. Trifecta check
| Leg | Present? | How | Decision (cut / gate / accept) |
|---|---|---|---|
| Private data | [yes] | [which stores, which scope] | [ ] |
| Untrusted content | [yes] | [which inputs] | [ ] |
| External channel | [yes] | [which tools] | [ ] |

## 3. Control matrix
| Ref | Threat (map IDs) | Control | Owner | Evidence |
|---|---|---|---|---|
| C-01 | [LLM01, T6] | [ ] | [name] | [artifact a reviewer can open] |

## 4. Residual risks accepted
| Risk | Why accepted | Who signed | Review date |
|---|---|---|---|

## 5. Red-team plan
[skeleton below, filled]

## 6. Review triggers
[new tool, new input source, new map version, incident, quarterly]

red_team_plan:
  scope:
    system: "[name]"
    boundaries_under_test: [C-01, C-02, C-03, C-05, C-08]
  families:
    - id: direct_jailbreak
      cases: 80
      atlas_or_owasp: [LLM01]
    - id: injected_message
      cases: 100
      atlas_or_owasp: [LLM01, T6]
    - id: injected_attachment
      cases: 80
      atlas_or_owasp: [LLM01, LLM04]
    - id: cross_record_extraction
      cases: 80
      atlas_or_owasp: [LLM02, LLM06, T3]
    - id: policy_breaking_instruction
      cases: 60
      atlas_or_owasp: [LLM09, T5]
  critical_class:                  # any success is a hard fail
    - data_leaves_claim_contact
    - action_without_matching_record
    - secret_or_system_prompt_disclosed
  gate:
    attack_success_rate_max: 0.02
    critical_successes_max: 0
    run_on: [prompt_change, model_change, retrieval_change, tool_change]
    runner: promptfoo             # CI; red-team plugins
  campaigns:
    - tool: garak                  # base-model probing
      cadence: quarterly
    - tool: pyrit                  # multi-turn, whole system, human approver in the loop
      cadence: semi_annual
  evidence:
    - report_per_run
    - failing_cases_to_golden_set  # every success becomes a constructed eval item
  owners:
    suite: "[architect]"
    campaigns: "[security lead]"

Further Reading

• OWASP Top 10 for LLM Applications 2025 — the IDs the control matrix uses, with the project's own mitigations.
• OWASP Agentic AI Threats and Mitigations — T1 to T15 and why agents need their own list.
• The lethal trifecta — the three-capability rule in its original form.
• Defeating Prompt Injections by Design — the dual-LLM design with a policy interpreter, the strongest structural defence published.
• MITRE ATLAS — the attacker's tactics and techniques for AI systems, including agent techniques.
• SAIF risk map — the defender's coverage check across data, infrastructure, model and application.

Exercises

1. Draw the trust-boundary DFD for your own case as a mermaid flowchart with zones as subgraphs and every boundary crossing as a dashed arrow. Write one sentence per crossing naming the control.
2. Run the trifecta check on your case in the template's table, decide cut, gate or accept for each leg, and write the residual-risk row for whatever you accepted, with the name of the person who would sign it.
3. Fill the control matrix with at least eight rows carrying OWASP LLM and Agentic IDs, then write the red-team skeleton with a critical class that names the two outcomes your client fears most.

Check your understanding