AI Threat Model
Traditional applications execute predefined code paths. AI agents also interpret natural language, retrieve untrusted content, select tools, and generate actions. The security boundary therefore includes every source that can influence the model and every system the model can affect.
Attack Surface
Review these surfaces for every production agent:
| Surface | Typical risk | Primary control |
|---|---|---|
| User prompt | Instruction override, data extraction, unsafe request | Input policy, Prompt Shield, scoped permissions |
| Retrieved document or website | Indirect prompt injection, poisoned knowledge | Treat content as data, source review, output constraints |
| System prompt and skills | Excessive authority or unclear boundaries | Controlled editing, version review, least privilege |
| Memory and data collections | Unauthorized or stale context | Ownership, classification, access filters, review cadence |
| Model output | Hallucination, harmful or biased result | Evaluation, grounding, human review, content safety |
| Tool response | Malicious instructions or unexpected data | Validate tool output, isolate it from instructions |
| Tool execution | External communication, deletion, spend, privilege change | Function-level policy and human approval |
| Model provider | Processing location, retention, availability, model change | Approved model inventory and deployment review |
Minimum Threat-Modelling Questions
For each agent or workflow, document:
- Who can invoke it and whether anonymous access is possible.
- Which datasets, memory collections, files, and page context it can read.
- Which tool functions it can call and which functions modify external state.
- What a malicious user or malicious document could cause it to disclose or change.
- Which actions require human approval and who can approve them.
- Which events are logged and how an investigator can correlate them.
- Which model deployment processes prompts and where processing may occur.
- What happens when a provider, credential, index, or approval flow fails.
Review Trigger
Repeat the threat model when access, tools, datasets, prompts, models, public interfaces, or workflow logic changes. A previously low-risk assistant can become high risk as soon as it receives a write-capable connection or sensitive data source.
See Prompt Injection, Tool Governance, and Layered Security Architecture.