How to Design a Secure AI Agent: Make Unsafe Actions Impossible

The hard part of securing an AI agent is accepting a slightly uncomfortable premise: prompt injection is not a bug we can reliably filter away.

An agent reads natural language from multiple places—user requests, emails, web pages, PDFs, search results, tickets, and tool responses. The model cannot reliably tell which of those words are authoritative instructions and which are merely data. A malicious document can therefore try to redirect the agent into revealing data, sending a message, changing infrastructure, or making a purchase.

The useful question is not, “How do I make the model ignore every malicious instruction?” It is: “What damage is possible if the model follows one?”

This shift leads to a more dependable design principle:

Treat the model as an untrusted planner. Put security decisions and consequential actions behind deterministic, testable boundaries.

Start with the dangerous combination

Simon Willison calls the most dangerous combination the lethal trifecta:

access to private data or sensitive systems;
exposure to untrusted content; and
the ability to communicate externally.

An inbox assistant shows why this matters. It reads an email from an attacker, can search the user’s mailbox, and can send mail. The attacker’s email tells the agent to find password-reset emails and forward them elsewhere. The model does not need to be “hacked” in the traditional sense: it just needs to treat hostile text as an instruction.

Meta’s Agents Rule of Two turns this into a practical architecture check. In one session, an autonomous agent should have no more than two of these capabilities:

Process untrustworthy input.
Access sensitive systems or private data.
Change state or communicate externally.

If a workflow truly needs all three, add a reliable break in the attack chain: a human approval, a new context window with a carefully validated hand-off, or a deterministic policy engine. Do not let one model context read hostile data, access sensitive data, and act on the outside world autonomously.

Give the model proposals, not power

The safest agent designs make an important distinction:

The model can recommend an action.
Application code decides whether that action is valid and permitted.

Avoid giving an LLM a general-purpose execute_sql, shell, browser, payment, or HTTP tool. Those tools turn a safety policy into a sentence in a prompt. Instead, expose narrow operations with typed inputs and enforce policy outside the model.

For example, a support agent should not emit SQL:

LLM output:
  QueryProposal {
    report: "failed_payments_by_country",
    date_range: "last_7_days",
    country: "GB"
  }

Policy service:
  - validates the schema
  - maps the report to a pre-approved query template
  - checks caller and tenant authorization
  - applies row, column, and time-range limits
  - records an audit event

The model can choose from a small vocabulary of useful actions, but it cannot invent a new database query or bypass tenant isolation. This is close to the action-selector and plan-then-execute patterns in Design Patterns for Securing LLM Agents against Prompt Injections: constrain the system so untrusted content cannot trigger a consequential action.

A reference architecture

Untrusted content ──> Quarantined reader/model ──> typed proposal
       │                                               │
       │                                               v
       └──── never receives privileged credentials  Policy engine
                                                        │
                         ┌──────── denied / approval ──┤
                         │                              v
User identity + policy + resource limits ─────> Narrow action service
                                                        │
                                                        v
                                              audited, scoped execution

This is defence in depth, but the layers have different jobs:

The model layer extracts, summarises, classifies, and proposes.
The policy layer makes deterministic allow/deny decisions.
The action layer uses narrowly scoped credentials to perform one approved operation.
The audit layer captures the request, policy decision, action parameters, result, model/version metadata, and approver where applicable—while redacting secrets and sensitive payloads.

The security boundary is the policy and action layer, not the system prompt.

Separate reading from acting

For agents that must use untrusted material, a useful default is a two-stage workflow:

A low-privilege worker reads web pages, attachments, tickets, or tool output. It has no production credentials and no external side-effect tools.
It emits a small, typed result: extracted facts, citations, a classification, or a proposed plan.
A privileged workflow validates that result, fetches only the data it needs, and either executes a restricted action or requests approval.

Keep the hand-off deliberately narrow. Passing raw document text, hidden instructions, or a free-form “do this next” plan into the privileged context simply recreates the original risk.

For higher-risk workflows, fix the plan before the agent sees untrusted data, or use an approval boundary between planning and execution. A one-way transition can also help: after browsing the open web, discard that context before accessing internal systems. The point is to break the path from attacker-controlled input to a meaningful capability.

Design tools as security products

Tool descriptions, schemas, and return values are all part of an agent’s attack surface. This matters especially for MCP deployments, where the model may see tools from several servers in the same context. The OWASP MCP Security Cheat Sheet highlights tool poisoning, rug pulls, confused deputies, excessive permissions, and cross-server data flows.

Build tools with the same care as public APIs:

Give every tool a single purpose and a strict schema. Reject undeclared fields.
Use per-tool, short-lived credentials with the smallest possible scope.
Re-authorize against the calling user on every request; do not let a server’s broad service identity become a confused deputy.
Allowlist destinations, resource IDs, and operations. Never let the model turn arbitrary text into a URL, shell command, or file path.
Pin and review tool definitions and versions. A tool that changes after approval is a supply-chain event.
Isolate servers and prevent data from one server flowing into another without an explicit policy.
Require meaningful approval for money movement, destructive changes, external communications, and data disclosure. Show the real parameters, not only a friendly summary.

Put limits around blast radius

Least privilege is necessary but incomplete. An agent can still be harmful while operating within its granted permissions, especially at high speed. Add limits that make a mistake bounded and reversible:

read-only by default;
tenant and data-class isolation for retrieval and RAG indexes;
limits on records, bytes, tokens, tool calls, retries, runtime, and spend;
idempotency keys and a rollback path for mutations;
network egress restrictions and destination allowlists;
sandboxed execution without ambient credentials;
staged environments before production; and
kill switches plus anomaly alerts for unusual tool use.

These controls also address availability and cost abuse. A looping agent is not merely inefficient; it can be a denial-of-service or denial-of-wallet incident.

Treat RAG and memory as data systems

Retrieval does not neutralise untrusted instructions. It is another path by which hostile content enters the context window. Treat embeddings, document stores, conversation logs, cached responses, evaluation data, and tool traces as first-class data stores.

For each one, define:

who can write to it;
who can retrieve from it;
tenant and classification boundaries;
retention and deletion rules;
encryption and access logging; and
how poisoning or suspicious content is detected and removed.

This is a much better starting point than asking only whether the source documents are “trusted.” Most useful agents will need to handle material that is not.

Test the architecture, not just the prompt

Prompt-injection test suites are still useful, but they cannot prove that a filter is secure. The paper The Attacker Moves Second found that adaptive attacks bypassed 12 recent jailbreak and prompt-injection defences, with over 90% attack success against most of them. A static list of known bad strings is therefore a regression test, not a security guarantee.

Test the system with adversarial goals such as:

make the agent disclose data across a tenant boundary;
smuggle sensitive data through a permitted outbound channel;
persuade the agent to use an unintended tool or parameter;
trigger an unapproved state change;
poison retrieval so later tasks adopt malicious instructions; and
exhaust its time, tool-call, or cost budget.

The most valuable assertions are deterministic: an untrusted reader never receives privileged credentials; a proposal with an unknown field is rejected; an unapproved recipient cannot receive data; an action has a tenant-bound authorization decision; and a tool-definition change blocks execution.

A practical shipping checklist

Before shipping an agent, be able to answer these questions:

Which inputs can an attacker influence, directly or indirectly?
Which sensitive data and systems can the agent reach?
Which actions can change state or send data outside the trust boundary?
Does any one context combine all three? If so, where is the enforced break or approval?
Which decisions are made by code rather than model text?
Are tools narrowly scoped, versioned, isolated, and authorized per user?
What prevents cross-tenant access, arbitrary egress, loops, and excessive spend?
Can we reconstruct what happened without logging secrets or sensitive payloads?
Have we tested adaptive attacks against the complete workflow?
Can we disable the agent or a single capability quickly?

The bottom line

Useful agents need capabilities. Secure agents make those capabilities explicit, small, observable, and independently enforced. They assume hostile instructions will reach the model and make sure that the model’s response is only a proposal—not authority.

That trade-off is not a failure of ambition. It is how we build agents that are safe enough to deserve real responsibility.