VaikoraVaikora › Blog › Developer Guides
Secure AI Agent Architecture: Developer Patterns Guide
What Is Secure AI Agent Architecture?
Secure AI agent architecture is a design pattern that enforces least-privilege access, policy-based action approval, and complete audit trails between an AI agent and its external capabilities. The agent makes a decision. A policy layer evaluates whether that decision is permitted under current rules and context, and only then does the tool execution proceed. This layered approach prevents unauthorized actions, contains prompt injection, enforces business constraints, and creates tamper-proof records of every decision. Unlike hand-coded guardrails embedded in each agent, a centralized policy layer scales across multiple agents and teams while maintaining consistent enforcement and compliance.
Why AI Agent Security Matters Now
AI agents are fundamentally different from traditional applications. They're autonomous: they make independent decisions about which tools to call, what data to access, and what actions to take, all based on user input and the current context. This autonomy is powerful for automation, but it's also the core security risk.
A compromised prompt, a jailbreak attempt, or even benign user input misinterpreted by the model can cause an agent to execute actions it shouldn't. An agent might delete records, transfer money, expose sensitive data, or modify critical infrastructure without explicit user intent. The OWASP LLM Top 10 (LLM06 Excessive Agency) and the OWASP Agentic Security Initiative both flag this as a top-tier risk: agents with unrestricted tool access create systemic liability.
Traditional security perimeters assumed humans authorize actions before execution. AI agents reverse that flow: authorization must happen at runtime, after the agent has already decided what it wants to do. Secure agent architecture solves this by inserting a policy enforcement layer that catches dangerous decisions before they execute.
The Three Pillars of Secure Agent Architecture
1. Layered Design: Agent, Policy, Tools
The cleanest secure architecture is three-tier. The agent (LLM + orchestration logic) lives in one layer. The policy enforcement layer sits in the middle. Tools and external systems sit behind that.
The agent calls the policy layer with a proposed action: "Call GET /api/orders with user_id=42." The policy layer checks that action against rules: Is this agent allowed to call that endpoint? Does it have the right scope? Is the parameter within safe bounds? Is there a high-risk flag that requires human approval? The policy layer responds ALLOW, LOG, CONSTRAIN, or BLOCK. Only ALLOW actions proceed to the tool.
This design works because: - The policy layer is stateless and fast, so you don't add noticeable latency to agent execution. - You can update policies without redeploying agents or LLMs. - Every decision is loggable and auditable. - You can test policy rules independently of agent behavior.
2. Least-Privilege Tool Scoping
Agents should never have access to all tools or all data. Define a minimal capability set per agent, per user, per context.
If an agent is a customer support chatbot, it needs to read support tickets and append notes. It doesn't need to delete customer records, transfer money, or access payroll data. Define its tool scope explicitly: it can call GET /tickets/{id}, POST /tickets/{id}/notes, and nothing else.
When an agent tries to call a tool outside its scope, the policy layer blocks it with a clear denial reason. This is harder than it sounds in practice: you need to track which agent is running, who invoked it, what role they have, and what tools they should have access to. But the payoff is enormous. Least privilege prevents lateral movement if an agent is compromised or misdirected.
3. Append-Only Audit Trails
Every action an agent proposes to take must be logged to an append-only record. This log should capture: - The agent's identity and version. - The user or service that invoked the agent. - The proposed action (tool name, parameters, timestamp). - The policy decision (ALLOW/LOG/CONSTRAIN/BLOCK and reasoning). - The outcome (success, error, data returned).
Append-only means the log cannot be edited or deleted after it's written. This is not just compliance theater. Immutable logs are your evidence of what happened, who authorized it, and whether the system behaved correctly. They're also your forensic tool if something goes wrong.
Addressing OWASP LLM06 Excessive Agency
OWASP LLM06 describes a scenario where an AI agent has excessive agency, unchecked tool access, or weak output validation, allowing it to cause unintended harm. The concrete risks are: - An agent modifies data it shouldn't because it has write access it doesn't need. - Prompt injection tricks an agent into calling a tool with malicious parameters. - An agent calls tools in an unsafe order, e.g., deleting records before logging them. - An agent outputs sensitive data in a response because there's no DLP filter.
The secure architecture patterns above address each of these: - Least-privilege scoping prevents the agent from having write access to non-essential data. - Policy enforcement can reject suspicious parameter patterns (e.g., SQL injection syntax, exfiltration to external URLs). - The policy layer can enforce action ordering (e.g., "always log before delete"). - A data loss prevention (DLP) stage after tool execution can redact sensitive data from the agent's output.
The OWASP Agentic Security Initiative expands this with a framework for threat modeling agents: threats, mitigations, and controls at the agent design, model behavior, tool integration, and orchestration levels. Secure agent architecture is a mitigation at the orchestration level, one that applies across all agents in your system.
How to Build Secure Agent Architecture: Key Patterns
Pattern 1: Policy Engine as a Sidecar
Deploy the policy engine as a sidecar service, separate from the agent and tools. The agent makes an RPC call to the policy service with a proposed action, and gets back a decision.
Advantages: - The policy engine is language-agnostic. You can update it or replace it without touching agent code. - You can scale the policy engine independently. - You get a natural audit trail because every call is logged. - You can test policies with synthetic agent calls.
Trade-off: You add one network round-trip per action. Modern policy engines can return a decision in under 100ms, so this is acceptable for most use cases.
Pattern 2: Context-Aware Policies
Policies should be stateless, but they should evaluate context. When an agent proposes an action, the policy layer should know: - The agent's identity, version, and training data version. - The user or service that invoked the agent, and their role. - The sensitivity of the data being accessed. - Whether the action is high-risk (e.g., irreversible, affects payment systems, PII access). - Time-of-day, geographic location, or other contextual flags.
A simple policy might say: "Agent customer-support-v2, when called by a user with the support-staff role, can read customer orders but only for the customer account that the user is currently assisting." This requires the policy engine to know the user's role, the current customer context, and the agent's name.
Pattern 3: Human-in-the-Loop for High-Impact Actions
Some actions are too risky to auto-allow. Define a tier of high-impact actions and require human approval before they execute.
High-impact actions might include: - Deleting records. - Modifying sensitive data (PII, financial records). - Triggering external payments or refunds. - Accessing data outside the normal scope. - Bulk operations.
When an agent proposes a high-impact action, the policy layer returns a CONSTRAIN response: "This action requires human approval. Queuing for review." A human (or a secondary automated gate) then reviews the proposed action, the context, and the reasoning, and approves or rejects it. Only after approval does the tool execute.
This pattern is inspired by HITL (human-in-the-loop) design in safety-critical systems. It's not a blocker for every action, just the high-risk ones.
Pattern 4: Parameterized Policies
Policies should be data-driven, not hard-coded. Store policies in a configuration file or database, parameterized by agent, user role, tool, and resource sensitivity.
Example structure:
policies:
- name: "support-agent-ticket-access"
agent: "customer-support-v2"
user_role: "support-staff"
tool: "GET /tickets"
conditions:
- ticket_customer_id == user_assigned_customer_id
action: "ALLOW"
audit_level: "INFO"
- name: "support-agent-payment-access"
agent: "customer-support-v2"
user_role: "support-staff"
tool: "POST /refunds"
conditions:
- refund_amount < 500
action: "CONSTRAIN"
approval_required: true
audit_level: "WARN"
- name: "default-deny"
agent: "*"
action: "BLOCK"
audit_level: "ERROR"
Policies are evaluated in order. This approach makes it easy to update rules without code changes, and easy to see what's permitted and what's not.
Pattern 5: Threat Detection and Anomaly Flags
The policy engine should also detect common attack patterns and flag them for review or blocking. Common patterns include: - Prompt injection syntax in parameters (e.g., "ignore previous instructions"). - SQL injection patterns in database queries. - Attempts to access files or URLs outside the agent's scope. - Requests to exfiltrate large amounts of data. - Sudden spikes in action frequency from a single agent.
Detection can be rule-based (regex patterns, keyword detection) or statistical (deviation from normal behavior). Flag high-confidence detections as BLOCK or CONSTRAIN to require human review.
Code Example: A Minimal Policy Enforcement Layer
Here's a working example of a simple policy engine you can adapt. This is a Python implementation that evaluates policies against a proposed action.
from dataclasses import dataclass
from typing import Literal
from datetime import datetime
import json
@dataclass
class PolicyDecision:
action: Literal["ALLOW", "LOG", "CONSTRAIN", "BLOCK"]
reason: str
requires_approval: bool = False
audit_entry: dict = None
@dataclass
class ProposedAction:
agent_id: str
agent_version: str
user_id: str
user_role: str
tool_name: str
parameters: dict
timestamp: datetime = None
class PolicyEngine:
def __init__(self, policies: list):
self.policies = policies
def evaluate(self, action: ProposedAction) -> PolicyDecision:
"""Evaluate a proposed action against all policies."""
# Threat detection: check for suspicious patterns
if self._detect_threat(action):
return PolicyDecision(
action="BLOCK",
reason="Threat detection flagged suspicious pattern",
audit_entry={
"agent": action.agent_id,
"user": action.user_id,
"tool": action.tool_name,
"timestamp": action.timestamp.isoformat(),
"decision": "BLOCK",
}
)
# Find matching policy (first match wins)
for policy in self.policies:
if self._policy_matches(policy, action):
decision_type = policy.get("action", "DENY")
approval_required = policy.get("approval_required", False)
if decision_type == "ALLOW":
decision_action = "ALLOW"
elif approval_required:
decision_action = "CONSTRAIN"
else:
decision_action = decision_type
return PolicyDecision(
action=decision_action,
reason=f"Matched policy: {policy['name']}",
requires_approval=approval_required,
audit_entry={
"agent": action.agent_id,
"user": action.user_id,
"tool": action.tool_name,
"parameters": action.parameters,
"timestamp": action.timestamp.isoformat(),
"decision": decision_action,
"policy": policy["name"],
}
)
# Default deny
return PolicyDecision(
action="BLOCK",
reason="No matching policy found (default deny)",
audit_entry={
"agent": action.agent_id,
"user": action.user_id,
"tool": action.tool_name,
"timestamp": action.timestamp.isoformat(),
"decision": "BLOCK",
}
)
def _policy_matches(self, policy: dict, action: ProposedAction) -> bool:
"""Check if a policy matches a proposed action."""
# Agent match
if policy.get("agent") != "*" and policy.get("agent") != action.agent_id:
return False
# User role match
if "user_role" in policy and policy["user_role"] != action.user_role:
return False
# Tool match (exact or wildcard)
tool_pattern = policy.get("tool", "*")
if tool_pattern != "*" and tool_pattern not in action.tool_name:
return False
# Evaluate conditions
if "conditions" in policy:
for condition in policy["conditions"]:
if not self._evaluate_condition(condition, action):
return False
return True
def _evaluate_condition(self, condition: str, action: ProposedAction) -> bool:
"""Evaluate a condition string against action context."""
# Simple example: check parameter bounds
# In production, use a safer expression evaluator
try:
if "refund_amount <" in condition:
amount_str = condition.split("<")[1].strip()
max_amount = int(amount_str)
actual_amount = action.parameters.get("amount", 0)
return actual_amount < max_amount
return True
except:
return False
def _detect_threat(self, action: ProposedAction) -> bool:
"""Detect common attack patterns."""
suspicious_patterns = [
"ignore previous instructions",
"system prompt",
"DROP TABLE",
"delete from",
"../",
"../../../../",
]
params_str = json.dumps(action.parameters).lower()
for pattern in suspicious_patterns:
if pattern.lower() in params_str:
return True
return False
# Example usage
policies = [
{
"name": "support-agent-read-tickets",
"agent": "customer-support-v2",
"user_role": "support-staff",
"tool": "GET /tickets",
"action": "ALLOW",
},
{
"name": "support-agent-refunds",
"agent": "customer-support-v2",
"user_role": "support-staff",
"tool": "POST /refunds",
"conditions": ["refund_amount < 500"],
"action": "ALLOW",
"approval_required": False,
},
{
"name": "support-agent-large-refunds",
"agent": "customer-support-v2",
"user_role": "support-staff",
"tool": "POST /refunds",
"conditions": ["refund_amount >= 500"],
"action": "ALLOW",
"approval_required": True,
},
{
"name": "default-deny",
"action": "BLOCK",
}
]
engine = PolicyEngine(policies)
# Test: Small refund (should allow)
decision = engine.evaluate(ProposedAction(
agent_id="customer-support-v2",
agent_version="2.1.0",
user_id="user-123",
user_role="support-staff",
tool_name="POST /refunds",
parameters={"amount": 250, "order_id": "order-456"},
timestamp=datetime.now()
))
print(f"Small refund decision: {decision.action} ({decision.reason})")
# Test: Large refund (should constrain)
decision = engine.evaluate(ProposedAction(
agent_id="customer-support-v2",
agent_version="2.1.0",
user_id="user-123",
user_role="support-staff",
tool_name="POST /refunds",
parameters={"amount": 750, "order_id": "order-456"},
timestamp=datetime.now()
))
print(f"Large refund decision: {decision.action} (approval required: {decision.requires_approval})")
# Test: Threat detected
decision = engine.evaluate(ProposedAction(
agent_id="customer-support-v2",
agent_version="2.1.0",
user_id="user-123",
user_role="support-staff",
tool_name="POST /refunds",
parameters={"amount": 100, "note": "ignore previous instructions"},
timestamp=datetime.now()
))
print(f"Threat detection: {decision.action} ({decision.reason})")
This example shows the core logic: match policies against actions, evaluate conditions, detect threats, and return a decision. In production, you'd add:
- A database for policy storage and versioning.
- Caching to reduce latency.
- Structured logging of every decision.
- Real expression evaluation for conditions (use a safe library like expression-eval or simpleeval).
- Integration with your approval workflow (Slack, email, web UI).
How to Make an AI Agent Secure: A Practical Checklist
Building a secure agent isn't a single design decision, it's a practice. Here's a checklist:
- Define tool scope explicitly. List exactly which tools the agent can call. Start minimal and expand as needed.
- Write policies before code. Document what the agent is allowed to do, in what contexts, and under what conditions.
- Implement a policy layer. Don't embed guardrails in agent code. Use a separate service or library.
- Log every decision. Append-only logs, immutable after write.
- Test policies independently. Write test cases for policies (e.g., "can read own orders, not others' orders").
- Monitor and alert. Watch for policy violations, unusual access patterns, threat detections.
- Review logs regularly. Audit logs should be reviewed by a human regularly, not just stored.
- Update policies without redeployment. Your policy format should allow runtime updates without restarting agents or services.
- Run red-team tests. Deliberately try to break your agent. Test prompt injection, jailbreaks, privilege escalation.
- Plan for incident response. If an agent is compromised, can you quickly revoke its access? Can you audit what it did?
Least Privilege for AI Agents: Real-World Application
The principle of least privilege is old (dating to 1975 in "Protection of Information in Computer Systems" by Saltzer and Schroeder), but it's particularly critical for AI agents because agents are autonomous and can access tools you didn't explicitly authorize in the moment.
Here's how to apply least privilege to an agent:
- Role-based tool access. Map user roles to agent capabilities. A junior support agent should not have the same tool scope as a senior engineer.
- Resource-based scoping. An agent serving customer data should access only the specific customer's data, not all customers.
- Time-based restrictions. Some actions might be allowed during business hours but not at 2 AM.
- Request-size limits. An agent reading historical data should have limits on batch size to prevent accidental DoS.
- Approval gates for mutations. Read operations can be fast, but writes (especially deletes) should require approval.
- Data sensitivity tags. Mark data as public, internal, sensitive, or PII. Restrict agent access based on tags.
What Is the Safest AI Agent Architecture?
The safest architecture combines all the patterns above:
- Isolation. The agent runs in a containerized environment with network restrictions. It can only call known external services via the policy layer.
- Policy enforcement. Every tool call goes through a policy engine. No direct tool access.
- Minimal tooling. The agent has access to the smallest set of tools that solves the problem.
- Approval workflows. High-risk actions require human review before execution.
- Immutable audit trails. Every decision is logged to an append-only store.
- Intrusion detection. The policy engine detects and blocks suspicious patterns.
- Regular red-teaming. Security testing is continuous, not a one-time gate.
- Incident response. There's a documented process for quickly revoking agent access, quarantining logs, and remediating compromises.
This architecture trades some automation speed for strong security guarantees. It's appropriate for agents that touch sensitive data or mission-critical systems. For low-risk agents (e.g., a simple Q&A chatbot), you can relax some controls.
Regulatory Context: OWASP, NIST, ISO 42001, and AI Act
Your agent architecture should align with emerging standards and regulations:
- OWASP LLM Top 10 and Agentic Security Initiative. Framework for identifying and mitigating LLM risks. LLM06 Excessive Agency is the core threat this architecture addresses.
- NIST AI RMF (Risk Management Framework). Guidance on risk management for AI systems, including governance, mapping, measuring, and managing risks.
- MITRE ATLAS (Adversarial Threat Environment for Artificial-Intelligence Systems). Repository of adversarial tactics and techniques for AI systems. Includes agent-specific threats like prompt injection and model stealing.
- ISO 42001 (AI Management Systems). Emerging standard for managing AI system risks. Requires controls on tool access, model governance, and incident response.
- EU AI Act. Regulatory framework for high-risk AI systems. Requires risk assessments, documentation, human oversight, and audit trails. Agents that handle financial or safety-critical decisions fall into the high-risk category.
- ISO 27001 and SOC 2. Existing standards for information security and compliance. Agent systems should implement access controls, logging, and audit trails per these standards.
A secure agent architecture is your mechanism for demonstrating compliance across all these frameworks.
How Vaikora Helps
Building a policy engine from scratch is feasible but high-friction. The policy evaluation logic is small, but the surrounding machinery (configuration management, logging, performance optimization, threat detection, approval workflows) adds up.
Vaikora's open-core offering (the vaikora-llm-gateway and vaikora-guard-mcp, both MIT-licensed) provides the policy engine as a drop-in service. You define policies in YAML or JSON, Vaikora evaluates them at runtime against agent actions, and you get structured decision logs. The gateway caches policy decisions to minimize latency.
For enterprises needing compliance presets (SOC 2, HIPAA, GDPR, PCI DSS, ISO 27001), approval workflows, and a hosted control plane with a dashboard and SLA, Vaikora Control Plane adds those layers without requiring you to build compliance infrastructure from scratch.
The point is the pattern, not the tool. Any architecture that enforces least-privilege policy, HITL for high-risk actions, and immutable audit trails will be secure. Vaikora is one way to implement that pattern at scale.
Frequently asked questions
How do you make an AI agent secure?
Make an agent secure by defining its tool scope (minimal capabilities per role), inserting a policy enforcement layer that evaluates every proposed action, requiring human approval for high-impact operations like deletes or payment transfers, and maintaining an append-only audit log of every decision. Test the agent adversarially with prompt injection and jailbreak attempts, and review audit logs regularly to catch unauthorized or suspicious behavior.
What is the safest AI agent architecture?
The safest architecture is a three-tier design: agent (LLM + orchestration) in one layer, a policy enforcement layer in the middle that approves or blocks actions, and tools and external systems behind that. Combine least-privilege tool scoping, human-in-the-loop approval for high-risk actions, threat detection (prompt injection, suspicious parameter patterns), immutable audit trails, network isolation, and regular red-team testing.
How do you limit what an AI agent can do?
Limit an agent's capabilities by defining a minimal tool set per agent per user role, writing policies that specify which tools are allowed under which conditions, using role-based access control to map user permissions to agent permissions, and implementing size limits and rate limits on agent operations. A policy engine evaluates every tool call against these rules and blocks or requires approval for any out-of-scope requests.
What is least privilege for AI agents?
Least privilege for AI agents means each agent has access to only the minimum set of tools and data required to accomplish its task. If an agent supports customers, it should read customer tickets but not have write access to financial records or the ability to delete user accounts. Least privilege prevents lateral movement if an agent is compromised or misdirected by a jailbreak or prompt injection attack.
What does OWASP LLM06 Excessive Agency cover?
OWASP LLM06 Excessive Agency describes the risk that an AI agent has too much autonomy, unrestricted tool access, or weak approval workflows, allowing it to cause unintended harm like unauthorized data deletion, money transfers, or sensitive data exposure. Mitigations include defining minimal tool scope, enforcing policies at runtime, requiring human approval for high-impact actions, and maintaining audit trails.
How do I know if my agent architecture is compliant with ISO 27001 or SOC 2?
Agent architecture is compliant with ISO 27001 and SOC 2 if it implements access controls (policy enforcement), secure logging (append-only audit trails), regular security testing (red-teaming), incident response procedures (quick access revocation), and governance (documented policies). Compliance is verified by audit, so documentation and evidence of controls matter as much as the controls themselves.
Should I use a third-party policy engine or build my own?
Start with a third-party engine if you need to move quickly, support multiple teams, or need compliance presets. Build your own if your policies are simple, your threat model is low-risk, or you have specific integration requirements. The pattern (policy layer, least privilege, audit trails) is more important than the tool.
How often should I review agent audit logs?
Review audit logs at a minimum weekly, more frequently if the agent handles sensitive data or payment systems. Automated alerting on policy violations (BLOCK decisions, threat detections, approval queue size) is also essential. A daily or weekly summary of agent activity per role or per agent helps catch anomalies early.
See Vaikora enforce policy on your AI
Open-core AI runtime control. Self-host the MIT gateway free, or run the hosted Control Plane.
Get a demo Self-host the gateway