Building Production AI Agents: Architecture and Oversight

A sound production AI agents architecture limits what the model can do, not only what it can say. In practice that means a loop the code controls, a small set of narrowly scoped tools, permissions enforced outside the model, approval gates before consequential actions, and tracing that shows what happened on each run. This guide walks through those parts in the order a Kenyan engineering or operations team would need to decide them.

Why the architecture comes before the model

Many failures of an agent in a live business come from design choices: the agent could reach a system it did not need, ran more steps than the task required, or acted on text it should not have trusted.

Anthropic's engineering guide separates two kinds of system. A workflow is one in which "LLMs and tools are orchestrated through predefined code paths." An agent is one in which "LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks" (Anthropic, Building effective agents). The same guide recommends that teams "find the simplest solution possible, and only increasing complexity when needed."

The practical rule follows from that. If a process can be written as fixed steps, build it as a workflow with a model call at one or two points. Use an agent loop only where the number of steps cannot be predicted in advance. Agents bring "higher costs, and the potential for compounding errors," so they need a task that justifies the extra control work.

The agent loop and its stopping conditions

A production agent loop has a small, repeatable shape:

• Build the context: the task, the rules, the relevant memory and the current state.
• Ask the model for its next action, which is either a tool call or a final answer.
• Check the requested action against the permission rules, then run it if it is allowed.
• Add the tool result to the context and repeat, until a stopping condition is met.

The stopping conditions matter as much as the loop. Anthropic's guide notes that "it's also common to include stopping conditions" such as "a maximum number of iterations" (Anthropic, Building effective agents). Add a hard cap on steps, a timeout per run, and a rule that sends the task to a person when the agent is blocked or uncertain.

Tool design: fewer, clearer, token-aware

Tools are the agent's contract with the systems it can touch. Anthropic's engineering team writes that "tools define the contract between agents and their information/action space" (Anthropic, Effective context engineering for AI agents). Three design habits follow from that.

First, give each tool one clear purpose and a precise description. Anthropic recommends describing each tool "to a new hire on your team," making implicit context explicit and enforcing "expected inputs and outputs" with strict data models (Anthropic, Writing effective tools for agents). A tool called get_order_status that accepts one order ID is easier for an agent to use correctly than a general query_database tool.

Second, return only the information the agent needs. The same guide suggests "pagination, range selection, filtering, and/or truncation" so that a broad query returns a bounded result.

Third, write errors the agent can act on. The guide asks for error responses that "clearly communicate specific and actionable improvements, rather than opaque error codes or tracebacks." An error such as "order ID must start with ORD- and be 10 characters" lets the agent retry correctly. A raw stack trace does not.

Least-privilege AI agent permissions

Permissions should be enforced by the system around the model. The model should never be the component that decides whether an action is allowed. OWASP's entry on excessive agency names three root causes: "excessive functionality," "excessive permissions," and "excessive autonomy" (OWASP, LLM06 Excessive Agency).

The OWASP guidance translates into a short set of design rules:

• Give each tool only the functions it needs. OWASP's example is a read-only document tool that also includes delete functions. That tool should not exist in that form.
• Grant the tool's service account the minimum database or API scope. A reporting agent should not hold write rights.
• Enforce authorization in downstream systems. OWASP advises implementing authorization "in downstream systems rather than relying on an LLM to decide if an action is allowed or not."
• Prefer narrow tools to general ones. A dedicated function for creating a refund request is safer than shell access.

Human in the loop AI agents: approval gates

An approval gate is a rule that pauses a run before a sensitive action. The model proposes the action, the system holds it, and a named person approves or rejects it. OWASP recommends "human-in-the-loop control to require a human to approve high-impact actions before they are taken" (OWASP, LLM06 Excessive Agency).

The OpenAI Agents SDK documents one way to build this. A tool can be marked to always need approval, or a rule can decide per call. The SDK documents that a callable approval rule fails closed when the arguments cannot be safely inspected, so the call then requires manual approval. A paused run exposes its pending items, which a reviewer approves or rejects before the run resumes (OpenAI Agents SDK, Human-in-the-loop). The same documentation makes an important point about consequential tools: "Approval permits a tool call to execute; it does not guarantee exactly-once execution." Use idempotency keys so that a retried call after a crash does not create a duplicate payment or message.

A practical approval policy has three tiers:

• Read-only actions run without approval.
• Reversible writes, such as a draft or an internal note, run and are logged.
• Irreversible or high-value actions, such as a payment, a customer-facing message or a deletion, wait for a person.

The thresholds belong to the business owner, not to the model. Record who approved each action and what the agent proposed, so that the decision can be reviewed later.

Memory types and what to store

Memory in an agent system covers several different things, and each needs its own rules. Anthropic's context guide describes two working patterns: an agent can write "notes persisted to memory outside of the context window," and it can use "compaction," which means summarising a conversation that is nearing the context limit (Anthropic, Effective context engineering for AI agents).

Anthropic's guide describes just-in-time retrieval as loading data "into context at runtime using tools." It also names "context rot" as a risk, since recall weakens as the context grows. Store less, keep the source with each note, and let old notes expire.

Prompt-injection defences

Prompt injection happens when text the agent reads, such as an email or a support ticket, contains instructions the agent follows. OWASP lists it as LLM01 and states that "it is unclear if there are fool-proof methods of prevention for prompt injection" (OWASP, LLM01 Prompt Injection). That is why the defences are layered rather than a single filter.

The OWASP entry recommends restricting the model's access "to the minimum necessary for its intended operations," and it advises handling some functions in code rather than giving them to the model. For agents, the layers are:

• Least privilege, so a successful injection has little to reach.
• Approval gates on consequential actions, so injected instructions cannot complete a payment or send a message unattended.
• Treating retrieved text as data, not instructions, and logging it.
• Input and output checks, including guardrails that run before or after the agent.

The OpenAI Agents SDK documents input and output guardrails, and also "tool guardrails" that "validate or block calls to those tools before and after execution" (OpenAI Agents SDK, Guardrails). The same page notes that parallel input guardrails can let the agent use tokens or run tools before a tripwire cancels the run, while blocking guardrails finish before the agent starts. For actions with side effects, choose the blocking option.

Failure handling and resumable runs

Agents are stateful, and failures compound. Anthropic's multi-agent research write-up says that "agents are stateful and errors compound," and that "when errors occur, we can't just restart from the beginning." The team built systems that "can resume from where the agent was when the errors occurred" (Anthropic, How we built our multi-agent research system).

For a business agent, save the run state after every tool call, retry only idempotent actions automatically, and route repeated failures to a person with the run history attached. Mark half-finished tasks clearly so they are not mistaken for completed ones.

Evaluation harness and observability

Evaluation is the test suite for an agent. Anthropic's guide says that "good evaluations are essential for building reliable AI applications, and agents are no different" (Anthropic, How we built our multi-agent research system). For each agent, build a fixed set of tasks with a checkable outcome, such as "the refund request exists and the amount matches the order." Include adversarial cases: a ticket that contains an instruction to ignore the rules, a tool that returns an error, and a request that should be refused.

Tracing is the other half. The OpenAI Agents SDK records agent runs, model generations, tool calls and guardrail events, and tracing is on by default (OpenAI Agents SDK, Tracing). The same documentation warns that "certain spans may capture potentially sensitive data," and that capture of inputs and outputs is on unless the team changes the setting. Decide before launch what a trace may contain, and restrict access to traces that hold personal data.

Whatever tooling you use, log the decision path, the tool calls, the approvals and the stop reason for each run.

Cost controls

Agents cost more to run than single model calls. Anthropic's multi-agent write-up reports that "agents typically use about 4× more tokens than chat interactions," and that "multi-agent systems use about 15× more tokens than chats" (Anthropic, How we built our multi-agent research system). Those figures come from Anthropic's own systems, so treat them as a reference point rather than a forecast for your workload.

OWASP's entry on unbounded consumption describes the risk of "excessive and uncontrolled inferences," including "Denial of Wallet attacks on pay-per-use pricing." Its prevention list includes "rate limiting and user quotas," timeouts, limits on queued and total actions, and monitoring of unusual consumption (OWASP, LLM10 Unbounded Consumption).

The table below shows the controls to set. The numbers are illustrative assumptions for planning, not measured values.

Set the budget from the value of the task. Anthropic's write-up makes the same point for multi-agent systems, which need a task where "the value of the task is high enough to pay for the increased performance."

Frequently asked questions

What is the difference between a workflow and an AI agent?

A workflow follows predefined code paths. An agent decides its own next step and tool use. Anthropic's guide suggests agents for open-ended tasks where the number of steps cannot be predicted (Anthropic, Building effective agents).

Do production AI agents always need a human approval step?

Not for every action. Read-only actions and low-risk reversible drafts can run without approval. Irreversible or high-value actions, such as payments, deletions and messages to customers, should wait for a named person. OWASP recommends human approval for high-impact actions (OWASP, LLM06 Excessive Agency).

How many tools should an agent have?

Give the agent the fewest tools that let it complete its task, since each one widens what a compromised run can reach. Anthropic's guide recommends a distinct purpose and a clear description for each tool (Anthropic, Writing effective tools for agents).

How do you control the cost of an AI agent?

Cap the number of steps and the run time, set a budget per workflow, apply quotas per user, and monitor spend for anomalies. OWASP's guidance on unbounded consumption covers these controls (OWASP, LLM10 Unbounded Consumption). Set the limits from the business value of each task.

Next step

If your team is scoping a first production agent, see the AI agents consultancy page for Kenya on Pawanax. Start with one bounded process, a short list of tool permissions and a named approver for exceptions.

Sources

• Anthropic, Building effective agents
• Anthropic, Writing effective tools for agents
• Anthropic, Effective context engineering for AI agents
• Anthropic, How we built our multi-agent research system
• OWASP, LLM01 Prompt Injection
• OWASP, LLM06 Excessive Agency
• OWASP, LLM10 Unbounded Consumption
• OpenAI Agents SDK, Guardrails
• OpenAI Agents SDK, Tracing
• OpenAI Agents SDK, Human-in-the-loop

Continue exploring

WhatsApp AI Agents in Kenya: Architecture, Costs and M-Pesa. Why Most AI Pilots Fail Before Production: Evaluation Framework.

Cover: Photo: daudi_mukiibi, via Wikimedia Commons ("Software developer at work 01", 2017), licensed under CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/). Illustrative image; not a photograph of a Pawanax system or client. Credit: daudi_mukiibi (Wikimedia Commons user; uploader's own work). CC BY-SA 4.0. Original source.