Many enterprise agent pilots stall because nobody designed how the agent plans, acts, remembers and gets stopped, not because the model is too weak. Agentic AI architecture is the set of layers (perception, reasoning, memory, tools, orchestration and governance) that lets an AI system pursue a goal over several steps, call tools and check its own results. Unlike a chatbot or a single call to a large language model (LLM), it acts instead of only answering.
What are agentic AI systems? They are systems in which one or more AI agents plan and carry out multi-step tasks with limited human input. We explain the difference between AI agents and agentic AI in a separate guide. This guide covers the six layers, one request traced end to end, five patterns, a decision table, an enterprise checklist and the route to production.
What is agentic AI architecture?
Agentic AI architecture is the blueprint for an AI system that receives a goal, plans the steps, uses tools and data, remembers context, checks its own results and escalates to people when needed. All of this runs inside policy and monitoring controls. At its core is a repeating sense-plan-act loop: observe, decide, act, then observe the outcome.
A non-agentic system follows a path a person defined. That might be a single prompt-and-response call or a fixed rule-based workflow. An agent is an AI component that decides its own next action toward a goal, and an agentic architecture lets the model choose part of the path. Anthropic draws the same line in its research: workflows run LLMs and tools through predefined code paths, while agents let LLMs dynamically direct their own processes and tool use (December 2024).
The six layers of an agentic AI architecture
Published models use anywhere from three to eight layers. We use six because it is the smallest set in which each layer has one job and one owner on an enterprise team. The agentic AI architecture diagram below shows how the six layers connect.
![]()
`` ┌─────────────── 5. ORCHESTRATION (sequence · hand-offs · retries · state) ───────────────┐ │ │ │ [1. Inputs / perception] ──► [2. Reasoning / planning] ──► [4. Tools / integrations] │ │ ▲ │ ▲ │ │ │ │ │ └────── result ◄─────┘ │ │ ▼ ▼ │ │ [3. Memory: short-term ⇄ long-term] │ │ │ └──────────────────────────────────────────────────────────────────────────────────────────┘ ═══ 6. GUARDRAILS, GOVERNANCE & OBSERVABILITY: policy checks · audit log · human escalation ═══ `` A vendor-neutral agentic AI reference architecture.
The table below sets out the agentic AI architecture components, layer by layer. It also shows where each layer usually breaks first.
|
Layer |
Its job |
Typical components |
Common failure |
|
Inputs / perception |
Turns requests, documents, events and data into structured context |
Channels, document parsing, speech, event triggers |
Garbage in: poor optical character recognition (OCR), missing fields |
|
Reasoning / planning |
Interprets the goal, breaks it into steps, decides the next action |
LLM, prompts, planner, reflection step |
Plans that loop or drift away from the goal |
|
Memory |
Keeps context within a task and across tasks |
Context window, session state, vector store, knowledge graph or database |
Stale context, or context leaking between users |
|
Tools / integrations |
Acts on business systems |
APIs, function calls, connectors, MCP (Model Context Protocol) servers, robotic process automation (RPA) |
Wrong tool, wrong parameters or over-broad permissions |
|
Orchestration |
Controls the sequence, hand-offs, retries and state |
Workflow engine or agent framework, router, queue |
Unbounded loops; state lost on failure |
|
Guardrails / governance / observability |
Enforces policy, logs every step, routes cases to humans |
Policy engine, identity and access, content filters, tracing, evaluation |
No audit trail; failures nobody notices |
If you have seen other layer counts, here is how they map to these six:
- Three-layer "sense-plan-act" models fold memory into planning and fold tools into acting.
- Four-layer models (perception, reasoning, memory, action) leave out orchestration and governance as separate layers.
- Eight-layer models split governance into security, observability and policy.
This post covers how agents are built. For how agents are classified, see the main types of AI agents.
Improve Your Business with Agentic AI!
How the layers work together: one request, end to end
Diagrams hide the order of events. Tracing one ordinary request shows where each layer earns its place and where failures start.
Walkthrough: an invoice that doesn't match its purchase order
This is an illustrative example of an accounts-payable (AP) agent handling an exception:
1. Goal received: "Resolve invoice exception for supplier X." [inputs]
2. Context assembled: the invoice, the matching purchase order (PO) and the supplier's history. [memory and inputs]
3. Plan made: compare the line items, check the price-tolerance policy, propose a fix. [reasoning]
4. Tool called: an ERP (enterprise resource planning) lookup and a goods-receipt check. [tools]
5. Result checked: the variance is tested against policy, for example the price-variance tolerance. [reasoning and guardrails]
6. Memory updated: case notes are saved and the supplier pattern is recorded. [memory]
7. Escalation: the amount is over the approval limit, so the case goes to an AP clerk with a summary. [governance]
Orchestration held the state across all seven steps. It also decided what happened when the ERP call timed out: retry once, then park the case.
Short-term vs long-term memory
Short-term memory is the working context of the current task: the conversation so far and the intermediate results. It sits in the context window or session state. When the task ends, it is either discarded or summarised.
Long-term memory persists across tasks in a vector store, database or knowledge graph. It holds things like policies, past cases and user preferences. The design question here is a governance question: what may be stored, for whom, and for how long.
Why orchestration matters more than the model
Swapping one model for another is now a routine change. In practice, production failures more often come from control flow:
- Loops that never end;
- State lost after an error;
- Retries that repeat an action, such as paying the same invoice twice;
- Hand-offs that drop context.
That is why it pays to understand how agentic workflows are orchestrated before you pick a model.
Five agentic AI patterns, and when each fits
![]()
Agentic AI architecture patterns describe how agents and steps are arranged around a goal. Most enterprise systems combine two of them, for example a supervisor with a human-in-the-loop checkpoint.
Cloud providers' own guidance groups patterns in similar ways:
- According to Microsoft's AI agent orchestration patterns guide, the Azure Architecture Center documents five orchestration patterns: sequential, concurrent, group chat, handoff and magentic.
- According to Google Cloud's guide to choosing a design pattern for agentic AI systems, the options include single-agent, sequential, parallel and hierarchical designs, among others.
The five patterns below are a simplified, vendor-neutral grouping of those options.
Single agent with tools
- When it fits: one goal, one permission scope, and fewer than about a dozen well-described tools. That number is our rule of thumb, not a benchmark.
- Trade-offs: the simplest pattern to debug and audit, but limited by context size.
- Main risk: as the tool count grows, the agent starts choosing the wrong tool.
Sequential (pipeline)
- When it fits: work with fixed stages, such as extract → validate → draft → review.
- Trade-offs: predictable, and each stage can be tested on its own. Latency adds up with every stage.
- Main risk: an early error flows downstream unless each stage validates its input.
Hierarchical (supervisor and workers)
- When it fits: a supervisor agent can split the goal and route subtasks to specialist agents.
- Trade-offs: tools and permissions are cleanly separated, but the supervisor becomes a bottleneck and a single point of failure.
- Main risk: the extra model calls add cost and latency.
Collaborative multi-agent (peers)
- When it fits: open-ended analysis, where agents debate or critique each other in a group-chat style. This is the multi agent architecture most people picture.
- Trade-offs: it can cover a problem more thoroughly, but it is the hardest pattern to bound and audit.
- Main risk: conversations are non-deterministic, token costs can run away, and errors get amplified when agents agree on a wrong answer.
Human-in-the-loop
- What it is: a checkpoint layered onto any of the patterns above. The agent proposes an action, and a person approves it before anything irreversible or high-value happens.
- Trade-offs: the lowest-risk option, but it adds human latency and staffing cost.
- Main risk: reviewers start rubber-stamping when approval volumes are high.
For examples of these patterns in use, see these real agentic AI use cases by industry.
Build governance, observability and escalation into the architecture
An agent takes actions, so every action needs an identity, a permission boundary and a record. If you add these after a pilot, you usually have to re-plumb every tool call. Three controls are the minimum:
- Audit trail: every prompt, plan, tool call, result and approval, traceable per request.
- Runtime policy: what the agent may call and with what data, plus spend and rate limits. These must be enforced at execution time, not only written into the prompt.
- Escalation rules: named thresholds (amount, confidence, data sensitivity) that route a case to a person, with a summary attached.
The governance layer can map to a neutral framework. The NIST AI Risk Management Framework 1.0 (January 2023) organises AI risk work into four functions: Govern, Map, Measure and Manage. Teams that want one place to manage policies, approvals and audit evidence can look at Centric Governance Central.
Knowing the layers and patterns is the easy part. Choosing one is where most projects stall.
Single agent vs multi-agent: which should you start with?
Start with a single agent. Split into multiple agents only when you can name the limit you have hit:
- The context window is overloaded;
- The tool sets conflict or confuse the model;
- The work can genuinely run in parallel;
- Steps need separate permission or data scopes.
This follows Anthropic's advice in the guidance cited above: find the simplest solution that works, and add complexity only when it clearly improves outcomes.
The table compares single agent vs multi-agent designs on the factors that matter in production.
|
Factor |
Single agent |
Multi-agent |
|
Build complexity |
Low |
High (hand-offs, shared state) |
|
Latency |
Lower |
Higher (more model calls in series) |
|
Run cost |
Lower |
Higher; grows with agent-to-agent turns |
|
Debuggability |
One trace |
Distributed traces needed |
|
Auditability |
Straightforward |
Needs a separate identity and log for each agent |
|
Typical failure |
Tool confusion, context overflow |
Lost hand-offs, agents reinforcing each other's errors |
Which pattern fits which use case?
Reflection (the agent critiques its own output), planning (it breaks down the goal) and tool use are agentic design patterns that sit inside any architecture. The table below is about something different: how the agents themselves are arranged.
|
Pattern |
Task complexity |
Risk level suited |
Latency tolerance |
Auditability |
Main risk |
Example fit |
|
Single agent with tools |
Low to medium |
Low to medium |
Low (needs fast responses) |
High |
Tool confusion at scale |
IT ticket triage |
|
Sequential |
Medium, fixed steps |
Medium to high |
Medium |
High |
Error propagation |
Document intake |
|
Hierarchical |
High, separable subtasks |
Medium |
Medium to high |
Medium to high |
Supervisor bottleneck, cost |
Research across several systems |
|
Collaborative multi-agent |
High, open-ended |
Low |
High |
Low |
Unbounded cost, consensus errors |
Analysis drafts reviewed by people |
|
Human-in-the-loop (overlay) |
Any |
High or regulated |
High |
Highest |
Approval fatigue |
Payments, HR, customer commitments |
The pros and cons of each pattern, in one line:
- Single agent: fastest to ship, but struggles as tools multiply.
- Sequential: easiest to test, but the slowest to change.
- Hierarchical: clean separation of duties, but more calls per task.
- Collaborative: the broadest reasoning, but the hardest to govern.
- Human-in-the-loop: the safest, but only as good as the reviewers' attention.
How to choose an architecture for your enterprise
In agentic AI enterprise architecture, your constraints decide the right pattern, not the newest framework. You can classify the types of agentic AI architecture by agent type or by single vs multi-agent, and both are useful. The checklist below is what decides the build.
An enterprise checklist
- Task predictability: are the steps known in advance? If yes, prefer a sequential workflow over an autonomous agent.
- Risk and compliance exposure: what is the worst irreversible action the agent could take? The answer sets where human-in-the-loop goes.
- Latency and cost budget: what response time and what model-call cost per task can the process absorb?
- Audit trail need: must auditors or regulators be able to reconstruct each decision?
- Integration surface: how many systems are involved, and do they have stable APIs?
- Team skills: can your team run evaluation, tracing, and prompt and version management?
- Data residency: where may prompts, memory and logs be stored and processed?
If several answers are "not sure", start with an AI readiness assessment before choosing a pattern.
Build, buy or combine?
You have three options:
- Custom build on an open-source agent framework: maximum control and portability. Your team owns maintenance, upgrades and evaluation.
- Managed agent platform from a cloud or SaaS provider: the fastest time to value. The cost is lock-in risk and less control over orchestration and logs.
- Hybrid: managed models and identity, with your own orchestration and governance. This is usually the pragmatic enterprise choice.
An agentic AI frameworks comparison matters less than one question: does the framework support your governance and observability requirements? For teams going the custom route, we have written a step-by-step guide to building an AI agent.
One example implementation on Azure
Here is how the six layers map to one cloud, Microsoft Azure, as an example. Equivalent services exist on other clouds. Azure AI product names change often, so confirm the current names before you build.
|
Layer |
Example Azure services |
|
Inputs |
Azure AI Document Intelligence, Azure AI Speech |
|
Reasoning |
Azure OpenAI models through Microsoft Foundry |
|
Memory |
Azure AI Search (vector search), Azure Cosmos DB |
|
Tools |
Azure Functions, Azure Logic Apps, Azure API Management |
|
Orchestration |
Foundry Agent Service, Microsoft Agent Framework |
|
Governance and observability |
Azure AI Content Safety, Microsoft Entra ID, Azure Monitor and Application Insights, Microsoft Purview |
Once you have a shortlist, the next step is a small, governed pilot.
From architecture to production
A good architecture on paper earns nothing until one workflow runs reliably in production. This order of work keeps risk and spend contained.
Four steps from shortlist to live workflow
- Use-case selection: pick one workflow with clear volume, a measurable outcome and a worst case you can tolerate. This is the job of AI strategy and use-case discovery.
- Architecture review: test the pattern choice, tool permissions, memory policy and escalation thresholds against the checklist above.
- One-workflow pilot: run it with human-in-the-loop on, tracing on, and exit criteria agreed up front: accuracy, cycle time, escalation rate and cost per task. We cover this stage in our guide to moving enterprise AI from pilot to production.
- Production hardening: load and failure testing, evaluation sets, monitoring and alerting, a rollback plan and a regular review cadence. This is where AI deployment support comes in.
Gartner's 2025 forecast shows why the pilot comes first. It predicts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls. That is a forecast, not a measured result. Still, a bounded pilot with exit criteria answers all three of those reasons.
Build in-house, partner or buy?
Keep the work in-house if you already run ML or LLM systems in production, own the integration APIs, and have evaluation and tracing skills on the team.
Bring in an AI agent development company or consultant when:
- Integration spans several legacy systems;
- Governance and audit design is new to the team;
- You need to move from pilot to production without hiring first.
That can make more sense than trying to hire agentic AI developers before the use case is proven.
What drives the cost
Five factors set the cost of an agent project:
- The number of agents and tools;
- The number of system integrations;
- Data readiness (whether data is clean, accessible and permissioned);
- Governance and audit requirements;
- Model calls per task × tasks per month.
Scope sets the price, which is why a scoped review should come before any quote.
How Centric approaches agent projects
Centric's AI services cover the full path:
- Strategy and use-case discovery;
- Deployment;
- Intelligent automation and AI workflows.
For conversational use cases, Centric also offers an Azure OpenAI chatbot.
Not every team needs end-to-end AI agent development services. If you want agentic AI consulting services limited to the architecture review, that work can stand on its own and feed your in-house build.
Frequently asked questions
These are the questions teams ask most often when they start planning an agent build.
What are the main components of an agentic AI architecture?
Six: inputs and perception, reasoning and planning, memory, tools and integrations, orchestration, and a governance and observability layer that spans all of them.
Should I start with a single agent or multiple agents?
Start with a single agent. Split into several only when a named limit appears: an overloaded context window, conflicting tools, work that can run in parallel, or steps that need separate permission scopes.
How is agentic AI different from an AI agent?
An AI agent is one component that acts toward a goal. Agentic AI describes the wider system and behaviour of agents planning and acting with autonomy. The guide linked in the introduction covers this in depth.
Do I need an orchestration framework?
Not for a single agent with a few tools. You need one once the system has multiple steps, retries, hand-offs, or state that must survive failures.
Why is governance part of the architecture rather than a later add-on?
Agents take actions, so identity, permissions, audit logs and escalation rules must sit on every tool call. Adding them later usually means rebuilding the integrations.
Key takeaways: agentic AI architecture
Agentic AI architecture comes down to six layers and five patterns. Start with a single agent, and add agents only when you can name the limit you have hit. Choose the pattern with the checklist: risk, latency, audit needs, integration surface, team skills and data residency. Treat governance as a layer from day one.
A pilot-first path keeps cost and risk bounded: one workflow, a human in the loop and exit criteria agreed up front. When you are ready to test your design or check your readiness, Centric's AI team can review your architecture with you.
