The AI agent deployment checklist: 34 gates to clear before, during, and after launch

Most AI agent deployments do not fail because the model is bad. They fail because nobody wrote down what “ready” meant before the agent got access to production credentials.

Gartner predicts that more than 40% of agentic AI projects will be canceled by the end of 2027, and the three reasons it names — escalating costs, unclear business value, and inadequate risk controls — are all scoping and governance problems. None of them are model problems. All three are catchable with a checklist applied before the agent ships.

This is that checklist: 34 items grouped into four gates. A gate is not a phase you pass through on a calendar. It is a point where you either satisfy every item or you do not advance. Treat the exit criteria as binding and the sequence stops being a formality.

The checklist

What does “AI agent deployment” actually mean?

AI agent deployment is the process of moving an autonomous or semi-autonomous LLM-driven system into production, where it acts on real data with real permissions and real consequences. It is distinct from deploying a model and distinct from deploying a chatbot.

The distinction matters because it changes what you are shipping:

• A model deployment ships inference. The output is text or a prediction. A wrong answer is a wrong answer.
• A chatbot deployment ships a conversation. The output reaches a human who evaluates it before acting.
• An agent deployment ships action. The system plans, calls tools, writes to systems of record, and takes further steps based on what came back. A wrong answer becomes a wrong transaction.

That last category is what this checklist covers. If your system reasons across multiple steps, holds state between them, and can invoke tools without a human approving each call, it is an agent, and the failure surface is operational rather than conversational. See what an AI agent is and agentic AI for the underlying definitions.

Why doesn’t a standard MLOps checklist cover this?

Classic MLOps assumes a model that takes an input and returns an output. Agents break four of its load-bearing assumptions.

Non-determinism across steps. A traditional model is evaluated on input-output pairs. An agent’s behavior depends on the path it took — which tool it called third, what that tool returned, what it remembered from turn six. Single-call evaluation misses most of what can go wrong.

Compounding reliability. Per-step accuracy multiplies. An agent that is 95% reliable per step is roughly 60% reliable across a ten-step task. The arithmetic is why checklist items 21 and 24 test full sessions rather than individual calls.

Tool access is a privilege surface. A model has no permissions. An agent has credentials, and those credentials are the actual blast radius. This is why OWASP’s Top 10 for LLM Applications (2025) lists LLM06: Excessive Agency as a distinct risk (see our breakdown of the OWASP Top 10 for LLM Applications) — the harm is not what the model says, it is what it is allowed to do.

Untrusted input reaches the control path. With an agent, retrieved documents, tool outputs, and other agents’ messages all enter the same context that carries instructions. That is LLM01: Prompt Injection, and in an agentic system it escalates from a content problem to a privilege-escalation problem. See how prompt injection attacks work.

The four gates below are ordered so that each one closes off a class of failure before the next gate can expose it.

Gate 1 — Design and scope (items 1–8)

Nothing in this gate requires code. Everything in it is a decision that becomes expensive to change once the agent exists.

1. Write the task boundary down. One paragraph: what the agent attempts, and explicitly what it does not. Unbounded scope is the origin of most of the cost and risk problems that show up later. “Handle support tickets” is not a boundary. “Draft responses to billing questions on accounts under $500 in ARR, and escalate everything else” is.

2. Choose an autonomy level explicitly. Pick one and name it: suggest (proposes, human executes), act-with-approval (proposes an action, human confirms), act-and-notify (acts, tells a human after), or act-autonomously (acts, logs only). Teams that skip this decision default to the most permissive level by accident, usually during a demo.

3. Enumerate the human-in-the-loop triggers. “Escalate when unsure” is not implementable. Implementable triggers are specific: confidence below a threshold, a monetary value above a limit, a customer flagged as sensitive, a tool call that touches production data, N failed attempts at the same subtask, or any request outside the item 1 boundary. The EU AI Act’s Article 14 requires that high-risk systems be designed so a person can effectively oversee them — if you are in scope, this item is a legal requirement, not a design preference.

4. Make success criteria measurable. Task success rate, p95 latency, and cost per completed task, each with a number. These become the promotion criteria in item 26 and the drift baseline in item 30.

5. Set a failure budget. Every agent will be wrong sometimes. Decide in advance what rate is acceptable and what happens when it is breached — whether that is automatic rollback, autonomy reduction, or a human review queue. Without a budget, every individual failure becomes a debate.

6. Pin the model. Name a specific model and version in configuration. Provider default aliases move underneath you, and an agent whose model silently changed is an agent whose evals no longer describe it. Pinning is also what makes item 34’s “rerun evals on model change” a detectable event rather than a surprise.

7. Define cost ceilings. Three limits: per session, per user, and aggregate per day. Enforce them as hard cutoffs, not alerts. Agents fail expensively in loops — a retry cycle that would cost a rounding error once costs real money ten thousand times. OWASP tracks this as LLM10: Unbounded Consumption.

8. Classify the data. List every source the agent can read or write and label it by sensitivity. This list determines the permission scoping in item 9 and the retention decision in item 32, and it is the artifact an auditor asks for first.

Gate 1 exit criterion: all eight decisions exist in writing, and someone who was not in the room can read them and predict what the agent will do.

Gate 2 — Build and harden (items 9–17)

This gate is where the agent acquires permissions. It is the highest- leverage gate for reducing blast radius, and the one most often deferred until after launch.

9. Scope every tool to least privilege. Each integration gets the narrowest permission set that still works. Read-only where reads suffice. Scoped to a single project, table, or repository rather than an organization. The test is simple: if the agent were fully compromised tomorrow, what could it reach? That answer is your actual risk exposure.

10. Give the agent its own identity. Not a shared service account, not a human’s borrowed credentials. Per-agent identity is what makes attribution possible in the audit log and what makes revocation surgical rather than disruptive. See AI access control best practices.

11. Externalize secrets. No API keys in prompts, in code, or in committed configuration. Prompts get logged, and logged prompts with credentials in them are a credential leak with extra steps.

12. Defend against prompt injection. Isolate untrusted content — retrieved documents, tool returns, user files, inter-agent messages — from the instruction context. Treat anything the agent did not author as data. Then test the defense adversarially in item 20; an untested injection defense is a hypothesis.

13. Allowlist the tools. The agent should not be able to discover and invoke tools outside a declared set. Dynamic tool discovery at runtime is convenient and it is also how an agent ends up calling something nobody reviewed. This applies with particular force to Model Context Protocol servers — see shadow MCP.

14. Bound memory scope and retention. Persistent memory needs a time-to-live, a scope (per user, per session, per tenant), and an answer to what happens if something false gets written into it. Poisoned memory is durable in a way a poisoned single prompt is not.

15. Version-control prompts and tool definitions. A prompt is production configuration. Changing it changes behavior as surely as changing code, and it should be as reviewable — a diff, an author, a revert path. Teams that edit prompts in a console lose the ability to answer “what changed?” during an incident.

16. Sandbox or gate destructive actions. Writes, deletes, payments, outbound messages, and schema changes get either a dry-run mode, a confirmation step, or both. This is the concrete implementation of the autonomy level chosen in item 2.

17. Build and execute a rollback path. Not “we could roll back” — someone has actually reverted the agent to a previous version in staging and confirmed it worked. Rollback for an agent means reverting the model version, the prompt, the tool definitions, and any memory written since the change. Discovering the memory dimension during an incident is a bad time to discover it.

Gate 2 exit criterion: you can state the agent’s maximum blast radius in one sentence, and you have tested the mechanism that shrinks it.

Gate 3 — Evaluate and pilot (items 18–26)

Evaluation is where the checklist stops being paperwork and starts producing numbers. See AI agent evaluations for the mechanics; this gate covers what has to be true before rollout.

18. Build a held-out eval set. Cases the agent was not developed against, drawn from the real task distribution. Evaluating on the examples you built against measures memorization, not capability.

19. Include failure cases. Ambiguous requests, adversarial phrasing, out-of-scope tasks, missing data, and tool timeouts. An eval set of happy paths certifies an agent that has never been asked a hard question.

20. Run adversarial testing. A deliberate red-team pass against prompt injection, tool misuse, and privilege escalation, with findings tracked to closure rather than noted. See AI red teaming. Item 12 built the defense; this item establishes whether it holds.

21. Test multi-turn behavior. Evaluate whole sessions, not individual calls, because the compounding-reliability arithmetic above means step-level metrics systematically overstate task-level success.

22. Run in shadow mode. Real production traffic, the agent makes real decisions, none of them take effect, and humans review the divergence. This is the cheapest high-signal test available and the one most often skipped for schedule reasons.

23. Stage the rollout. Internal users first, then a small percentage of real traffic, then expansion. The percentage matters less than the existence of a step where a problem affects a bounded population.

24. Measure latency and cost under load. Observed p95 latency and per-task cost, compared against the item 4 targets. Agent costs are sensitive to task complexity in ways that single-request testing will not reveal — the expensive sessions are the long ones.

25. Drill the escalation path. A real human received a real escalation, knew what to do with it, and closed it. Escalation paths that exist only in a design document route to nobody.

26. Write promotion criteria before collecting data. Define “ready to scale” in advance. Criteria set after the pilot data arrives get fitted to the data, which is a way of guaranteeing every pilot passes.

Gate 3 exit criterion: you have numbers for success rate, latency, and cost from real traffic, and they meet the targets set in Gate 1.

Gate 4 — Run and govern (items 27–34)

Deployment is the start of the operational obligation, not the end of the project. See how to secure the AI agent lifecycle for the fuller lifecycle view.

27. Trace every step. Prompts, tool calls with arguments, tool returns, intermediate reasoning, and final actions. Partial tracing produces incidents you can detect but not explain.

28. Structure the logs. Queryable fields, not free text. The operational test is whether you can answer “what did the agent do at 14:03 and why” in minutes. The compliance test, if you are in EU AI Act scope, is Article 12, which requires high-risk systems to support automatic event logging over the system’s lifetime.

29. Alert on guardrail violations. A blocked action is signal. If blocks accumulate silently, you lose the early warning that the agent is repeatedly attempting something it should not — which is usually either a scope problem or an attack. See AI agent guardrails.

30. Detect drift. Monitor output quality and input distribution against the Gate 1 baseline. Agent degradation is gradual and rarely announces itself; upstream model updates, changing user behavior, and tool changes all move the baseline. See how to detect model drift.

31. Test the kill switch. One action disables the agent, and someone has actually pulled it in a drill. An untested kill switch is a plan, and plans fail at 3am. This capability typically lives in an agent control plane rather than in the agent itself.

32. Set log retention deliberately. Choose a retention period against your actual obligations rather than accepting a platform default. Retention that is too short destroys the evidence for an investigation; retention that is too long turns a trace archive into a liability.

33. Review agent access on a schedule. Permissions and tool grants get re-certified periodically, and grants for retired agents get revoked. Permanent grants accumulate into AI sprawl — the organizational condition where nobody can enumerate which agents hold which credentials.

34. Set a re-evaluation cadence. Rerun the eval suite on every model change, every prompt change, and on a fixed interval regardless. Items 6 and 15 exist so that the first two triggers are detectable events.

Gate 4 exit criterion: you can detect a degradation, explain it, and stop it — and you have demonstrated all three.

How does this checklist map to NIST, OWASP, and the EU AI Act?

Most deployment checklists are written from experience. Mapping them to published frameworks is what makes them defensible in a review.

The NIST AI Risk Management Framework (AI RMF 1.0) organizes risk work into four functions — Govern, Map, Measure, and Manage — and its companion Generative AI Profile (NIST AI 600-1) extends them to generative systems. The gate structure above is deliberately ordered to follow that progression: you govern and map before you measure, and you manage continuously afterward. See our guide to the NIST AI Risk Management Framework.

For agent-specific threats beyond the LLM Top 10, OWASP’s Agentic Security Initiative publishes a dedicated threat taxonomy and a Top 10 for Agentic Applications covering risks that only appear once systems plan, remember, and delegate — goal hijacking, tool misuse, memory poisoning, and rogue agents among them.

EU AI Act obligations apply only if your system falls into the Act’s high-risk category, which depends on use case rather than architecture. Where they do apply, Articles 12 and 14 map almost directly onto checklist items 27–28 and item 3. See the EU AI Act explained and what AI compliance requires.

The five failure modes this checklist is designed to catch

Every item above exists because of a specific way agent deployments go wrong.

Cost blowups. An agent enters a retry loop, or a task turns out to be ten times longer than the sample suggested. Caught by items 7 and 24.

Silent degradation. Quality declines slowly enough that nobody notices until a customer complains. Caught by items 4, 30, and 34.

Over-permissioned tools. The agent holds credentials far beyond what its task requires, so an ordinary bug or an injection becomes a serious incident. Caught by items 9, 10, and 13.

No rollback. A bad change ships and the team discovers there is no tested path back, particularly for memory written since the change. Caught by items 15 and 17.

No audit trail. Something goes wrong and nobody can reconstruct what the agent did or why. Caught by items 27, 28, and 32.

Each of these is a governance failure rather than a model failure — which is the same conclusion Gartner reaches about the projects it expects to be canceled.

Frequently asked questions

How long does it take to deploy an AI agent to production? The gates, not the calendar, determine the timeline. Gate 1 is typically days of decision-making. Gate 2 depends on how much identity and permission infrastructure already exists. Gate 3 is the longest, because shadow mode and staged rollout need enough real traffic to produce meaningful numbers — usually weeks rather than days. Teams that report fast deployments have usually skipped Gate 3, and typically encounter its findings in production instead.

What is the difference between an AI agent pilot and a production deployment? A pilot is bounded in population and reversible by default; production is neither. The practical difference is item 26: a pilot becomes a deployment when it meets promotion criteria that were written before the pilot started. Without that, a pilot becomes production by drift — more users, more permissions, no decision point.

How do you monitor an AI agent once it is deployed? Four signal types: traces of every step (item 27), quality and drift metrics against a baseline (item 30), guardrail violation alerts (item 29), and cost and latency against the Gate 1 targets. Traditional uptime monitoring is necessary and badly insufficient — an agent can be fully available and completely wrong.

What does AI agent deployment cost? The variable cost is inference, which scales with task complexity and retry behavior rather than with user count, which is why item 7’s per-session ceiling matters more than a monthly budget. The larger cost is usually the Gate 2 and Gate 4 work — identity, tracing, evaluation infrastructure — which is one-time per organization and reusable across subsequent agents.

Do you need a separate checklist for each agent? Run every item for each agent, but most Gate 2 and Gate 4 infrastructure is shared. The per-agent work concentrates in Gate 1 (scope and autonomy are agent-specific by definition) and Gate 3 (eval sets do not transfer between tasks).

When should an AI agent escalate to a human? On the triggers enumerated in item 3, which should be specific and testable: confidence thresholds, monetary limits, sensitive accounts, production data access, repeated subtask failure, and anything outside the stated task boundary. The escalation path also has to be drilled (item 25) — a trigger that fires into an unmonitored queue is not oversight.

What is the minimum viable version of this checklist for a small team? Ten items, if you can only do ten: 1 (task boundary), 2 (autonomy level), 6 (pin the model), 7 (cost ceilings), 9 (least-privilege tools), 12 (injection defense), 17 (tested rollback), 22 (shadow mode), 27 (tracing), and 31 (kill switch). That set covers the five failure modes above at minimum depth. Everything else deepens coverage rather than adding a new category.

Where to take this next

Run the checklist against one agent you already have in production. The items it fails are your actual backlog, and the ordering above is a reasonable priority order — Gate 2 gaps are more urgent than Gate 4 gaps because they determine blast radius.

For the security controls in depth, including OWASP agentic threat mapping and the identity, tool, memory, and audit controls in Gate 2, see the agentic AI security checklist. For the surrounding concepts: guardrails, evaluations, the agent control plane, and AI runtime security. For the governance frame around all of it, what AI governance is and why AI governance fails. For the agentic-specific layer on top of that, see how companies govern agentic AI.