AI agents in production: what breaks and how to fix it

Short answer: AI agents usually break in production for ordinary engineering reasons: tools fail or time out, the agent loops, costs run away, it has more access than it needs, and nobody can see what it did. The fix is to bound every run (step, cost and time limits), give it least-privilege tools, trace every step, require human approval for risky actions, and test it against a fixed set of real cases before and after every change.

What is an AI agent, in plain terms?

An AI agent is a language model that works in a loop. It reads a goal, decides on a next step, calls a tool (search a database, read a file, open a ticket), looks at the result, and decides again. It stops when it thinks the job is done.

Anthropic's engineering guide draws a useful line here. A workflow runs a model through steps that you wrote in code. An agent lets the model choose its own steps and tools. Agents are more flexible, but that freedom is exactly what makes them harder to run safely.

The same guide recommends starting with the simplest thing that works and adding autonomy only when it pays for itself. Many "agent" problems disappear when part of the job is turned back into a fixed workflow.

Why do agents that work in a demo fail in production?

A demo has clean inputs, a short happy path and a person watching. Production has none of these. APIs are slow, records are half-filled, users ask vague questions, and runs happen at 3 a.m. with nobody looking.

Agents also compound errors. If each step is right most of the time, a ten-step run is wrong far more often than a single step. Anthropic notes that autonomy brings higher costs and the risk of errors that build on each other.

This shows up in the market. Reuters reported in June 2025 that Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, citing escalating costs and unclear business value. The rest of this article is about avoiding that outcome.

What actually breaks? The seven common failure modes

The sections below take each one in turn.

Why do tool calls fail, and how do you handle it?

Tools are where an agent touches the real world, so this is where most failures start. A tool can time out, return an error, return far too much data, or accept arguments the model got slightly wrong (a date in the wrong format, an ID from a different system).

The most damaging case is a silent one. If a tool fails and the agent is not told clearly, the model may fill the gap with a plausible guess and carry on.

Practical fixes:

1. Set a timeout on every tool call. The Model Context Protocol (MCP) specification, a common standard for connecting tools to AI apps, says clients should implement timeouts for tool calls. That applies whether or not you use MCP.
2. Return errors the model can act on. The MCP spec separates protocol errors from tool execution errors and recommends passing execution errors back to the model so it can correct itself. "Invalid date: must be in the future" is useful. "Error 500" is not.
3. Make tools hard to misuse. Anthropic's guide advises changing tool arguments "so that it is harder to make mistakes". Use strict input schemas, enums instead of free text, and one clear job per tool.
4. Retry only what is safe to retry. Reading is safe to repeat. Creating a ticket or sending a payment is not, unless the tool accepts an idempotency key so a retry does not create a duplicate.
5. Cap the size of tool output. Return the 20 rows the model needs, not 20,000. Large outputs burn tokens and push important context out of view.

How do you stop an agent from looping?

Loops happen when the agent cannot tell it is finished, or keeps trying a fix that does not work. Typical signs are the same search run again and again, or two tools called alternately with no progress.

Anthropic's guide says agents should include stopping conditions, such as a maximum number of iterations, to keep control. In practice, use several limits together:

• A step limit per run, sized to the task. A ticket-triage agent might need five steps; a research agent might need thirty.
• A time limit so a stuck run ends even if each step is fast.
• Repeat detection. If the agent calls the same tool with the same arguments twice, stop and escalate rather than trying a third time.
• A clear definition of done. Tell the agent what a finished result looks like, and check it in code where you can.

When a limit trips, the run should end in a known state: report what it did, what it did not finish, and hand off to a person.

How do you keep agent costs under control?

Every step in an agent loop is a model call, and each call resends the growing conversation. A run that loops, or pulls large tool outputs into context, can cost many times what you planned.

OWASP lists this as "Unbounded Consumption" in its 2025 Top 10 for LLM applications. It describes applications that allow "excessive and uncontrolled inferences", and recommends rate limiting, user quotas, and timeouts for resource-heavy operations.

For agents, that becomes:

1. A budget per run, in tokens or currency, that stops the run when spent.
2. A budget per user or tenant per day, so one runaway trigger cannot drain the month.
3. Rate limits on triggers, so a burst of events does not start a thousand runs at once.
4. Cost on every trace, so you can see which agents, steps and tools are expensive.
5. Smaller context. Summarise long histories and trim tool outputs before they go back to the model.

What permissions should an AI agent have?

An agent should have the least access that lets it do its job, and no more. OWASP calls the opposite "Excessive Agency" and breaks it into three causes: excessive functionality (tools that can do more than needed), excessive permissions (access beyond what the task requires), and excessive autonomy (high-impact actions with no independent check).

The fixes OWASP recommends map neatly onto agent design:

• Give the agent only the tools it needs. A support agent that reads orders does not need a "run any SQL" tool.
• Prefer narrow tools to open-ended ones. "Create ticket" is safer than "call any API".
• Act in the user's context. The agent should carry the requesting user's rights, so it cannot see data that user cannot see.
• Enforce authorisation in the target system. Do not rely on the model to decide what is allowed. OWASP calls this complete mediation.

Prompt injection makes this urgent. If an agent reads an email, web page or document that contains hidden instructions, it may try to follow them. You cannot fully prevent that, so limit what a fooled agent is able to do.

Why do you need traces for every step?

When an agent gets something wrong, the question is always "which step, and why?" Without a trace, you cannot answer it. Logs of the final answer are not enough, because the mistake is usually three tool calls earlier.

A useful trace records, for each step:

• the model input and output, including which tool it chose and with what arguments;
• the tool's response, error and duration;
• tokens used and cost;
• the trigger and the user or system the run was acting for.

Open standards are emerging for this. OpenTelemetry, the widely used observability project, publishes semantic conventions for generative AI that cover model spans, agent spans and metrics, so agent traces can sit alongside the rest of your telemetry.

Traces also do a second job: they are your audit trail. If an agent changed a record, you should be able to show who asked, what it saw, and what it did.

When should a human approve an agent's action?

Not every step needs a person. If it did, you would not need an agent. The rule of thumb is simple: let the agent read, search and draft on its own, and make it wait for a person before it does something hard to undo.

The MCP specification puts it directly: for trust and safety there should always be a human in the loop with the ability to deny tool invocations, and applications should show confirmation prompts for operations. OWASP also lists human approval for high-impact actions as a core mitigation.

Good candidates for approval:

Make approval easy. Show the approver exactly what will run, with the inputs, in plain language. An approval that people click through without reading is not a control.

How do you evaluate an agent before and after launch?

Agents change whenever you change a prompt, a tool, a model or the data behind it. Without a test set, you find out about regressions from customers.

A practical evaluation loop:

1. Collect 30 to 100 real cases, including hard and ambiguous ones, with what a good outcome looks like.
2. Score the outcome, not just the text. Did the right ticket get filed with the right fields? Did it stop when it should have?
3. Score the path too. Count steps, cost, tool errors and approval requests per case.
4. Run the set on every change and block releases that get worse.
5. Feed production failures back in. Every bad trace becomes a new test case.

Anthropic's guide also recommends extensive testing in sandboxed environments before deployment. Point agents at test systems first, not live ones.

A short production checklist

1. Write down the agent's job, its tools, and what "done" means.
2. Give each tool a strict schema, a timeout and clear error messages.
3. Set step, time and cost limits per run, and budgets per tenant.
4. Scope access to the requesting user and enforce it in the target systems.
5. Require human approval for external, financial and destructive actions.
6. Trace every model and tool call, with cost and timing.
7. Build a test set from real cases and run it on every change.
8. Review failed traces weekly and turn them into new tests.

How CloudOEngine helps

CloudOEngine is one platform and one console for connecting your data, running pipelines and running AI, including an AI runtime with an LLM gateway, agents with tool access, and RAG. Everything sits under one permission model and one audit trail, so an agent's access follows the same roles as your people.

Agents in CloudOEngine run real tools, such as filing a ticket, and every step and tool call is traced and inspectable. When a run goes wrong, you can open it and see each decision, input and result rather than guessing.

Autopilot runs triggered agents with approval rules. When an action needs sign-off, a person approves it in the app or through an email approve link before it runs. That keeps reading and drafting automatic while risky actions wait for a human.

Governance is included, not sold as an extra: tenant isolation, role-based access, audit logs and per-tenant encrypted secrets, so tool credentials stay scoped to the tenant that owns them.

CloudOEngine is built in Bengaluru, India.

Book a demo or see the AI runtime.

Frequently asked questions

Why do AI agents fail in production when they worked in the demo?

Demos use clean inputs, friendly tools and a person watching. Production adds slow or failing APIs, messy data, unclear requests and nobody watching each step, so small errors compound across a multi-step run.

How do you stop an AI agent from looping forever?

Give every run hard stopping conditions: a maximum number of steps, a maximum spend, and a wall-clock timeout. Also detect repeats, such as the same tool called with the same arguments, and stop or escalate when it happens.

What permissions should an AI agent have?

Only the tools and access its task needs, ideally acting with the requesting user's own rights. Authorisation should be enforced by the systems the agent calls, not left to the model to decide.

When should a human approve an AI agent's action?

Before any action that is hard to undo, costs money, changes records or contacts people outside the company. Reading and drafting can usually run unattended; sending, paying and deleting should wait for a person.

What should an AI agent trace record?

Every model call and tool call in the run: inputs, outputs, errors, timing, token use and who or what triggered it. That lets you replay a bad run and see exactly which step went wrong.

Sources

1. Building effective agents (Anthropic)
2. LLM06:2025 Excessive Agency (OWASP GenAI Security Project)
3. LLM10:2025 Unbounded Consumption (OWASP GenAI Security Project)
4. OWASP Top 10 for LLM Applications 2025
5. Over 40% of agentic AI projects will be scrapped by 2027, Gartner says (Reuters via The Star)
6. Model Context Protocol specification: Tools
7. OpenTelemetry semantic conventions for generative AI

#ai agents #agent observability #human in the loop #llm security #evaluation