Human-in-the-loop AI: when an agent should ask before it acts

Short answer: An AI agent should ask a person before it does anything that is hard to undo, moves money, contacts someone outside the company, changes access or deletes data, or falls outside its usual pattern. Reading, searching and drafting can run on their own and be logged. The approval request should show the exact action and its inputs, the approval should cover only those inputs, and a request nobody answers should expire without the action running.

What does human-in-the-loop mean for an AI agent?

An AI agent is a language model that works in a loop: it reads a goal, picks a tool, looks at the result and decides what to do next. Human-in-the-loop means that, at chosen points in that loop, the agent stops and waits for a person to approve, edit or reject its next step.

It helps to separate three levels of human involvement:

In the loop. The agent proposes; a person decides before the action runs.
On the loop. The agent acts; a person is told and can reverse or stop it.
Out of the loop. The agent acts and the result is only logged.

A well-run agent uses all three, depending on the action. Anthropic's engineering guide on agents describes this pattern: agents can "pause for human feedback at checkpoints or when encountering blockers". The design question is which checkpoints, and that is what this article is about.

If you are new to running agents, start with our overview of AI agents in production, which covers limits, traces and testing. This article goes deeper on one control: approval.

Why not let the agent act on its own?

Three reasons.

Errors compound. An agent takes many steps, and each one can go slightly wrong. A wrong guess in step two becomes a wrong record in step five. Anthropic's guide notes that even for coding agents, where results can be tested automatically, "human review remains crucial".

Agents can be tricked. OWASP's 2025 entry on Excessive Agency gives a plain example: a personal assistant with access to a mailbox reads a maliciously written incoming email, and the hidden instructions trick it into searching the inbox for sensitive information and forwarding it to the attacker. One of OWASP's fixes is "requiring the user to manually review and hit 'send' on every mail drafted". You cannot fully stop an agent from being fooled, but you can stop a fooled agent from acting alone.

Autonomy is a risk category in its own right. OWASP splits Excessive Agency into three causes: too much functionality, too many permissions, and too much autonomy. It defines excessive autonomy as an application that "fails to independently verify and approve high-impact actions", and lists "Require user approval" for high-impact actions as a mitigation.

The answer is not to approve everything. It is to approve the right things.

Which actions should need approval?

Score each tool, and where needed each call, on a few plain questions. The more "yes" answers, the more it needs a person.

Question	Why it matters	Example that needs approval
Is it hard or impossible to undo?	A mistake becomes permanent	Deleting records, sending an email
Does it move money or commit the company?	Direct financial loss	Issuing a refund, placing an order
Does it reach people outside the company?	Reputational and legal exposure	Messaging a customer or supplier
Does it change who can access what?	Security impact can spread	Granting a role, sharing a folder
Is it large in scope?	One error affects many records	Updating 5,000 rows instead of one
Is it unusual for this agent?	May signal confusion or manipulation	A first-ever call to a tool, an odd amount

From those answers, put each action into a tier. This is an illustration you can adapt, not a standard:

Tier	What happens	Typical actions
1. Run	Runs and is logged	Read, search, summarise, draft
2. Run and notify	Runs; the owner is told and can reverse it	Update a ticket status, add an internal note
3. Ask first	Waits for one approver	Email a customer, refund under a limit, close an account
4. Ask two people	Waits for two approvers	Large payments, bulk deletes, permission changes
5. Never	The agent does not have the tool at all	Anything it has no business doing

Tier 5 matters most. The safest approval is the one you never need, because the agent was never given that tool. OWASP's first mitigations are to limit the tools an agent can call and what each tool can do.

Thresholds can also move a single tool between tiers. A refund tool might sit in tier 2 for small amounts and tier 3 above a limit you set. That is better than putting every refund in tier 3 and training people to click "approve" without looking.

An approval flow. A risk check sends each proposed action to run, run and notify, or wait for a person. A waiting action runs only as approved, is rejected with a reason, or expires and stops. Every decision is logged.

Can you trust a tool's own "safe" label?

No, not on its own. The Model Context Protocol (MCP), a common standard for connecting tools to AI apps, lets a tool describe itself with hints such as readOnlyHint and destructiveHint. These are useful defaults, and the spec errs on the cautious side: if a tool says nothing, destructiveHint defaults to true and readOnlyHint to false.

But the spec is explicit that these are only hints. Its schema says they "are not guaranteed to provide a faithful description of tool behavior", and that clients should never make tool-use decisions based on annotations from untrusted servers. The tools page says clients must treat annotations as untrusted unless they come from trusted servers.

So decide each tool's tier yourself, record it in your own configuration, and treat any tool you have not classified as tier 3 or higher. Our explainer on the Model Context Protocol covers the wider security picture.

What should the approver see?

An approval is only a control if the person can tell, in a few seconds, what they are agreeing to. The MCP spec says applications should present confirmation prompts and "show tool inputs to the user before calling the server". In practice a good approval request shows:

The exact action. For example, "Refund ₹4,200 to order 18734" (an illustration), not "process refund".
The arguments, in full. Recipient, amount, record IDs, the text of the email.
Who or what started it. The user, schedule or event that triggered the run.
Why the agent wants to do it. A one-line reason plus the records it relied on.
What will change and how to undo it. Or a clear note that it cannot be undone.
Approve, edit or reject. With room for a reason on rejection.

Bind the approval to the arguments. Approval should cover the exact call shown, not "permission to refund". If the agent later changes the amount or the recipient, that is a new action and needs a new approval. Store the approved arguments with the decision and check them again at execution time.

Let people edit, not only approve. LangGraph's interrupt documentation describes a review-and-edit pattern, where a person can "review and modify LLM outputs or tool calls before continuing". Fixing a wrong amount is faster than rejecting and waiting for the agent to try again.

Feed rejections back. The OpenAI Agents SDK lets a rejection carry a custom message to the agent. A reason like "customer already refunded last week" lets the agent stop or change course instead of trying the same thing again.

How does an agent pause and wait for a person?

An approval can take minutes or days, so the agent cannot simply sit in memory waiting. The common pattern in agent frameworks is to save the run's state, end the current process, and resume later.

The agent proposes a tool call that your rules mark as needing approval.
The run pauses and its state is saved to durable storage: the conversation so far, the pending call and its arguments.
The approver is notified with the request described above.
The decision is recorded with who decided, when, and any edits or reason.
The run resumes from saved state. Approved calls run with the approved arguments; rejected ones return the reason to the agent.

Two widely used frameworks show the same shape. In the OpenAI Agents SDK, tools can be marked as needing approval, paused calls appear as "interruptions", and the saved run state includes "approval decisions, pending tool calls, and tool arguments". In LangGraph, an interrupt() call pauses the graph, and a checkpointer "writes the exact graph state so you can resume later".

Both carry a warning worth repeating. The OpenAI documentation says: "Approval permits a tool call to execute; it does not guarantee exactly-once execution." LangGraph notes that when a run resumes, the paused step starts again from its beginning. If the process crashes just after an approved payment runs, a naive retry could pay twice. Give side-effecting tools an idempotency key so a repeat does nothing, and check the target system before retrying.

What happens if nobody answers?

Unanswered requests are a normal case, not an edge case. People go on leave, emails get buried, and the facts behind a request go stale.

Set an expiry on every request. Choose it per action: a customer reply may be stale in a few hours, a monthly report in a few days.
Default to not acting. An expired request means the action does not run. Never treat silence as consent.
End the run in a known state. Record what was done, what was not, and why.
Tell the person who started it, so the work does not silently vanish.
Escalate where it matters. Route tier 3 and 4 requests to a backup approver after a set time.
Recheck before running late approvals. If an approval arrives after the underlying record changed, show the new state and ask again.

How do you stop approvals becoming a rubber stamp?

The biggest risk with human-in-the-loop is not the agent. It is the person who has approved the last two hundred requests and stops reading. The EU AI Act names this directly: Article 14 asks that people overseeing high-risk systems are able to stay aware of "the possible tendency of automatically relying or over-relying on the output", known as automation bias.

Practical ways to keep approvals meaningful:

Ask less often. Use tiers and thresholds so only consequential actions reach a person.
Make the request quick to judge. Highlight what is unusual: a new recipient, an amount above the normal range.
Watch your own numbers. Track approval rate, rejection rate and time to decide per action type. An action that is almost always approved in seconds is either safe to move down a tier or not being read.
Do not let the agent approve itself, and do not let the person who asked approve high-risk actions alone. Use a second approver for tier 4.
Review rejected and reversed actions. Each one tells you whether a rule, a tool or the agent's instructions needs to change.

Does regulation require a human in the loop?

Sometimes. This section is a summary, not legal advice.

The EU AI Act's Article 14 requires that AI systems classed as high-risk are designed so that people can effectively oversee them while in use. The people doing the oversight must be able to understand the system's limits, decide not to use it, "disregard, override or reverse" its output, and "interrupt the system through a 'stop' button or a similar procedure". For certain remote biometric identification systems, it goes further: no action may be taken on an identification unless at least two people have separately verified it.

When this applies has moved. According to a Lewis Silkin briefing, the Digital Omnibus on AI was published in the EU's Official Journal on 24 July 2026 and entered into force on 27 July 2026. It moved obligations for stand-alone high-risk systems listed in Annex III from 2 August 2026 to 2 December 2027, and for AI built into products covered by Annex I to 2 August 2028. Check the official text for your case.

Most business agents will not be high-risk under the Act. The design ideas still transfer well: a person who can see what the system is doing, can override it, and can stop it. For Indian companies, our guide to AI governance under the DPDP Act covers how oversight fits with consent and audit duties.

A short human-in-the-loop checklist

List every tool the agent can call. Remove any it does not need.
Put each remaining tool into a tier, with thresholds where amounts or scope vary.
Treat unclassified tools as needing approval by default.
Show approvers the exact action, full arguments, trigger, reason and undo path.
Bind each approval to the exact arguments and recheck them at execution.
Save run state durably so a run can wait for hours or days.
Make side-effecting tools idempotent so a resumed run cannot act twice.
Expire unanswered requests, default to not acting, and escalate where needed.
Log every request and decision with who, when and what ran.
Review approval and rejection rates monthly and adjust the tiers.

How CloudOEngine helps

CloudOEngine is one platform and one console for connecting your data, running pipelines and running AI, including an AI runtime with an LLM gateway, agents with tool access, and RAG. Everything sits under one permission model and one audit trail, so an agent's access follows the same roles as your people.

Autopilot runs triggered agents with approval rules. When an action needs sign-off, a person approves it in the app or through an email approve link before it runs. That lets reading and drafting stay automatic while consequential actions wait for a person, which is the pattern this article describes.

Agents in CloudOEngine run real tools, such as filing a ticket, and every step and tool call is traced and inspectable. When someone asks why an action ran, you can open the run and see what the agent saw, what it proposed and what happened.

Governance is included, not sold as an extra: tenant isolation, role-based access, audit logs and per-tenant encrypted secrets, so tool credentials stay with the tenant that owns them.

CloudOEngine is built in Bengaluru, India.

Book a demo or see the AI runtime.

Frequently asked questions

What does human-in-the-loop mean for an AI agent?

It means the agent pauses at defined points and waits for a person to approve, change or reject what it plans to do. The person decides before the action runs, not after.

Which agent actions should need human approval?

Actions that are hard or impossible to undo, that move money, that contact customers or other outside parties, that change permissions or delete data, and anything outside the agent's normal pattern. Reading, searching and drafting can usually run without approval.

Should every agent action need approval?

No. If people must approve everything, they soon approve without reading, and the control stops working. Keep approvals for the few actions where a mistake is costly, and let low-risk steps run and be logged.

What should an approval request show?

The exact action and its arguments, who or what triggered it, the data the agent relied on, what will change, and how to undo it. The approval should cover exactly those arguments; if they change, the agent should ask again.

What happens if nobody responds to an approval request?

The request should expire after a set time and the action should not run. The agent should stop in a known state, tell the person who started it, and if needed escalate to a backup approver.

Does the EU AI Act require human oversight of AI agents?

Article 14 requires human oversight for AI systems classed as high-risk, not for every agent. It requires that people can understand, override and stop such systems. This is not legal advice; check how the Act applies to your use case.

Sources

#human in the loop #ai agents #agent approvals #ai oversight #llm security

Keep reading

Agent observability: what to trace in every AI agent run

AI agents in production: what breaks and how to fix it

MCP explained: how Model Context Protocol connects AI to systems