A support team gives its AI agent authority to auto-approve refunds under a fixed limit, meant to clear a backlog of small, obvious cases without a human touching every ticket. It works cleanly for three weeks. Then a customer's message pattern-matches the wrong refund template, the agent reasons its way to a number well above the limit, and because nobody had explicitly defined what the agent should do when a case doesn't fit the boundary, it doesn't pause, escalate, or flag anything, it just executes. The team didn't lack a smart agent. They lacked a system that made the agent stop and ask before doing something it had never actually been authorized to decide alone.
Why passing every test case doesn't mean an agent is safe to run unsupervised
A study of 35 industry developers building agentic AI products, published on arXiv, found that developers consistently "lacked mature controls for containing agentic risks, often relying on constraining the same characteristics that make agents useful: e.g., autonomy and goal complexity." In practice, that means the most common safety measure teams reach for is limiting what an agent can do at all, rather than building a real mechanism to catch and stop the specific action that goes wrong.
That's a structural gap, not a testing gap. An agent can pass every scripted test case and still take a wrong action the first time it meets a real input that wasn't in the test set, because passing tests proves it behaves correctly on cases you anticipated, not that it has any internal concept of when a decision is above its authority to make alone.
The three-tier model: pre-approval, async review, or fully autonomous
- High-risk, synchronous pre-approval: irreversible or high-value actions, a refund above a threshold, a contract term, a data deletion, sending anything externally that can't be unsent. The agent proposes the action and a human must approve it before it executes, no exceptions for edge cases.
- Medium-risk, asynchronous review: recoverable actions with real but bounded consequences, updating a CRM record, sending a routine follow-up email. The agent executes immediately but the action is logged for a human to review afterward, catching drift before it compounds without blocking every routine task on a person's availability.
- Low-risk, fully autonomous: reversible, low-consequence actions, looking up an order status, drafting (not sending) a reply, summarizing a document. No human step needed, because the cost of an occasional mistake is trivial and recoverable.
- The tier is a property of the action, not the agent's confidence in it. A confident, well-reasoned agent proposing a $4,800 refund is still a high-risk action requiring pre-approval, confidence is not the same signal as authority.
How to actually build the approval gate, not just say you have one
Anthropic's own guidance on building agents is direct on this: agents should be designed to "pause for human feedback at checkpoints or when encountering blockers," with human review remaining crucial even where automated checks pass, and extensive testing in sandboxed environments before an agent is trusted with real actions. The checkpoint has to be architected in, not assumed from good prompting.
Concretely, that means tagging each tool or function definition with an explicit risk tier as metadata, not leaving the model to infer risk on its own, routing any high-risk tool call to a queue or approval interface instead of executing it directly, and requiring the agent to pass its full reasoning and the source data it used along with the request, so the human reviewer isn't starting from a blank slate and re-doing the agent's research from scratch. A pending high-risk action also needs a defined timeout and escalation path, an approval queue nobody is watching is a synchronous gate with an infinite wait, which in practice becomes no gate at all.
The mistake that quietly defeats human-in-the-loop: approval fatigue
A risk-tiering system fails in a specific, predictable way if too many actions get routed to synchronous approval: the human reviewer starts rubber-stamping requests without reading them, because reviewing forty low-stakes approvals a day trains a person to click approve on autopilot by the fortieth one. The fix isn't removing the human step, it's being disciplined about which actions actually belong in the high-risk tier. If the async-review tier is empty and everything sits in synchronous pre-approval, the tiering wasn't actually designed, it was skipped.
How AIBOOTSTRAPPER solved this for AI Doctor
AIBOOTSTRAPPER built exactly this kind of tiered oversight into AI Doctor, a symptom-triage and consultation assistant for a Dubai clinic. The assistant handles initial patient intake and triage autonomously, low-risk, reversible territory, but every case escalates to a human doctor before any clinical decision is made, with a complete, pre-filled summary handed off so the doctor isn't starting the consultation from zero. That's the model applied to medicine: full autonomy on gathering information, mandatory human approval on anything with real consequences.
The result was 24/7 triage availability and 68% faster consultation prep, because the human doctor's time is spent on the decision that actually needs their judgment, not on the intake work an agent handles safely on its own. Full results are on the case studies page.
How AIBOOTSTRAPPER helps
AIBOOTSTRAPPER's AI automation team designs the risk-tiering into an agent's architecture before it ever touches production, not as a patch after an incident. That means defining, in writing, which actions a given agent can take alone, which need a human's eyes first, and what happens if that human doesn't respond in time, before a single refund, email or record update goes out unsupervised.
If you're deploying an AI agent that touches real customer data, money or commitments, book a call and we'll map out exactly which of its actions need a human in the loop before you ship it.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.