Human-in-the-loop: when should an agent ask first?

The short answer

An agent should ask a human first when an action scores high on any of three axes: irreversibility (can you undo it?), blast radius (who else is affected?), and cost (what does it move or spend?). Score each action once, at design time, and encode the result as an allow / ask / block policy the runtime enforces — not as a prompt instruction the model can ignore.

Why “ask the human” needs a rule, not a vibe

Most teams handle approvals in one of two failure modes. Either everything asks — and humans rubber-stamp 200 approvals a day until the approval means nothing — or nothing asks, and you find out about the bad action from the customer. The 2025–2026 incident record is mostly the second mode: Uber exhausted its 2026 AI tools budget by April with zero per-action gates; Microsoft cut most internal Claude Code licenses in May 2026 after per-engineer usage reportedly hit $500–$2,000/month. And there’s the unverified one everyone cites — a reported ~$500M monthly Claude bill, per an AI consultant’s account reported by Axios; company unnamed, figure unconfirmed.

Approval fatigue is the real constraint. Every action you route to a human costs attention; spend that attention only where the model’s error is expensive and yours is cheap.

The framework: irreversibility × blast radius × cost

Score every action your agent can take on three axes, 0–2 each:

Irreversibility. Can the effect be undone, and by whom?

• 0 — trivially undoable (draft saved, branch created)
• 1 — undoable with effort (config change with rollback, refund you can claw back in theory)
• 2 — gone (email sent, payment settled, data deleted, tweet posted)

Blast radius. Who sees the consequence?

• 0 — the agent’s own sandbox
• 1 — your team or internal systems
• 2 — customers, production, money, or the public

Cost. What does the action move or spend?

• 0 — fractions of a cent (a read, a search)
• 1 — bounded and budgeted (an API call with a known fee)
• 2 — unbounded or large (refunds with a free-text amount, infra provisioning, bulk sends)

Then apply one rule: max of the three axes decides.

• max = 0 → allow. Don’t waste a human on it. Log it.
• max = 1 → allow with a budget, or ask above a threshold (e.g. refunds ≤ $50 allow, > $50 ask).
• max = 2 → ask, always — regardless of budget headroom. A correctly-budgeted catastrophe is still a catastrophe.
• Not in the list at all → block. If you didn’t score it, the agent doesn’t do it.

Using max rather than a weighted sum is deliberate. A weighted sum lets two harmless axes dilute one fatal one — “it’s cheap and internal” doesn’t matter if it’s irreversible. We tried sums; they produce arguments. Max produces decisions.

Encoding it

The framework only works if it’s enforced at the action boundary. Put it in the prompt and you’re relying on the model to police itself, which fails exactly when the model is confused — the case you built the gate for.

arc.defineActions({
  search_docs:   { defaultDecision: "allow" },                      // 0/0/0
  create_branch: { defaultDecision: "allow" },                      // 1/0/0
  issue_refund: {
    defaultDecision: "ask",                                         // 1/2/2
    allowIf: { amountLte: 50_00 },         // ≤ $50 auto-allowed, budgeted
  },
  send_email:    { defaultDecision: "ask" },                        // 2/2/0
  drop_table:    { defaultDecision: "block" },                      // why is this even callable
})
// Anything not declared here is blocked by default.

When an ask fires, Arc holds the action, notifies an approver, and the agent loop pauses — fail-closed, with the attempt recorded in a hash-chained audit log either way. Approved actions execute with an ES256-signed decision your application verifies before doing the work, so a forged “approved” can’t come from anywhere but the policy engine.

Pre-conceding the limits: this framework scores actions, not arguments. A scored-safe action with poisoned parameters (prompt injection steering send_email to an attacker’s address) passes the gate unless you also constrain parameters — which is why thresholds like amountLte and allowlisted recipients matter as much as the verb. And approval latency is real: if your agent needs sub-second autonomy on a max-2 action, the honest answer is that action shouldn’t be autonomous yet.

Recalibrate quarterly

Scores drift. An action that was internal-only (blast radius 1) becomes customer-facing when the product ships. Re-score on every new tool grant, and review the audit log for ask decisions that were approved 100% of the time — those are candidates for allow with a tighter parameter constraint, which is how you pay down approval fatigue without giving up the gate.

FAQ

How do I add human approval to an AI agent without rewriting it? Put the gate at the tool boundary, not in the agent. Wrap tool execution in a policy check that can return allow, ask (pause and wait), or block. For MCP-based agents, @geostack/arc-mcp-adapter wraps tool calls in the same evaluation without modifying the agent itself.

What happens to the agent while it waits for approval? It should pause, not crash or time out into a retry loop. Arc holds the pending action; on approval the agent resumes with a signed decision, on denial it gets a structured refusal it can plan around. Retry-on-deny is a bug — it spams your approvers.

Should every agent action require human approval? No. Blanket approval gates train humans to click yes, which is worse than no gate because it adds false confidence. Reserve ask for actions that are irreversible, externally visible, or expensive; auto-allow the rest under a budget and audit everything.

---

Arc is the spend-and-action guardrail for AI agents — free Developer tier, no credit card: npm i @geostack/arc. Quickstart →