Humans in the loop

Approval thresholds, from none to every call, over a suite of benign tasks: prompts per task against the risky actions they gate, and how approval fatigue turns more prompts into less safety.

Loading the animation…

Concept

When the policy cannot decide, it asks: "private data would leave to a colleague: allow?" A person at the gate is a strong defence, because they see the whole situation, and a weak one, because they get tired. Claude Code's engineers describe the problem plainly: “Constantly clicking "approve" slows down development cycles and can lead to ‘approval fatigue’, where users might not pay close attention to what they're approving”, and report that in their internal usage sandboxing “safely reduces permission prompts by 84%” (Anthropic engineering, 2025-10-20, opened 2026-10-14). Fewer, better prompts are the goal.

The animation runs the label policy in a classic agent over the engine's suite of benign tasks (the first 12 are shown, one per step; the totals cover all 120), and you choose who must approve:

nobody: the policy's asks are refused. No prompts at all, but only 45.8% of the tasks finish.
the policy's asks: 0.30 prompts per task, and 72.5% finish. But the policy allows writes and sends that carry only public data, so a steered attempt still has an unchecked route in 50% of tasks: risk 12.3% per task.
every write and send: 0.59 prompts per task, no unchecked route left, and the risk falls to 2.4%.
every call: 3.64 prompts per task, and the risk goes up again, to 7.4%. Reads and summaries now ask too, so by the time a send comes up the person has already clicked several times, and each extra prompt makes a mistaken approval likelier.

That last line is approval fatigue in the model: a mistaken approval is 10% at a task's first prompt and 45% at its fourth. The curve under the calls shows where each task sits on it. The lesson is the one the quote draws: put humans where only a human can judge (the writes and sends that carry data out), and make everything else safe without asking, with a sandbox (chapter 5) or a plan (chapter 7).

Maths

A task makes $n$ prompts. The engine models the person's attention at the $k$ -th prompt as $(1 - a_1)\, r^{k-1}$ , so the chance of approving a call they should refuse is $a(k) = 1 - (1 - a_1)\, r^{k-1}$ , increasing in $k$ . A steered attempt (at base rate $p$ ) needs a route: a write or send the task makes under untrusted control, or, in a classic agent, an unplanned send. A route the policy denies passes with probability 0, one a human must approve with $a(n)$ , an open one with 1, and the attempt takes the best route:

\text{risk} = p \cdot \max_{\text{routes}} \text{pass}.

Gating more calls closes open routes (pass 1 → $a(n)$ ) but raises $n$ , and with it $a(n)$ on every gated route. The engine's tests check that prompts per task rise with each threshold, that gating every write and send leaves no open route, and the closed forms against a seeded simulation.

Code

Approval fatigue, cut from the vendored engine (policy/utility.ts).

export function approvalMiss(n: number, cfg: Obj = UTILITY): number {
  let keep = 1.0 - cfg.approval_first_miss;
  for (let i = 0; i < n - 1; i++) keep = keep * cfg.approval_decay;
  return 1.0 - keep;
}