Human-in-the-Loop Escalation¶
Full autonomy is not always the goal. Build escalation points for situations the loop cannot resolve:
# In CLAUDE.md
If you encounter any of the following, stop and ask for human input:
- A destructive database operation (DROP, DELETE without WHERE)
- A push to main or production infrastructure
- A cost estimate exceeding $10
- Ambiguity about which of two approaches to take
- Three consecutive test failures with no clear fix
Auto mode escalates automatically (When auto mode falls back) when: - The classifier blocks 3 actions in a row — not necessarily the same action; any approved action resets this counter - Total denied actions reach 20 in a session
Where to Place a Checkpoint¶
The question is not whether to keep a human in the loop but where. Apply four tests to each step; a "yes" argues for a checkpoint there:
| Test | Ask | Checkpoint when |
|---|---|---|
| Irreversibility | If this output is wrong and not caught immediately, how hard is it to fix? | Hard to undo (send customer email) — not internal CRM tags |
| Confidence threshold | Does this input type produce unreliable output? | Agent should flag its own low-confidence outputs for review |
| External visibility | Does the action leave your internal systems? | Customer-facing email, published content, outbound API call |
| Context gap | Does tacit human context materially change the decision? | Relationship history / account context the agent lacks |
Place two to three well-placed checkpoints, not many — target roughly an 80/20 split: ~80% of cases handled autonomously, ~20% routed to a human. Too many checkpoints recreate the bottleneck loops were meant to remove.
Calibrate with the override rate. Measure how often a reviewer actually changes something at each checkpoint. "If reviewers are approving everything without changes 95% of the time, the checkpoint is probably unnecessary" — move or remove it. A good checkpoint is one where reviewers frequently catch something meaningful; track what gets flagged and what reviewers change to tune placement empirically rather than by assumption.
(MindStudio, "Human-in-the-Loop Checkpoints for AI Agents", Jun 2026.)
When the Checkpoint Has a Timeout¶
The four tests above decide where a checkpoint goes. A fifth decision is easy to skip: what happens if nobody answers in time.
A timeout on an approval gate converts it into an unattended gate whenever the default is to continue. The loop still asks the question and still pauses, so it looks like a checkpoint — but if silence means "proceed," the gate has become a fixed delay before the same unreviewed action, not a control on it.
Worked example. Claude Code's AskUserQuestion tool was changed to auto-continue after
60 seconds of no input instead of blocking indefinitely. The change shipped unannounced —
no changelog entry — and practitioners discovered it in production. Anthropic then reverted
the default, in 2.1.200: "Changed AskUserQuestion dialogs to no longer auto-continue by
default; opt into an idle timeout via /config."
No changelog entry names the version that introduced the behaviour — the reversal is in the
changelog and the introduction is not. The version is recoverable only from outside the release
notes: the article below pins it to 2.1.198 by diffing the shipped binaries, and Anthropic's
environment-variable reference now records it retroactively — "In v2.1.198 and v2.1.199,
auto-continue was on by default with a 60000 (60 seconds) timeout"
(env-vars reference). Auto-continue is now off by
default and opt-in via askUserQuestionTimeout. The thread drew 142 points and 120 comments (as of 2026-09-04)
on Hacker News (Hacker News, "Claude Code: Anatomy of a Misfeature",
Jul 2026; article at olafalders.com).
The Anthropic engineer responsible for the change acknowledged the process failure in-thread:
"the rollout should have been opt-in (like it is now) and on the Changelog."
A practitioner objected to the 60-second window itself, independently of how it shipped:
"How can I possibly provide any kind of informed answer in under 60 seconds? I can barely read some of its context for a question in 60 seconds!"
The two objections are distinct and both real: one is a process failure (an undocumented default change), the other is a design failure (60 seconds is not enough time for an informed answer to most questions worth interrupting a human for). A checkpoint that fails on either axis has stopped functioning as a checkpoint.
Design rule. State what happens on timeout as part of the checkpoint's definition, not as an implementation detail decided later: - Does the checkpoint block indefinitely, or does it time out? - If it times out, does the default action on silence continue the loop, or block it? - Is that default consistent with what the Irreversibility test (above) already said about this action?
Prefer default-block over default-continue wherever the action is not trivially reversible. A checkpoint on an irreversible action (a push to main, a cost overrun) that silently defaults to "continue" on timeout provides no real protection — it adds a delay before the unreviewed action happens anyway. Where a genuine liveness problem exists (a long-running job that must not deadlock on an absent reviewer), fix it with a timeout tuned to actual review effort, an escalation to a second reviewer, or a queued retry — not a short default-continue timeout on a decision the checkpoint exists precisely because it was not safe to make unreviewed.
This is one instance of the broader Silent Default Drift pattern: the gate was correct when the loop was designed, and it degraded without anyone touching the loop's own configuration.
The Three Feedback Loops¶
Human-in-the-loop is not one checkpoint — it is the middle of three nested feedback loops that run at different cadences in AI-powered product development:
| Loop | Who closes it | Cadence | What it corrects |
|---|---|---|---|
| Agentic coding loop | The agent itself | Every few minutes (build + test a new version) | Code-level correctness |
| Developer feedback loop | The human engineer | Tens of minutes to hours (reviews the product) | Direction, judgment, taste |
| External feedback loop | End users | Hours to weeks (usage data) | Whether the product is the right one |
"This idea of closing the loop took off around the end of last year, and it has been a game changer in enabling coding agents to work longer productively without human intervention."
Ng also gives the only first-party duration figure in his corpus, and it is worth holding against any claim about how long a loop can run unattended:
"my coding agent could easily work for around an hour, using a web browser to check what it had built multiple times before getting back to me"
The faster the inner loop closes on its own (verifiable checks, see Verification), the less often the human loop must fire — but the human loop never disappears, because of the context advantage: the human understands user needs better than current AI systems do. The engineer's job shifts from writing code to holding the context the agent lacks and steering at the cadence the agent cannot self-correct.
This complements the Inner/Outer Dual Loop: the dual loop nests execution inside strategy; the three feedback loops nest agent inside developer inside user by who provides the correcting signal.
(Andrew Ng, "Three Key Loops for Building Great Software", The Batch, 2026-06-26.)
Who Interrupts Whom, More Often¶
A first-party analysis of Claude Code traffic found that, on the most complex tasks, Claude asks for clarification more than twice as often as humans interrupt it mid-task. Read against the checkpoint-placement guidance above, this cuts against the intuition that autonomous loops mainly fail by over-reaching without asking — in practice the more common friction is the model pausing for input at a rate that itself needs tuning. This argues for calibrating checkpoint sensitivity symmetrically: the 80/20 target above is not just about adding enough human checkpoints, it is equally about not over-provisioning the model's own clarification triggers past the point where they start recreating the bottleneck loops were meant to remove.
Persona Document as a Calibration Feedback Channel¶
A concrete, low-overhead pattern for closing the Developer feedback loop on a judgment task rather than a code-correctness one: for a human-in-the-loop verification cycle on a news-filtering LLM app, the builder reads the model's daily justifications and, whenever the LLM's judgment diverged from his own, adds a rule to the persona document driving the loop. The persona document accumulates as a standing record of exactly where the human's judgment and the model's diverged — a running, example-anchored calibration log rather than a one-time prompt tuned once and left alone. (therundown.ai community, "Sam's full workflow (Rally News build)", Sep 2026.) (Anthropic, "Measuring AI agent autonomy in practice", Feb 2026.)