Verification: The Non-Negotiable Foundation¶
A loop is only as trustworthy as its ability to check its own work.
Without verification, the loop has no stopping condition — it either halts too early ("looks done") or spins forever. Always give Claude a check it can run.
Verification is the loop's success stop: it is the completion check in the Stop Condition Taxonomy. Budget, max-iteration, and no-progress stops only contain a loop; the completion check is the only one that lets it succeed — and it is exactly as trustworthy as the verifier behind it. "An agent loop without a verifier just compounds its own mistakes on a schedule." (@bojan_ai, Jun 2026.)
A check is anything that returns a pass/fail signal Claude can read:
- Test suite (
npm test,pytest,cargo test) - Build exit code (
npm run build) - Linter (
eslint,ruff) - Script that diffs output against a fixture
- Browser screenshot compared against a design
- A separate evaluator model (a subagent that did not write the code)
Verification strategies¶
| Scope | Mechanism | How to set it up |
|---|---|---|
| Single prompt | Ask Claude to run the check and iterate in the same message | "implement X. run tests after. fix any failures." |
| Across a session | /goal condition |
"my goal is: all tests pass and the build succeeds" |
| Deterministic gate | Stop hook | Runs a script; blocks the turn from ending until it passes |
| Independent review | Verification subagent | Fresh model reviews the diff, not the work-in-progress |
Making subjective goals gradable¶
Verification only works if the success criteria are specific enough for an agent to check them. For subjective goals (design quality, code style, UX), convert them into four measurable dimensions before writing the loop:
| Dimension | Question | Example criterion |
|---|---|---|
| Quality | Does it feel coherent — not fragmented or accidental? | No element looks like a default placeholder |
| Originality | Is there evidence of a deliberate decision? | At least one non-standard colour or layout choice |
| Craft | Are the technical details correct? | Typography scale consistent, contrast ≥ 4.5:1 |
| Functionality | Does it work independently of aesthetics? | All interactive elements respond correctly |
Weight the first two heavily to push models away from generic outputs. Encoding these as explicit criteria (rather than prose instructions) makes them evaluable by a separate agent and reduces the ambiguity that leads to self-evaluation bias.
(Prithvi Rajasekaran, Anthropic Engineering — "Harness design for long-running application development", Mar 2026.)
The self-verifying loop pattern¶
1. Claude implements the feature
2. Claude runs: npm test
3. If tests fail → Claude reads output, edits code, re-runs
4. Loop ends only when tests pass
5. Stop hook runs linter as a final gate before the turn closes
Show evidence, not assertions. Have Claude output the test result, the command it ran, and what it returned. Reviewing evidence is faster than re-running verification yourself.
Simplification Before Testing¶
Standard practice: write tests, then verify the implementation passes them. When AI generates the implementation, this creates a trap: if the AI produces a working but structurally poor implementation, tests written against it cement the suboptimal structure. Passing tests make refactoring feel risky.
Inversion: run a dedicated simplification pass on AI-generated code before writing tests. The simplification agent has one job — make the code cleaner and more intelligible — without the pressure of test coverage. Then write tests against the simplified code.
This is Wave 4 in the five-wave execution model. See Harness Patterns.
(session-orchestrator — Kanevry/session-orchestrator, Jun 2026.)
Verification of Memory, Not Just Code¶
Loops with persistent state (GOAL.md, STATE.md, PROGRESS.md) accumulate stale entries over time. A verification pass that checks only code output misses a category of failure where correct code acts on incorrect state.
Before acting on any persistent memory record: 1. Validate that referenced items still exist (PR not merged, issue not closed, branch not deleted) 2. Re-read the original source rather than trusting the memory summary 3. Treat any memory record older than a defined threshold (e.g. 24h) as unverified
"…memory records can become stale; before giving recommendations from memory, verify current state; when memory conflicts with present reality, trust the current observed state and update or delete stale memory." — wquguru/harness-books, ch. 7 §7.7
Apply this specifically to: STATE.md watchlists, GOAL.md execution logs, PROGRESS.md task statuses, and any cached API responses.
(wquguru/harness-books, AgentWay, Jun 2026.)
Verification Classification: Type A vs. Type B Work¶
Before designing a verifier, classify the work being verified:
| Type | Definition | Verification approach |
|---|---|---|
| Type A | Fully automatable mechanics — dispatch, execution, evidence collection, commit, index update | Machine-checkable gates (CI, exit codes, diff counts) — no human needed |
| Type B | Human judgment gates — reviewing raw run output for trustworthiness and failure points, real-time calls when an agent is stuck on policy or permissions, and approving irreversible actions (e.g. merging a PR) | Mandatory human approval; the resulting lessons feed future automated-verifier design, but the judgment itself cannot be delegated to one |
The loop's job is to automate Type A completely and route Type B to humans reliably. A loop that tries to auto-verify Type B work (using LLM judges to approve irreversible actions) introduces Verifier Theater.
(void2610/loop, Jun 2026.)
Loop Verdict Taxonomy¶
Every loop run should produce one of six verdicts — not just pass/fail:
| Verdict | Meaning | Next action |
|---|---|---|
| pass | All done conditions met; evidence present | Merge / deploy / close |
| fail | Done conditions not met; retryable | Retry with attempt cap |
| handoff | Type B work reached; human judgment required | Route to human inbox |
| timeout | Turn or budget cap hit before completion | Resume or escalate |
| stopped | Hard stop triggered (security gate, denial hook) | Investigate root cause |
| awaiting-merge | Completion gated on an external event | Monitor; do not retry yet |
A loop that only outputs pass/fail misses the handoff, timeout, and stopped states that require different downstream responses.
(void2610/loop, Jun 2026.)
Cross-Run Verification Patterns¶
Standard verification re-runs the same checks each loop iteration. Three patterns extend verification across independent runs:
Clean-Room Review: The reviewer agent runs in a fresh session with no access to the implementer's reasoning — only the output artifact (diff, PR, test results). This prevents the reviewer from reasoning from the same context as the implementer and catching failures the implementer reasoned itself into.
Held-Out Test Layer: A set of perturbed inputs is generated before coding begins and kept hidden from the implementing agent. After implementation, the held-out tests run. A passing implementation that fails on held-out perturbations reveals over-fitting to the expected examples.
The holdout wall — evaluator-only, path-only disclosure. A refinement that hardens where the held-out layer physically lives: the acceptance checklist exists only inside the evaluator's role, and the generator (which negotiates its own "done" contract) is passed the holdout's path, never its content, and receives back only the evaluator's parsed verdict — never the checklist itself. This closes a gap the basic pattern leaves open: a "hidden" test the implementer's agent could still read from disk if both roles share a filesystem view. Cross-model pairing (e.g. Claude builds, Codex judges) is configured explicitly per role rather than auto-selected, which sharpens how to hold a test out but leaves which model pair to use an open, user-judged choice. (Kristopher Baker, "Sparra", Jul 2026 — the GitHub repo this project page links to now returns 404, checked 2026-09-05.)
Cross-Task Defect Ledger: When a run produces a defect (test failure, type error, security finding), the defect is logged to a shared ledger with root-cause category. Subsequent runs read the ledger before starting and explicitly check for the same root-cause categories. Defects stop being repeated rather than just fixed.
(JeremyW1990/loop-engineering-skill, Jun 2026.)
Reviewer Freshness Enforcement¶
Clean-Room Review (above) says the reviewer must run in a fresh session. This pattern gives a mechanism for why that isn't enough on its own and what "fresh" must actually mean.
Two independence axes, not one. "Model independence" (a different AI system reviews the work) and "perspective independence" (the same model, but with no shared conversation history) catch different failure classes — a same-model reviewer with a clean session still avoids context-poisoning from the implementer's rationalizations, even though it shares the model's blind spots. Both axes are needed; neither substitutes for the other.
Priming defeats freshness. A reviewer told what verdict is expected — even implicitly, by seeing the implementer's draft or reasoning before forming its own judgment — produces compliance, not review. The enforcement rule: a fresh reviewer must see only the problem statement and constraints, never the draft, until it has produced its own independent solution. The divergence between the reviewer's independent solution and the original draft is the highest-signal output of the whole review — it surfaces hidden framing assumptions the implementer couldn't see from inside their own reasoning.
Priming also has a looped form: a reviewer asked to "review and improve" finds a change every round, and the next session reads it as spec — see Review-loop drift in Failure Patterns.
The synthesizer problem. Even with a perfectly fresh reviewer, the human or agent who reconciles the two solutions remains the most biased party in the loop — they already have an opinion before synthesis begins. Mitigation: log every rejection alongside every acceptance (not just the accepted changes), and define convergence operationally as "no further draft changes after synthesis" rather than "the synthesizer feels satisfied."
This is a concrete answer to the reviewer-freshness-enforcement question Clean-Room Review leaves open: freshness isn't just "spawn a new session" — it requires withholding the draft until independent judgment is formed, and auditing the synthesis step itself for the same bias the fresh reviewer was meant to avoid.
(beingcognitive/unprimed-dialectic, Jun 2026.)
Belief State Machine for Claim Verification¶
When agent output contains factual claims (about APIs, configurations, security posture), classify each claim before acting on it:
| State | What it means | Required evidence |
|---|---|---|
| source_prior | Claim comes directly from a cited source (doc, test, API response) | Source URL or command output |
| bounded_claim | Claim is agent-generated but grounded in cited source_prior evidence | Explicit derivation from source_prior |
| validated | Claim has been independently checked by a deterministic verifier | Test result, grep, CI output |
Never act on a claim that remains in the source_prior or bounded_claim state for irreversible actions (deploys, security changes, data migrations). Require validated evidence.
R0–R5 Risk Classification — score tasks at intake before execution. The ladder below is this KB's own generalized tiering (its R5 row folds in this KB's own SECURITY_MATRIX.md convention); it is not a verbatim policy from any single cited project:
| Level | Risk | Policy |
|---|---|---|
| R0 | Read-only | Auto-approve; no verifier required |
| R1 | Reversible write | Auto-approve; commit-level evidence required |
| R2 | Merge-gated | Human review gate; test suite must pass |
| R3 | Prod-adjacent | Human review gate + second reviewer |
| R4 | Irreversible (data, secrets) | Explicit human approval before execution |
| R5 | Security-critical | SECURITY_MATRIX.md gate + security reviewer (see Agent Security Hardening) |
The additive risk-routing idea — classify at intake, keep the highest level, union the required
gates — is shared with qimen039-code/agent-cognitive-continuity-framework
(Jun 2026), but that project's own R0–R5 tiers route by task shape, not by this
read/write/prod/security ladder: R0 is the default for trivial tasks, R1 read-only inspection,
R2 artifact/report, R3 implementation or governance-doc edits, R5 confirmed destructive actions
(git push, delete). It defines no SECURITY_MATRIX.md — that is this KB's own convention,
sourced in Agent Security Hardening to
jahwag/clem.
The adjacent source_prior / bounded_claim / validated taxonomy above is verbatim from
that project's README and stays as written.
A/A Baseline for Verifier Calibration¶
Before trusting a verifier's verdicts, establish a noise floor using A/A testing:
- Baseline run: run the verifier on two identical implementations — it should produce identical verdicts; any disagreement is pure noise
- Noise floor: derive the noise floor empirically from the A/A run itself rather than assuming a fixed cutoff — agent-ab's own design note calls an assumed threshold (its example, "3× median") "the exact fragility this tool exists to avoid". Treat any specific disagreement percentage as a per-verifier rule of thumb you calibrate, not a figure from this source
- Only deterministic graders for binary gates: test exit codes, diff line counts, grep match counts — never LLM-generated scores
- Bootstrap confidence intervals: when comparing two configurations, compute a 95% bootstrap confidence interval on the difference of means — if that interval excludes zero, the effect is larger than run-to-run noise (agent-ab's own criterion; it is not a test of two intervals "overlapping")
Verifiers calibrated only on happy-path inputs will fail on the edge cases that matter most — the A/A baseline catches this before deployment.
(thalys/agent-ab, Jun 2026.)
Verifier Integrity: Keeping the Check Unfakeable¶
A verifier only protects the loop if the agent cannot quietly defeat it. Five patterns, converging across independent Claude-Code loop harnesses (Jun 2026), keep the check honest:
1. The external verifier — the loop runs the check, not the agent. In the
loop-kernel design the control loop itself executes the real check command every
iteration and reads the exit code; the worker never reports its own success.
The repo puts it plainly: "The kernel runs it, not the worker. Editing or skipping
your tests changes nothing — the kernel re-runs the real command every iteration." The principle: the control system is fixed and deterministic; the worker
is stochastic and swappable. This is the architectural cure for Verifier Theater
— the agent cannot approve itself because approval lives outside the agent.
(uppifyagency/loop-kernel, Jun 2026.)
2. Mechanical gates vs. adjudicators — two kinds of check, kept separate.
| Check type | Observes | Example |
|---|---|---|
| Mechanical gate | Runtime properties only the environment can see | gate.sh exit code, build, test suite, CI |
| Adjudicator | Whether a diff satisfies discrete acceptance criteria | read-only judge agent emitting structured JSON |
Keeping them separate is what prevents infinite loops: adjudicators cannot mistake weakened tests for correctness, and gates catch what agents cannot verify structurally. The repo's own framing is "两道闸:机械门(退出码)+ 跨模型对抗裁判" — two gates: a mechanical exit-code gate plus a cross-model adversarial judge. Correctness rests on "git + exit codes + a structured JSON contract, decoupled from agent output format." (firegnu/herdr-loop-lab, Jun 2026.)
3. Frozen tests — pin the contract before the implementer can touch it. A separate
role authors the test, it is pinned by content-hash and made read-only, and only
then does the implementing agent write code against it. The implementer "writes against
a read-only test it cannot touch," so it cannot pass by weakening the assertion.
Completion is reconstructed independently (git diff from the freeze commit + a real
test re-run), "not reported by the model." A later hardening of the same harness anchors this
completion capture to a real run-record: the capture-verified gate re-checks the recorded
run rather than trusting a stored capture, so a stale or faked capture can no longer satisfy the
stop — closing the gap where the evidence of a passing run was itself forgeable.
This is complementary to Simplification Before Testing,
not in conflict with it: simplification decides what shape the code and test should
take (and may run first); freezing decides who may change the test afterwards (the
implementer may not). (orobsonn/claude-harness — repo no longer publicly reachable; 404 as of 2026-09-05, Jun 2026. The quotes above cannot be re-verified: this source was live when they were captured (Jun–Jul 2026) and has since been withdrawn, with no Wayback snapshot (checked 2026-09-06). They are kept, dated, and flagged rather than deleted — see CLAUDE.md § Quoting a source that has since died.)
4. Provenance-bound claims — every assertion must cite a verifiable artifact. The
three patterns above keep the check honest; this one keeps the report honest. The
agent does not get to say "done" in prose — every finding or completion claim must cite
a git SHA, and a separate guard re-verifies the citation against the object store
(git cat-file) before the claim is accepted. A claim that points at no real artifact,
or at an artifact whose content does not match, is rejected automatically. Layered on
top: a majority-vote / monitor council — several independent read-only judges score
the same diff and completion is gated on their agreement, never on a single grader the
worker could collude with. Together these defeat the failure where an agent fabricates
evidence ("tests pass", "bug fixed at commit X") that no artifact supports. The
principle generalises the external-verifier idea from "re-run the check" to "bind every
claim to an inspectable, unforgeable artifact and have more than one judge confirm it."
(krishddd/Strive_Engineering, Jun 2026.)
Isomorphic-perturbation checks (anti single-predicate reward-hacking). A refinement
from the same harness: validate each claim under two independent-but-equivalent checks
(e.g. a literal git cat-file -e existence test plus a full object-ID re-derivation).
A gap that passes one check but fails its logical equivalent has found a single-predicate
shortcut — it is reward-hacking the specific predicate rather than genuinely satisfying the
goal — and the verifier exits non-zero. This generalises the A/A baseline
idea from "run the same check twice" to "run two equivalent checks once each; demand they
agree." (krishddd/Strive_Engineering, Jun 2026.)
5. Cross-model independence — the checker runs a different model than the maker. The
first four patterns keep the checker structurally independent (fresh context, separate
role, external command). This pattern adds a model-diversity axis: the reviewer runs on
a different model from the implementer, so the two do not share training blind spots or
failure modes. The recurring Jun 2026 configuration is Claude implements, Codex reviews —
the implementer writes broad multi-file changes and a separate reviewer model independently
inspects the diff, with roles often not fixed (either model can implement or review). The
argument: a same-model checker, however fresh its context, can rationalise the same mistakes
the maker made because it reasons from the same priors; a different model is likelier to
catch what the first is systematically blind to, and the two models' disagreement becomes the
productive tension that drives another iteration. These harnesses pair the cross-model
reviewer with a structured verdict schema that separates blocking from non-blocking
findings — mateaix/loope's reviewers emit VERDICT: PASS / VERDICT: BLOCK plus advisory
SUGGEST: lines, while the other harnesses use their own verdict vocabularies (herdr-loop-lab's
judge emits met/unmet per acceptance criterion in JSON; forja's emits VEREDICTO: PASS |
REJECT | BLOCKER) —
and gate completion on a dual stop condition: the loop terminates only when the mechanical
test command exits 0 and the reviewer raises no BLOCK. Multiple reviewers on the same
workspace aggregate by "any blocker ⇒ blocked." (Happenmass/omux,
mateaix/loope, firegnu/herdr-loop-lab,
Llicklair/forja, Jun 2026.)
Information asymmetry — the checker sees the output, never the maker's reasoning. A design principle that sharpens patterns 2–5: deliberately withhold the implementer's reasoning trace from the validator, so the validator judges only the artifact. A checker that never sees why the maker did something cannot inherit the maker's buried assumptions, cannot collude with its rationalisations, and — in the source's framing — "can't lie about code it didn't write." The loop rejects-and-retries until every independent validator approves or the gates pass. This is the anti-collusion rationale behind the fresh-context critics of the 8-phase DAG and the majority-vote council above, stated as a first-class rule rather than a side effect of fresh context. (the-open-engine/zeroshot, Jul 2026.)
A practitioner statement of pattern 5's own rationale, independently arrived at: "One model reviewing its own work agrees with itself. That is not verification, that is fluency." The prescribed fix is not just a different model but a different lens: ask several independent instances to argue the change is wrong, each from a different angle, rather than asking one model to confirm it is right. The same thread names what to trust in that setup: "Then the signal is convergence, not the verdict. What every lens flags independently is real. What they split on is where my judgement goes." It gives one instance: "Last month six lenses plus a cross-model pass all landed on the same finding, separately. That one was real." This is one practitioner's report, not a measured rate. (@kachar136; @kachar136; @kachar136, Sep 2026.)
These five together — external verifier, mechanical-gate/adjudicator split, frozen tests, provenance-bound claims (with isomorphic-perturbation checks), and cross-model independence — are the converging community answer to Verifier Theater: the verifier, the contract, and the evidence all live outside the agent's reach, no single judge can wave work through, and the judge that does check does not share the maker's blind spots.
A sixth mechanism, orthogonal to the five above: a reviewer with no shell at all, and an
advisory-only skeptic that cannot start a round. A CLI driving a Codex-implementer /
Claude-panel review loop found that restricting tool access is not the same as denying it:
"an allowlist as narrow as Bash(git diff:*) still permits git diff --output=FILE, which
git creates," so a reviewer holding any shell — however constrained — can still write. The
fix is structural rather than a tighter allowlist: the reviewer gets no shell at all, and
the diff it needs is written to a run-dir patch file for it to Read instead. Separately, the
panel splits into two files with different authority — .rb-lite-reviewers (gating: a finding
here starts another round) and .rb-lite-skeptics (advisory: findings reach the implementer
in every round the gating reviewers keep alive, but a skeptic alone never starts one and never
blocks a clean verdict). The skeptic's job is explicitly to argue for removing scope — "the
only one that can argue for removing something, which is what keeps a run from ratcheting" —
and making it advisory-only is what stops a skeptic from becoming a second gate that can never
converge: across eight runs on one drive, the panel went clean zero times when a skeptic's
findings were allowed to gate. (douglaz/rb-lite, Sep 2026.)
Enforcing read-only mechanically, per provider, not by prompt. A second, independent instance of "revoke the capability, don't just ask for restraint": a cross-vendor planner/coder/reviewer loop enforces the reviewer's read-only status at the tool layer — an OS sandbox for Codex, an SDK tool allowlist for Claude Code — rather than only instructing it not to write. The harness also qualifies each model for its assigned role before trusting it there: a per-provider adapter-qualification script emits a dated PASS/FAIL whitelist, and a later release pins the qualification script's own reference effort level so a user's global CLI effort setting can't silently fail re-qualification — keeping the harness's verification of itself deterministic, not just its verification of the work. (navels/neal; v0.6.10, Sep 2026.)
The plan is also a reviewable artifact — echo-chamber framing applied before any code exists. The cross-model patterns above gate the implementation; one harness applies the same "the model that wrote it can't be trusted to grade it" logic one stage earlier, to the plan. Claude locks a plan with the human, then Codex attacks the locked plan read-only across bounded rounds; on the build stage the roles invert for cross-inspection. Producer and grader are named as configuration ("orange"/"green"), not tied to a specific vendor, so the same harness works with either model in either seat. (ujconsulting/claudex-loop, Sep 2026.)
Arbitrating disagreement between cross-model reviewers. Once you pair multiple
independent models as reviewers (pattern 5), a new question follows: what do you do when
they disagree? A multi-model (Claude + Codex + Gemini) reconciliation harness found that
~85–90% of findings are caught by exactly one reviewer — not by consensus — which
argues for model-family diversity over adding more reviewers of the same family for
consensus voting: the marginal reviewer that catches something new is more valuable than
a second vote on something already caught. To reconcile severity across reviewers it uses
verdict-driven severity: a verifier's confirmed/refuted judgment overrides the raw
severity a reviewer assigned, with a "promote-on-confirm" rule that floors any
independently-confirmed bug at high severity regardless of how mild the original
reviewer rated it. Cross-reviewer overlap is tracked mechanically (a findings database
with an agreement_n count per finding) rather than argued about at review time, and the
whole reconciliation is bounded to 3 rounds, gating only on blocker/high severity so minor
disagreements don't stall the loop. The cap is not arbitrary, and the reason is the useful part:
the source's own second design rule is that "iterating a fix→re-review loop past ~3 rounds
introduces bugs." Rounds four and up are not diminishing returns — they are negative ones. Nits
go to a backlog and never extend the loop, and re-reviews are scoped to "are these prior blockers
fixed, any new blocker?" rather than re-run whole.
(houshuang/compound-review, Jun 2026 — README
read from master, which is this repo's default branch; main does not exist and a raw fetch
against it 404s.)
The same non-overlap holds outside LLM-judge harnesses, in shipped code-review products. Four commercial AI code reviewers (CodeRabbit, Sentry Seer, Greptile, Cursor BugBot) run in parallel across 146 real PRs over three weeks produced 679 findings, of which 93.4% were caught by exactly one tool. No line was ever flagged by all four at once, and overlap where it did occur was thin: 37 lines drew exactly two tools, 4 drew exactly three. That is largely non-overlapping, not perfectly so — and it corroborates the ~85–90% figure above with an independent, non-LLM-judge data source: the non-overlap is not an artifact of how LLM-as-judge harnesses are built, it recurs in production tools built by different vendors on different review philosophies. (dev.to, "Best AI Code Reviewer in 2026?", May 2026.)
Which pair, and in which direction — two 2026 results that sharpen this¶
The advice above stops at "use a different family." Two controlled studies push past it, and the second complicates the first.
1. The pairing is asymmetric — order matters more than difference. A controlled experiment over 116 hard and medium LiveCodeBench tasks, across six conditions (both solo baselines, both cross-model orderings, both same-model orderings), with the reviewer seeing the problem and the draft but unable to execute tests:
| Condition | Pass rate |
|---|---|
| Codex solo | 71.6% |
| Claude reviewing Codex | 89.7% (p_BH = .001) |
| Codex self-review | 84.5% (p_BH = .022) |
| Claude solo | 91.4% |
| Codex reviewing Claude | 82.8% — worse than solo (p_BH = .046) |
| Claude self-review | 91.4%, unchanged |
Its own conclusion: "the useful pairing is asymmetric: use Claude to review Codex, not the other way around." Note the shape, which generalises past these two models: a cross-model reviewer can make results worse than no review, and "Codex self review raises them to 84.5%" — a same-family reviewer beat the cross-family one in that direction. So "different family" is not the operative property, and a pairing must be validated in the direction you intend to run it. (arXiv 2607.21656, Xiang et al., Jul 2026; accepted at Agentic SE @ KDD'26 per the paper's own comment field.)
2. And diversity may not come from lineage at all. In the adjacent domain of trusted monitoring — a cheap trusted model scoring a stronger untrusted model's actions on backdoored code — a study of 24 open-weight monitors across nine pretraining lineages found that buying decorrelation through family diversity does not pay: "At matched member capability, cross-lineage panels detect no better (permutation p = 0.13)." What predicts a monitor's agreement with the pool is its skill, not its lineage (Spearman 0.84). Worse for panel-building generally: "Panel gain over the best member falls monotonically with panel skill", and "no correlation-weighted selection beats picking the single best monitor out of sample."
How to hold these together without overreading either. The domains differ — backdoor detection by trusted monitors is not code review — and this does not overturn the ~85–90% non-overlap finding above, which is about what gets caught, not about where the diversity comes from. Both can hold: findings are largely non-overlapping, and pretraining family may not be the reason. The practical correction is to stop treating "different family" as the design rule and treat it as a proxy that has now failed one direct test:
- Prefer a strong reviewer over a merely different one. At matched cost, a second reviewer that is better beats one that is differently descended.
- Validate the pair you actually run, in the direction you run it. Result 1 shows the same two models helping in one direction and hurting in the other.
- Diminishing returns arrive fast. Panel gain over the best member fell as panel skill rose — ensembles help most when every member is weak.
(Decorrelation Is Not Complementarity: Skill, Not Lineage, Governs Trusted-Monitor Ensembles, Anik Jha, arXiv, Aug 2026. Worth reading for its method as much as its result: it reports a failure against itself — "on our own 22-monitor pool the same test read +0.104 at p = 0.037 until two monitors were added" — which is the behaviour this KB asks of its own claims.)
Two-tier verdicts: never let a judgement call fail the build¶
A verifier that emits one verdict has to answer two different questions with it — did a mechanical check fail? and does this look wrong? — and the second is an inference that can misread. Collapse them and you get one of two bad outcomes: block on a guess, or downgrade the certainties to match the uncertainties.
A release-readiness gate that separates them splits every finding in two:
| Tier | Applies to | Fires when |
|---|---|---|
| BLOCK (hard, fail-closed) | mechanical boundaries — objective, zero inference | the check has a single right answer and it failed |
| WARN (advisory, surfaced to a human) | the one judgement call | a classification suspects a mismatch |
Its BLOCK set is entirely decidable — version bumped, version is valid SemVer, changelog has a
matching dated entry, tag does not already exist, [Unreleased] not left populated, working tree
clean. Its single WARN is whether the severity of the changes matches the size of the version bump,
which is an LLM judgement. The rule attached to it is the transferable part, verbatim:
Never escalate this to a BLOCK — the classification is an inference and can misread; auto-blocking a legitimate release on a guess is exactly the verifier-theater the constitution forbids. Be explicit that this is a suspicion, not a verdict.
Why this belongs in a loop's stop condition. An unattended loop cannot ask, so its verifier must be honest about which of its outputs are facts and which are opinions. Give the loop fail-closed authority over the decidable set only, and route everything inferential to a human as a labelled suspicion. A loop that blocks on inference will be switched off after its second false positive; a loop that blocks on nothing is Verifier Theater. The split is what lets one verifier be both trustworthy and unattended.
(Claude-Warp claude-warp-release,
read 20260906. First-party: same maintainer as this KB — see
the disclosure. Presented as a
worked example of the split, not as independent evidence for it.)
What is still not settled. No source found gives a general ranking of which model pairs catch
the most distinct failure classes in code review. Result 1 is two models on one benchmark; result 2
is a different task. Treat pair selection as something you measure on your own diffs, not something
you look up. Logged in KB_GAPS.md as still open after four retries.
Per-criterion independent verification as the stopping condition. A distinct refinement of pattern 1 above: instead of one verifier judging the whole diff, define "done" as a manifest of acceptance criteria, and spawn one independent verifier per criterion rather than one verifier for the whole task — no single verifier holds (and can rationalize around) the full picture, and completion requires every criterion plus a global invariant check to pass, not an aggregate judgment call. This is preceded by an adversarial understanding gate that challenges assumptions and reads the codebase before any manifest is written, on the premise that most agent failures are actually upstream — confidently solving the wrong problem — not downstream verification gaps. The manifest is a more granular version of GOAL.md's done-conditions checklist: where GOAL.md tracks one checklist for the whole goal, a manifest gives each criterion its own independent verifier. (doodledood/manifest-dev, Jul 2026.)
Adversarial gate quantified in production: 43% fabrication rate caught. A zero-polling fleet orchestrator (one orchestrator dispatching to Claude Code workers, Fan-Out) ran a separate, cheaper reviewer model against every worker's output before integration. Across 28 dispatch cycles and roughly 100 reviewed changes in one overnight run, the adversarial gate caught a 43% fabrication rate in generated summaries — plus fabricated benchmark numbers, wrong pricing, and broken UI links that the implementer's own report had missed entirely. A concrete number for what "the checker catches what the maker cannot self-see" is worth in practice, not just an argument for why cross-checking matters. (walidboulanouar/loop-engineering, Jul 2026.)
Language-independent test suite as verifier for a full rewrite. A distinct technique for keeping the verifier honest during a cross-language rewrite: write the test suite in a third language, independent of both the old and new implementation. Bun's Zig→Rust port (535,496 lines, 11 days) verified every change against an existing TypeScript test suite with roughly a million assertions — a verifier that cannot be gamed by matching implementation-specific quirks in either Zig or Rust, because it was never coupled to either. This generalises pattern 1 above (external verifier) one step further: not just "the loop runs the check, not the agent," but "the check is written in a language the rewrite cannot influence." (Bun, "Bun, in Rust", Jul 2026.)
Blind adversarial review, quantified on a full production rewrite. A concrete large-scale instance of the information-asymmetry pattern above: on the same Bun rewrite, one Claude Code instance implemented while two or more separate instances — with no visibility into the implementer's reasoning — were each charged with a single mandate: "find bugs & reasons why the code does not work." Bun states the ratio and the role separation verbatim: "1 implementer, 2 or more adversarial reviewers per implementer... The implementer doesn't review. The reviewer doesn't implement," and "Every line of code was reviewed by two separate adversarial reviewers (also Claude) and went through a round of fixes before committing." The duplication is not redundancy for its own sake — two independent reviewers are what make a single reviewer's miss recoverable, which is the same argument Fleet Engineering records at fleet scale. Run at fleet scale (peak 64 parallel instances across 4 worktrees, see Fleet Engineering's case study), this produced 128 bug fixes in the v1.4.0 release with only 19 regressions introduced across the full 535,496-line rewrite — a real-world data point for what blind adversarial review buys at scale, not just at single-diff granularity. (Bun, "Bun, in Rust", Jul 2026.)
Review Loops That Converge: Anthropic's Own Process and Open-Source Ledgers (Oct 2026)¶
The patterns above say a reviewer must be independent. They say less about the review loop itself — how many rounds, who may end it, what happens when the reviewer is down. Oct 2026 added a first-party account from the team that builds Claude Code, and a cluster of small open-source harnesses that converge on the same mechanics.
Anthropic's account — layers, then convergence. The Claude Code team's post (Addy Osmani, Morgan Westlee Lunt and Qing Wang, Oct 15 2026; first-party, figures self-reported) describes six layers, each catching some problems and missing others, with rollback underneath:
| Layer | What it does |
|---|---|
| Verified before the PR | Claude runs tests, build and linter and shows the output; a second agent reviews the diff in a fresh context |
| Required CI | Lint, types, tests across platforms, build; red blocks the merge |
| AI review + security review | Agents hunt for bugs in parallel, a separate group votes on each finding and only survivors are posted, ranked by severity; open findings from either review block the merge |
| Risk-tiered approval | A classifier reads the change (not the author) and sets the tier: no person, one, or two. An agent's approval never fills a slot that calls for a person, and an approval covers one specific commit |
| Merge queue, flags, gradual release | Checks re-run on the latest main; new behaviour ships behind a flag to employees first |
| Fast detection and rollback | Flags turn off without a deploy; incident reviews ask which layer should have caught it |
The post reports that from June to September merged PRs to the Claude Code repos more than doubled while the share reverted fell by more than 40 percent, and that the share of internal PRs with substantive review comments rose from 16% to 54% when the reviewer launched in research preview. Code Review has since left preview for Team and Enterprise with admin-selectable Quick/Standard/Deep levels per repository; the GA page is the vendor's own claim of ~3x faster reviews (about 20 minutes down to 6–10).
The most transferable part is that the reviewer is itself a loop that must converge:
- The failure. "An agent author and an agent reviewer can keep each other busy indefinitely";
one CLI PR went through more than fifty review-and-fix rounds overnight.
git blameon what the reviewer kept flagging showed about 59% was code present since the reviewer's first pass — it was "finding a few more old ones each round", not new bugs. - The fix. A cap of three rounds, after which the reviewer asks the author to step back and reconsider the approach; diff-only re-review after the first few passes; flag the class of bug so the author fixes every instance; no repeats of findings already posted.
- The result (internal PRs, before → after). Passing after one review-and-fix round 65% → 78%; needing five or more rounds 8.9% → 1.5%; mean reviews to pass 2.08 → 1.32. This independently matches the ~3-round cap, and the finding that rounds beyond it introduce bugs, reported for compound-review above.
- The reviewer has its own eval. Past incidents and reverts, with each bug confirmed by an engineer who knows the code, form an eval set the reviewer is scored against daily; a change to its prompts, tools or model is kept only if the score rises. Noise is measured by how authors respond to each comment (fixed, wrong, intentional, did not matter) and by running the reviewer on PRs known to be clean. The stated priority is real bugs first, fewer rounds second.
- What is admitted missing. The team has no circuit breakers that tighten approval when revert or incident rates move, and says engineers want them before more code moves to zero-human merges. That is the unbuilt half of the STOP-condition design: a budget or round cap contains the reviewer loop, but nothing yet feeds production outcomes back into how much human review a change gets. See also Human-in-the-Loop.
Open-source review-loop harnesses — the same mechanics, small and unproven. All of the following are young repos (0 stars at capture, dates are last push); they are designs to compare against, not evidence.
| Harness | Mechanism worth taking | Caveat |
|---|---|---|
| review-ledger | Claude authors, Codex reviews; a per-repo ledger gives every finding an id and forces a fix / reject / dispute / defer reply; four terminal states (converged, escalated, capped, stalled); a Stop hook blocks ending the turn while blocking findings are open; a per-turn check that the reviewer's model and effort did not drift when the thread resumed | README claims one-way review was "measured at 10-20 rounds" vs 2-3 with the ledger, but the protocol doc gives no data — an author claim |
| Reviewgate | Fail-closed Stop-hook gate over a multi-provider reviewer panel; a crash, timeout or quota outage is never turned into PASS (it blocks, defers within a bound, or escalates to a human) | Its alpha benchmark (30 hand-written cases x 3 repeats) reports baseline precision 0.3505, recall 0.8095 and clean-case false-positive rate 0.7292, with a critic stage cutting clean false positives by ~16.7 points; the page itself says the corpus is hand-written and the result is not a leaderboard |
| dev-loop | Opus writer, Codex reviewer, and a recorded-trust table that decides which findings may be auto-fixed: a route moves from ask to auto after at least 9 of its last 10 findings were accepted with no veto, and falls back to ask after 2 rejections or at once after one veto of an auto-fix. Also treats reviewer-induced needless complexity ("both models add code") as a defect | Thresholds are the author's own defaults |
| cc-builder-loop | Four independent gates (machine, blind tester, proof, reviewer), each bound to the digest of its inputs so evidence expires when the inputs change; the tester subagent writes tests blind to the implementation; the same failure three times is a hard stop that needs a recorded user decision to resume | README is in Chinese |
| review-fix-loop | Stops when a round has no verified Major finding, or hands back after 4 rounds, when the same Major survives two fix attempts, when findings flip-flop, or when tests fail twice; state lives in a ledger file so it survives context compaction | Largely restates patterns already in this doc |
What these add beyond the earlier sections: the reviewer's own infrastructure failing must not read as approval (Reviewgate; this KB's own rule that a check which cannot tell must fail, never pass, applied to the reviewer — see Verifier Theater); evidence should expire when its inputs move (cc-builder-loop); and autonomy can be earned per route from a track record rather than set once (dev-loop) — a graduated version of the risk-tier table in Belief State Machine.
Turning review verdicts into structured facts. Three Oct 2026 commits to agent-ready-repo show the work loop moving off adjudication prose. 3.0.5 adds closed review schemas, deterministic obligation selection, atomic report persistence and authority-bound dispositions with a pure readiness query, so a verdict becomes a fact a gate can read. #1480/#1483 sizes review units by whether they can be independently understood, verified or reviewed (replacing a ~2,000-line gate) and derives a position-free family key (location plus stable title) so recurrence detection survives renumbering — the mechanical counterpart of Anthropic's "flag the class of bug, no repeats" (companion commit). 3.1.3 feeds unsuperseded review-failure receipts into a read-only, candidates-only mode that proposes bounded "gotcha" candidates (it stores nothing) — a path from review failures back into project knowledge.
Critic mechanics in tenet. Three releases of tenet
harden the critic stage described in Structured Critic Finding Taxonomy:
v26.10.0 gives critics an
applicable_jobs list and makes tenet_retry_job refuse to re-run a critic that already produced a
verdict (red or green) or to retry after dev/integration jobs have completed since dispatch — so the
"infra-failure retry only" rule becomes server-enforced instead of advisory, closing verdict
fishing (re-asking until a critic says green); the commit cites roughly 69 hours of critic eval
against 6 of dev as the motivation. v26.10.1
adds a per-critic parallel_safe flag so read-only critics run concurrently while app-exercising
critics chain sequentially. v26.10.2
adds a per-round append-only evidence ledger: the first critic to run the suite, lint or build logs
the result and siblings cite it skeptically (excerpt-bounded, stale if HEAD moves, never a gate,
entries carry no pass/fail verdict so the rubric parser cannot false-green). Its commit notes two
adversarial reviews hardened that design.
Isolating the cross-model reviewer, and keeping the plan honest. The claudex-loop family and
ECC's reviewer command add operational detail to pattern 5. Upstream
claudex-loop #28
records that a read-only shell sandbox does not govern MCP, plugin or app side effects, so each
Codex review disables apps and plugins and every user MCP server (read from codex mcp list
--json), failing closed when that listing is unusable. The
fork's wrapper 2.6.0
detects a blind review by counting executed commands in the event stream (a newer codex-cli
stopped printing the string it used to look for) and refuses to log one, and refuses a project
.codex/config.toml after measuring that its developer_instructions reached the reviewer; it
reports a 49-mutant probe of the blind-run logic with no survivor (self-reported).
Wrapper 2.6.1
treats a model/CLI bump as a measured procedure — a preflight proves a real command ran and a shell
write is refused — and the upstream record
adds the reviewer's effective model, effort and sandbox, plus a rule of no cross-model fallback
when the reviewer is out of quota. Delegated builders
likewise load no user hooks, MCP tools, apps or plugins. ECC's
/astra-review
sends Claude-written code to another provider's CLI for a PASS/FAIL review with the same isolation
(read-only sandbox, web search off, user config skipped, every configured MCP server disabled by
name, filtered environment).
On the plan side, the fork and upstream add two rules from field reports: when a REVISE changes behaviour, every plan passage stating the old behaviour (acceptance criteria and Verification included) must be changed or marked superseded, with the new plan's SHA256 logged; and post-build findings that change the build must edit PLAN.md in the same pass and be listed at the human gate as a post-approval plan change. The PLAN.md template gains a Verification section requiring proof on the real output of the producing program, with old-vs-new code run over the same inputs.
Eval Metrics: pass@k vs. pass^k¶
A single pass/fail run cannot tell you whether a loop is reliably correct or just got lucky. Because agents are stochastic, measure success across repeated trials:
| Metric | Definition | Use it to answer |
|---|---|---|
| pass@k | At least one success in k attempts |
"Can the loop do this?" — capability |
| pass^k | All k trials succeed |
"Is it safe to run unattended?" — reliability |
pass@1 is the raw first-attempt rate; pass@3 allows up to two retries. A useful
target for capability evals is pass@3 > 90%; for critical paths that must not
fail even once, require pass^3 = 100%. The gap between a high pass@k and a low
pass^k is exactly the flakiness a stochastic loop hides — a loop can look capable
(pass@3 high) while being unsafe to automate (pass^3 low).
Grade each trial with one of three grader types (escalating in cost and subjectivity):
| Grader | Mechanism | When |
|---|---|---|
| Code-based | Deterministic — grep, test runner, build exit code | Objective, binary criteria (prefer for gates) |
| Model-based | A judge model scores open-ended criteria | Subjective quality (see Opik below) |
| Human | Flagged for manual review, risk-tiered LOW/MED/HIGH | Type B / irreversible work (see Human-in-the-Loop) |
Only code-based graders belong in a binary pass/fail gate — this is the same constraint the A/A Baseline imposes: never gate on an LLM-generated score. pass^k on a code-based grader is the quantitative form of the "safe to run unattended" question the Loop Readiness Levels answer qualitatively.
(affaan-m/ecc eval-harness skill, Jun 2026.)
Blend human and model grading with an explicit weight. When neither a code-based grader nor an LLM judge alone is trustworthy, blend them at a stated ratio rather than picking one. One reusable model-comparison harness scores with 70% human "vibe" + 30% LLM-judge, using a local HTML page that exports gut-feel ratings to JSON for a blind comparison — the weighting is a design decision, not an accident, and it keeps a human majority-stake in subjective calls while still scaling. (Claire Vo, "Sonnet 5 review: I ran 64 generations", Jun 2026.)
Proof-of-Work Artifacts: the Verifiable Demo¶
Not every hand-off can be reduced to a pass/fail exit code. For UI/UX work, the verification artifact can be a recorded demo the agent produces itself: the agent writes a YAML "storyboard" of interaction steps, Playwright drives the browser, and the result is a video that shows the feature actually working — "the importance of having coding agents produce demos of their work." This is a human-viewable verification artifact, complementary to (not a replacement for) the machine-checked gates above: it closes the gap for behaviour a test suite can't fully assert, and it is what a human reviews at a handoff checkpoint.
(Simon Willison, "shot-scraper video", Jun 2026.)
LLM-as-a-Judge Verification with Opik¶
Comet's Opik (open-source, 40M+ traces/day) provides a verification layer specifically designed for LLM output quality — not just pass/fail exit codes.
Evaluation metrics (callable via .score() API):
- Hallucination — detects factual claims not grounded in the context
- AnswerRelevance — scores whether the agent's output addresses the actual task
- ContextPrecision — measures how precisely the context supports the answer
- Moderation — flags unsafe or policy-violating output
(Instantiate the class, then call .score(...) on it — e.g. Hallucination().score(...).)
Online Evaluation Rules — configure continuous scoring against a running production loop: a judge model monitors every run and alerts when metric scores fall below a threshold. This is the fleet-level equivalent of a test suite: instead of checking one run, it watches the whole fleet continuously.
PyTest CI/CD integration — evaluations run in CI pipelines as part of the test suite. A harness that passes code tests but fails answer-relevance scores is not production-ready.
The distinction from the A/A Baseline (which establishes noise floor on a verifier's consistency): Opik's metrics measure the quality of agent output directly.
repo: github.com/comet-ml/opik
(Comet ML, "Evaluation Metrics Overview", Jun 2026.)
Real-world case study: Mozilla Firefox security harness¶
Brian Grinstead, Christian Holler, and Frederik Braun (Mozilla, May 2026) built a security bug-finding harness for Mozilla Firefox with explicit verification at every stage:
- LLM file prioritization — a scoring model ranked files by bug likelihood before allocating agents; agents spent time on high-signal targets, not a full codebase scan
- score → verify → fix pipeline — a bug candidate is confirmed real (a proof-of-concept testcase that trips the sanitizer) before any fix is attempted, not after
- Dedicated verifier subagent — a fresh agent, not the bug-finder, confirmed each fix and eliminated false positives; tuned explicitly to avoid accepting fixes with unresolved edge cases
Result: 423 security bug fixes in one month.
Brian Grinstead has put the credit close to 50/50 between harness and model — the harness did about as much of the work as the model itself. (Paraphrased: the interview is a podcast and the exact wording could not be verified against a transcript, so it is not quoted here.)
(Mozilla Hacks — Brian Grinstead, Christian Holler & Frederik Braun, "Behind the Scenes Hardening Firefox with Claude Mythos Preview", May 2026; credit-split framing from Brian Grinstead interviewed by Claire Vo, "How Claude Mythos found a 15-year-old bug in Mozilla Firefox", Jun 2026.)
A second, higher-volume case study for the same "verification is the constraint, not review capacity" thesis: a "pstack" workflow where a dedicated verification skill lets one engineer ship roughly 2,000 PRs a month (~100/day) to production — the case argues human review structurally cannot be the verification layer at that volume, so the verification skill has to be. (This is a different case study from Warp's similarly-sized PR volume in The Factory Model — coincidentally similar numbers, unrelated companies.) (The New Stack, "One engineer shipped 2,000 PRs a month to production. Verification is the key.", Sep 2026.)
A single-contributor instance of the same discipline. Peter Steinberger reports that OpenAI's Astra "found a ~14 year old bug in libuv" while he chased ChatGPT crashes on macOS 27 (@steipete, Sep 2026). The age is his own claim; no PR states it. The fix PR is the useful part. It discloses the method, from an author who notes he works at OpenAI: "I did use GPT 6 Astra Ultra for the review, with many verification and adversarial subagent rounds". Its regression test comes with negative controls ("the original runtime starts native streams successfully but fails with 17 outstanding allocations"). It also scopes its causal claim: the leak is "a plausible mechanism" for the reported crashes, and the reproduction "does not establish that this leak is the sole cause of those crashes" (libuv/libuv#5283, Sep 2026). A follow-up pass with Daybreak "found 8 more long-standing leaks" (@steipete, Sep 2026). As of 2026-09-28, one of his nine libuv PRs (#5283-#5291) was merged and eight were open.
"Surface" — the Canonical Stopping Verb¶
When a loop reaches a point requiring human judgment, the agent's action has a precise name:
Surface — stop the current loop, emit a short description of the situation (what happened, what you tried, what state things are in), and wait for human direction. Do not retry, do not redispatch, do not silently reset.
This is the distinction from other verdicts:
- fail — retryable; the agent should try again with attempt cap
- stopped — hard gate fired; investigate root cause
- handoff — human judgment required; this is the Surface action: emit a clear situation
report and halt
A loop that surfaces correctly is more trustworthy than one that retries indefinitely: it shows the agent knows the boundary of its own competence.
(eugenelim/agent-ready-repo, Jun 2026.)
Verification Mode Discipline¶
Before writing any test or verifier, declare which of three verification modes applies. A mode-mismatched test passes for the wrong reason.
| Mode | When to use | Key constraint |
|---|---|---|
| TDD | Pure functions, state machines, protocols | Test must pin a real invariant — not mirror the implementation. Tests that change in lockstep with production code are mirrors, not contracts. |
| Goal-based check | Verify an artifact exists or has a shape | The one-liner verification is the contract; no extra test file needed. |
| Visual / manual QA | UI behaviour, rendering, layout | Invariants must be explicitly named — e.g. "no crash, no overflow, layout holds" — rather than left implicit; input variation must be recorded or seeded reproducibly. |
Level-of-abstraction rule: verification level must match the behavior boundary being tested. UI behaviors need tests that simulate the user's gesture and assert on rendered/visible state — not unit tests on the controller. "Mode-mismatched verification produces tests that pass for the wrong reason."
(eugenelim/agent-ready-repo, Jun 2026.)
Self-Coverage Gate¶
A design-decision-disposition discipline for a loop's convergence gate: every open design or scope item must be marked either resolved-with-referent (cite what settled it) or surfaced-with-reason (value origination, irreversible risk, or value conflict) before the loop can declare its gate reached (eugenelim/agent-ready-repo, RFC-0051, Jun 2026).
The gate asks: for every scope item, does a corresponding artifact exist (test, goal-check, visual proof)? If not, the loop is incomplete — it has missing coverage, not just failing tests.
The self-coverage check differs from test pass/fail: - Test failure → implementation is wrong; retry - Coverage failure → the verification layer itself is incomplete; the loop must write the missing check before it can exit
Implementation: the gate is enforced as a done-checklist refusal inside the loop controller's own context — the loop may not declare itself done until the disposition record exists and every fresh-context finding is resolved. RFC-0051 is explicit that this is doctrine, not a runtime lock: "Make it non-skippable by a controller-gate + checklist-refusal mechanism — doctrine + a mechanical coverage record, not a runtime lock."
Traceability-lint — a related gate checking that every output artifact carries a traceable chain from scope item → task → implementation → verification → done evidence. A traceability-lint failure means the evidence chain is broken: the artifact exists, but its link to the original scope requirement is missing. Invoked in one of two sanctioned ways — as an agent-run finish-time checklist step, or as an explicit CI gate. It walks the nine-node chain (outcome → opportunity → capability → screen → action → service → contract → spec → component) and flags structural orphans: nodes with no producer above them or no consumer below them.
(eugenelim/agent-ready-repo, RFC-0048/RFC-0051, Jun 2026.)
A Sustained Finding Is Not Proof the Fix Is Right¶
Two adjacent lessons on reviewing the reviewer, from the same source as the RFCs above. Adjudicate, then mutation-prove: an independent re-review of a prior fix returned five findings; four were sustained on adjudication, and each accepted fix was still mutation-tested before landing — sustaining a finding is not itself proof the fix that addresses it is correct. Verify a reviewer's own fix against the code, not its stated intent: re-reviewing a landed fix found its new early-return branch discarded every finding accumulated earlier in the walk, not just the one case it targeted — a fix can regress prior-good behavior while looking, from its own commit message, like a narrow correction. (eugenelim/agent-ready-repo, Sep 2026.)
A verification-chain bug in the same family: a failed prerequisite step could still supply passing evidence to steps that depended on it. The fix adds a demotion pass that retroactively fails every downstream step resting on a since-failed prerequisite — closing a gap where a chain's later "pass" outlived the earlier failure it depended on. (affaan-m/ecc, commit 8b951d3, Sep 2026.)
A design tension worth flagging rather than resolving: continuing on an unverifiable
check. A freshness check in the same repo replaced a binary success/failure result with a
three-state one (ok / skipped / surface), so "temporarily unavailable" no longer reads
as "genuinely wrong." The loop then continues on skipped rather than failing — the
opposite of this KB's own rule that a check which cannot tell must fail, never pass (see
CLAUDE.md). The difference may be legitimate: a base-freshness probe that is merely
unavailable is not the same class of failure as a check that ran and found a problem, and
treating every transient unavailability as a hard stop would make the loop fragile to its
own infrastructure. But the resolution is not free, either — recorded here as an open
tension rather than a settled pattern to copy.
(eugenelim/agent-ready-repo #1449, Sep 2026.)
A repair round can no longer borrow a prior round's dispatch record. A related fix scopes dispatch accounting to the individual repair round that produced it, so a previous round's dispatch record can't silently satisfy the current wave's requirement — adding a "loop-cohort wave reopen" command with guarded transitions that refuse to proceed while live dispatch records remain. A verification-loop/maker-checker pattern for stopping a stale approval from carrying a repair cycle forward. (eugenelim/agent-ready-repo, Sep 2026.)
Oracle Problem in AI-Generated Tests¶
When the same agent writes both code and tests in the same session, tests exhibit very low precision — they verify what the implementation does rather than what it should do.
This is the oracle leakage problem: the agent uses its knowledge of the implementation
to construct tests that are tautologically true. A test for add(2, 3) that expects 5
is a valid oracle; a test for process_record(x) that expects the same output as the
function currently produces is a tautology, not a contract.
Mitigation: the critic or verifier agent must explicitly guard against oracle leakage — checking that tests would still fail if the implementation returned a semantically different result, not just a different bit pattern.
(JeiKeiLim/tenet, Jun 2026.)
A concrete authoring gate against oracle leakage, from a live production skill.
OpenClaw's test-audit skill gates every new or changed test at write time with four
questions, and "a missing answer means do not add it yet":
- "What observable behavior, invariant, or independent contract does it protect?"
- "What credible regression makes it fail?"
- "Why does existing coverage not already catch that failure?"
- "Does it need a production seam (export, flag, wrapper, injection hook) that no production caller needs?"
For bug fixes the gate adds a rule a reviewer or agent can check mechanically: "Bug regression tests must fail on the pre-fix code for the intended reason and pass after the owner-boundary repair. A regression test that never demonstrably failed proves the mock, not the fix." The same file lists 15 junk patterns — test shapes plus test-only production seams. The gate rejects a new test that matches one, and audit sweeps hunt for existing ones; among them are "assertion-free coverage probes", "self-comparisons and identity copiers" and "mocks that implement the asserted behavior". A separate campaign mode applies the same bar to one whole subsystem's tests. Steinberger reports OpenClaw deleted "around 400k LOC of its own tests without much change in code coverage" and says the skill helped (@steipete, Sep 2026). That is the maintainer's own report, not an independent measurement.
(openclaw/openclaw, test-audit/SKILL.md at commit 80930af, Sep 2026.)
Formal Models as a Bug-Finding Loop¶
Boris Cherny, a Claude Code engineer at Anthropic, opened with a concession on the word itself: "'Verify' is a little loose yeah." (@bcherny, Sep 2026.) He restated the same method six hours later, again short of claiming full verification: "It's not that the whole codebase is formally verified (yet!..), more that the hairiest parts of the code are modeled, checked for counter-examples, and fixed." (@bcherny, Sep 2026.) The method itself, in his words:
- "Building a model of the program, targeting a tricky state machine or race-prone part of the code"
- "Finding counter-examples in the model. These are suspected bugs"
- "Reproducing the bugs"
- "Fixing the bugs in the code"
He reports applying it to the Claude Agent SDK: "I used Opus 5.5 to formally verify the
Claude Agent SDK using Lean. A couple short prompts = 16 PRs fixing various bugs and race
conditions." TLA+ is a second tool in the same loop, "to look for issues around data flow,
concurrency, and state mgmt," and he adds: "I don't know either language well, but Claude is
excellent at both."
(@bcherny, Sep 2026.) A reply to that
post attaches a model-produced infographic (captioned only "Opus made an infographic") whose
figures appear nowhere in text: 16 pull requests (5 merged, 11 open); 24 bugs fixed (19 found
by the proofs, 5 more in review, plus 6 gaps in the first fixes caught in review); 6 Lean
models; 1,529 theorems; 0 sorry.
(@bcherny, Sep 2026.)
This is a first-party self-report by an Anthropic Claude Code engineer about Anthropic's own SDK, not an independent result, and the infographic's tallies are a model-generated summary at post time (Opus produced the image), not an independently counted one.
Structured Critic Finding Taxonomy¶
A critic that produces a binary pass/fail verdict cannot be triaged, routed, or tracked over time. Critic output should use structured finding categories:
| Category | Meaning | Downstream action |
|---|---|---|
product_bug |
Incorrect behaviour in the feature being built | Block merge; loop must fix |
test_bug |
Test is wrong, not the implementation (oracle leakage) | Fix the test; loop continues |
harness_bug |
Issue is in the loop infrastructure itself | Pause loop; human fixes harness |
evidence_mismatch |
Verdict claim not supported by submitted evidence | Reject; require evidence |
contention |
Two concurrent loop changes conflict with each other | Route to coordination layer |
scope_conflict |
Change touches paths outside the declared scope | Reject; loop must re-scope |
All six categories are retryable in some form in tenet; there is no non-retriable handoff
verdict. harness_bug is retried or remediated with scope limited to build/CI/scripts, and
contention is retried from report scope with a context steer — either one escalates to a human
only when the source job is report-only, or when contention recurs after a readiness re-check.
(JeiKeiLim/tenet, Jun 2026.)
This confirms the compound probability argument in The Paradigm Shift: the verification chain converts per-step model accuracy into end-to-end reliable output.
The Non-Probabilistic Node Rule¶
A concrete design rule falls out of the loops-vs-graphs cost debate: put at least one non-probabilistic node (a compiler, type-checker, linter, or test run — something that returns the same verdict on the same input every time) on every critical verification path, not just LLM judges. The reasoning is the same one behind Verifier Integrity's cross-model independence pattern, sharpened with a cost argument: **parallel same-model reviewers are correlated, not independent — the article's own framing is that a fan-out of identical reviewers "is not an ensemble. It is a chorus" by default. Three instances of the same model reviewing the same output in parallel can share the same blind spot the way three microphones on one melody add volume, not information, and that redundant spend often costs more than a single well-verified sequential pass. A deterministic node breaks the correlation entirely, because it has no model-specific blind spot to share. (LinkedIn — Rubén Domínguez Ibar, Sep 2026.)
Oct 2026 corroboration: the oracle is what makes the agent's work checkable. Several Oct 2026 pieces make the same point from different angles, none with a measured rate:
- Andrew Ng argues LLMs are far stronger on verifiable tasks (automatic reward checks) than on judgement tasks with no ground truth, and that test sets for verifiable tasks make progress look better than it is. For a loop, the split decides which steps can carry an automatic stop condition and which still need a human.
- Addy Osmani's Agentic Verification frames quality as loops the agent can run itself: E2E tests as the "done" oracle, property-based tests, differential testing against the old system, and fast deterministic loops (slow or flaky ones teach the agent to retry rather than fix). His The Code Nobody Reads (carries a sponsored block) names the independence of the verifier — a spec or reference implementation, not a second copy of the same model — as the source of trust once line-by-line review goes.
- Two reference-oracle cases: Christian Schaller's open-source hw-harness lets Claude run a Windows driver in a KVM VM, capture USB traffic and compare traces against a Linux driver — an agent given that oracle succeeded where earlier unaided attempts stalled (The New Stack); and an explainer (secondary source) gives an eight-step reverse-engineering workflow that treats differential-test mismatches as findings.
- The counter-case: a 10-patch Claude Code-assisted Linux page-cache series from Linus Torvalds (23x throughput on a synthetic benchmark) was then found by human reviewers to contain a race (The New Stack) — human review remaining the verifier where no oracle covered concurrency.
- Not-reading-the-code strategies come with constraints attached: Robert C. Martin's approach, as discussed on the ploeh blog, surrounds agents with unit and gherkin tests, QA procedures, mutation testing and coverage rather than reading the output; the post notes Martin has since walked back some harness use and raises liability questions.
Benchmark and Eval Integrity (Sept 2026 corpus)¶
Several independent findings converge on the same warning: a claimed result is only as trustworthy as the eval that produced it, and evals themselves fail silently far more often than practitioners assume.
- ~30% of a widely-used benchmark was broken. OpenAI audited SWE-Bench Pro and found roughly 30% of its 731 tasks were unsound, withdrawing its recommendation of the benchmark entirely — the oracle problem above, at the scale of an industry-standard eval rather than one AI-generated test file. (OpenAI, "Separating signal from noise in coding evaluations", Jul 2026.)
- A headline benchmark score depended on an undisclosed harness, not the model. OpenAI's 98.6% ARC-AGI-3 score for GPT-6 Astra came from an undisclosed evaluation harness/system wrapped around the model — a harness OpenAI does not sell to developers — not from the raw model under the settings a developer could reproduce. This is the eval-integrity mirror of Harness Patterns' quantified harness>model corpus: the same fact that makes harness investment worthwhile (the harness dominates the score) is exactly what makes an undisclosed harness a misleading benchmark claim. (The New Stack, "OpenAI will sell you Astra, but not the system that scored 98.6%"; The New Stack, "GPT-6 Astra's score of 98.6% looked like AGI. Then researchers read the fine print.", Sep 2026.) The benchmark itself is explicitly designed around this gap: ARC-AGI-3 scores action efficiency on novel, turn-based environments with no explicit instructions or win conditions given, specifically to penalize brute-forcing — as of March 2026, humans solve 100% of its environments while frontier AI systems solve under 1%, a far starker gap than the model-generation benchmarks this KB usually cites. (ARC Prize Foundation, arXiv 2603.24621, Apr 2026.)
- A rubric induced before execution, not graded after it. AutoSciRub builds a task-specific, executable rubric before a research agent runs — decomposing a vague instruction into atomic, literature-grounded goals — then uses that rubric both to guide execution and to grade the result, rather than writing the grading criteria after seeing what the agent produced. Reported gains: +2.08 points across three backbone models under a fixed harness, +2.95 points across three different harnesses under a fixed backbone, and +16.8 points on a research-discovery benchmark subset — three separate comparisons, not one continuous range. The design principle generalizes past research agents: writing the verifier before the run, from the spec rather than from the output, is the same discipline Goal Engineering's GOAL.md schema applies to coding tasks. (arXiv 2608.31076, "Learning to Evaluate Before Improving", Aug 2026.)
- Early-trajectory confidence does not predict failure — only late-trajectory confidence does. On long-horizon research tasks, verbal self-reported confidence reliably distinguishes a failing run from a succeeding one only at the very end of the trajectory (mean AUROC 0.85 at completion); every uncertainty signal tested stays below AUROC 0.60 at the halfway point. The proposed cause is "path switching" — agents frequently abandon their current approach mid-run, which decouples an early confidence reading from the eventual outcome. Practical implication for a loop's own stopping condition: a low-confidence mid-run signal is not yet reliable evidence to abort early — save an uncertainty-based abort decision for the run's final step, or rely on an external verifier instead of the agent's self-reported confidence for early intervention. (arXiv 2608.29685, "Last Step Matters", Aug 2026.)
- Eval gates belong in the product, not just the test suite. Repeatable evaluation gates, fixed test scenarios, and execution-path tracing need to be built into the product itself so agent reliability holds across releases — treating eval as a one-time pre-ship check rather than a standing gate is how regressions ship silently. (The New Stack, "AI agent evaluations are part of the product", Sep 2026.)
- DISH — making safety audits harder to detect. DISH (Deployment-Imitating SWE-Agent Harness) wraps a target model in an agent harness such as Claude Code so its system prompt, tool definitions and system reminders match deployment. Paired with "critique refinement", it addresses evaluation awareness: a capable model that can tell it is inside a safety audit behaves differently, which weakens what the audit can conclude. This is a different class of problem from the Astra story above — that one is a benchmark score depending on an undisclosed harness; this one is about making a safety test indistinguishable from deployment. (arXiv 2609.02302, "Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffolds", Sep 2026.)
- A consumer-facing instance of the same discipline. A "Loop Method" guide for improving ChatGPT workflows runs three improvement loops with a panel of sub-agents adversarially reviewing each cycle, identifying the biggest weakness and validating against success criteria before proceeding — the same maker/checker discipline this doc documents for coding agents, appearing independently in a non-engineering product context. (therundown.ai, "Use the loop method to get better results from ChatGPT", Aug 2026.)
Measuring Whether a Guard, Gate or Judge Works (Oct 2026)¶
Benchmark and Eval Integrity is about whether a score can be trusted. This section is about the narrower, more common question a loop builder faces: does the guard I added actually change outcomes, and is the judge I use failing safe? Oct 2026 produced unusually candid worked examples.
A pre-registered eval that measured nothing, then one that did not confirm. ECC's GateGuard is a fact-forcing PreToolUse hook gate. Its first pilot (72 sessions; off / gate / minus-target / placebo arms) returned 18/18 in every arm — a null that carried no information, for three reasons the write-up names: every trap hid its deciding fact in the repo, the asking channel was disabled, and the pass criterion used the gate's own vocabulary. It did establish the gate's cost (+45% turns, +28% tokens). The redesign uses hidden intent, a simulated user, a hole-class taxonomy, and grader checks run before any session is billed. The pre-registered confirmatory run then did not confirm the suggestive exploratory result: gate 25/48 vs off 18/48, exact McNemar p=0.210 (the exploratory run had shown +29 points at p=0.065; the replication showed +15). It was reported as not confirmed with no pooling, a follow-up showed an added question "helped" only because the ungated baseline fell, and the doc concludes that obtaining the deciding fact was the only reliable predictor of passing — the gate's one stable number is its cost (35–45% more turns/spend across 336 billed trials). Transferable method: (1) check the eval can fail before spending on it (ceiling arms measure nothing); (2) never score with the guard's own vocabulary; (3) replicate a suggestive result before adopting it, and report the miss. Compare the noise-floor discipline in A/A Baseline.
Judge discipline in caliper. The skill-eval harness caliper
shipped three releases that read as a checklist for any LLM judge.
v0.18.0: a judge that hits its spending
cap stops the run (exit 2) instead of marking every remaining attempt judge_error, and a malformed
judge answer (e.g. "passed": "false" or {}) is never read as pass or fail; specs that silently
skewed scores (duplicate YAML keys, blank assert:) are refused.
v0.16.0: the judge now follows the
--model backend by default, with the notes recommending a judge from a different model than the
one evaluated because a model grading its own output can be biased; compare warns when two runs
used different judges, and saved runs keep enough to rebuild the judge prompt.
v0.19.0 adds history: so a task
resumes from planted real-session turns instead of a simulated user (the author rejects an
LLM-played user as unrepresentative) and records cli_version/judge_cli_version per run. Note the
contrast with GateGuard's redesign above, which adopted a simulated user: how to supply the
conversation is an open design choice, not a settled one.
Evaluating loop gates themselves (InjayTseng). The loop-engineering-on-product kit's eval series (record every call's cost, label shipped commits from git, seeded gate cases with known answers, replay, whole-loop A/B) starts from a limit: git labels cannot rank gates when the human merges the whole branch (39 of 39 decided changes merged on the first adopter), so a bench script runs the real value-critic and validator on cases with fixed answers. Self-reported results include Sonnet-to-Haiku gates costing more ($1.27 to $2.37), with quality differences the author's own intervals show as overlapping, and a prefilter rejecting 18 of 32 bad ideas and 0 of 12 good ones. A later commit prints 95% Wilson intervals beside every rate and flags deltas inside overlapping intervals; a Sonnet validator failed 9 of 9 changes that keep their CLAIM but break the next flow (and passed the correct version 3 of 3), so the proposed guardrail lines were not added to the PRD template — the cases were kept as regression tests. These are small-n, single-author numbers; the transferable part is the order: measure a loop change before adopting it. Its v3.1 release and lessons 11–17 (from an overnight iOS-plus-backend adoption; each fix shipped with a test that fails on the old code) add failure modes: a usage limit mistaken for a structural failure, a veto that must read every channel the checked thing reports through, tests writing to the data they are later judged by, and "another judge agreed" is not ground truth. The corresponding fix requires the validator to name the specific test covering its claim, since PASSes backed only by code reading, a test count and screenshots were unsupported.
Anthropic's eval-design post. Automating eval design and hillclimbing with Claude
(Sep 28 2026) lists principles — headroom below 100%, low run-to-run variance, adversarial sampling
— and a one-change-at-a-time hillclimb (/claude-api build-eval and hillclimb) with a held-out
test set: if the train score improves while the test score stays flat, Claude suspects overfitting
and reverts the patch. That revert-on-flat-held-out rule is a ready-made stop condition for any
loop that tunes its own prompts; it is the same keep-only-if-the-score-rises rule Anthropic applies to
its own reviewer above.
Two measurements of the harness layer. Complex Agents, Shallow Tests (arXiv 2610.04921, Oct 2026) is the first empirical study of harness test adequacy and reports under half line/branch coverage for LLM-dependent harness code, proposing HarnessTester to improve it; AgentBug-Smith (arXiv 2609.37864, Sep 2026) automatically reproduces real harness bugs from open-source agentic systems, yielding Live-Harness-Bench with 200 reproducible harness bugs. Both are abstract-level reads: they say the loop's own scaffolding is thinly tested and that its bugs are reproducible, not how to fix either.
A vendor-run benchmark ranks the vendor's product first. GitHub and Microsoft's ReviewBench (219 PRs, 187 repos) puts Copilot code review first at 40.1% grounded F1, with Claude Sonnet 5 classifying findings; The New Stack notes an independent benchmark tells a different story. Read it as the review-tool counterpart of the undisclosed-harness Astra score above, and as another reason to run a reviewer against your own incident set, as Anthropic does.