Retrospectives

ℹ️ Version numbers below reflect the convention in use when this was written. Unbuilt work is now named, not numbered (STATUS.md). This record is left as it was.

One section per increment of the project's build plans (plans/), written during that increment's retrospective review (the workflow is in CLAUDE.md). Only findings worth keeping land here: a real defect the review caught, or a fact that would be expensive to rediscover. Fixes themselves live in the commits; this file records what was learned.

Every heading and claim here carries YYYYMMDD HH:MM (local, 24h) — several increments can land in one day, and a bare date loses their order.

Severity follows the design review's scale: HIGH — wrong behaviour or false confidence; MEDIUM — would block or mislead; LOW — worth remembering, not urgent.

The seven pre-implementation design review passes are at the foot of this file: Design review passes 1–7.

Start here — by what you are about to touch

Added 20260801 01:11. Forty-odd sections in date order is an archive, not something anyone reads before starting work — so this table is the way in. It is keyed on what you are about to do, not on when the lesson was learned, and it is deliberately short: only the classes that have recurred. A rule that hardened into a standing instruction lives in CLAUDE.md or BUILDING.md; this file is where the evidence for it is.

About to…	Read	Because
write a test	I2, L3–L4, L5, L5b	The recurring defect of this whole project: an assertion satisfied by something other than the property it names. A test that could not fail; a field with no assertion is a field that can be a constant; a tidy fixture defeats a mutation test; test the discriminating case, not the two sides separately; a fixture can be right for the wrong reason and hide the defect it was written for
claim a test is mutation-verified	L5b, L3–L4	"Mutation-verified" is a per-assertion claim, never a per-commit one. A failing test proves the mutant is caught, never that it is caught for the stated reason — under -x, or when the failure lands on an earlier assertion, the one encoding the claim never runs
touch a sidecar or any YAML	I5, L5b	Writes must be rename-atomic. An explicit empty value was silently deleted on round-trip. Swapping a YAML library is not a swap; existing[:] = keep wipes ruamel's comment metadata outright; a comment before a sequence entry belongs to the entry above it; a warning is not an error, and a library that downgrades one is changing behaviour
add a gate or a check	I7a, L3–L4, the eval-harness section	A gate that has never been shown to fail is a claim, not a check. A gate that never reads the artifact it guards is checking a copy. The free-path gate was defeated by its own harness. The exit criterion was the thing nobody ran
write an error message	L3–L4, I8	An error message is part of the interface; a remedy inside one is a claim, and it was false. A fix applied to one surface is half a fix
change retrieval	I9, the eval-harness section, I6	Three defects under one green suite. Overlap could push a chunk past max_tokens; heading text landed in no chunk at all
edit a doc, a plan or an exclusion list	L5b, I9, the shared-file section	An exclusion list is a set of claims, and claims rot. A fix instruction can carry its own defects. Four silent str.replace no-ops in one session — an edit that does not match is an edit that did not happen. A clean auto-merge is not a correct merge
cut a release	the post-v0.1 housekeeping section, the two release-sweep sections (20260811)	A CHANGELOG entry and a __version__ are only claims: 0.1.0 had both for two days with no tag, no release and nothing published. Verify with git tag -l, gh release list, and the index itself. A release sweep is table-shaped and misses prose — grep for the superseded version number — and ordering is a property of the sequence: read the sequence, not the neighbourhood
trust CI	the cross-platform scanned-fixture section	main was CI-red for three pushes and nobody noticed. Green proves the tests ran, never that they can detect the defect
run a mutation pass	G1, L6 § Mutation testing, T3, T7, E5	Commit before mutating — git checkout restores to the last commit and has silently reverted uncommitted fixes six times. A harness must prove it can kill before its silence means anything: assert the anchor matched exactly once, clear __pycache__ between mutants, run without -x, restore from a copy. A killed run poisons everything after it, and a surviving mutant first asks is this reachable at all?
write a measurement tool, or publish a number	the reachable-ceiling probe, measure_sync_cpu.py, E2, E6	A tool that absorbs input it cannot measure reports a plausible number — strictly worse than a crash. Validate everything the measurement consumes, not the bugs already reported; the artifact names every input its numbers are a function of; one test runs the tool the way it is really invoked, because the shipped one measured the launcher and not the work. A probe is code and earns the same adversarial pass — two of five constants were measured wrong on E6's first run
lean on a fake, a seam or a fixture	E6, I7b (passes 6–7), the [light] backend error, Running it found what reading it could not	A seam built for testability defines a region no test reaches, and that region needs its own gate: the deep client's schemas 400d on every live call for four releases of green fixtures. "Each part works" is not "the parts are connected". For any new knob, turn it on a real KB before landing — only running proves the parameter arrives
apply a review fix	T3, T4, L6, 2d round two	A fix applied under review inherits the review's confidence and none of its scrutiny — every review loop here has found its worst defects inside the previous pass's fixes. "Pinned by test X" is a claim about a failing test: revert the fix and watch it go red, or do not write the word. Re-run the whole battery after every fix — a fix can silently disarm a test written for something else

The pattern across all of them, and the reason this file is worth keeping: the worst finding of a review pass is usually inside the previous pass's fix. That has held from I5 through L6's thirteen rounds, and it is why a fix is re-reviewed rather than assumed.

I1 — Package skeleton, errors, CLI dispatch (20260725 13:40)

MEDIUM — PinakesError could not be pickled, so an error crossing a process boundary raised TypeError instead of reporting itself. Exception.__reduce__ replays self.args through type(self), but every subclass here takes its own constructor arguments (NotImplementedYetError takes a command name and an increment), so rebuilding blew up on the missing remedy keyword. Confirmed by probe before fixing, not reasoned about. Fixed with an explicit __reduce__ routing through a module-level helper. Lesson: an exception class with a non-(message,) constructor is unpicklable by default — the failure only surfaces under xdist/multiprocessing, i.e. exactly when something else has already gone wrong.

MEDIUM — the subcommand dispatch target sat on the public namespace attribute run. Any future command declaring --run would have silently overwritten the function main() then calls. Moved to a reserved _runner dest with a test asserting no public namespace attribute is ever callable. Lesson: set_defaults shares one namespace with every option; anything the framework itself dispatches on must be underscore-reserved.

LOW (reference) — measured ty 0.0.63 against pyright strict on a 6-defect probe. pyright caught 6/6; ty caught 1/6 (the str | None → len error, with better diagnostics); ruff caught the unused import. ty currently has no strict mode: it accepts unannotated defs and Any leakage, which is precisely what pyright strict is in this project for. Decision (user): keep pyright as the gate, add uv run ty check as a fast pre-check. Worth re-measuring when ty leaves beta — the gap is a missing feature, not a design difference.

I2 — ULID identity and pnk:// URIs (20260725 14:05)

MEDIUM — a test that could not fail. test_an_unresolved_uri_cannot_be_formatted asserted not hasattr(parsed, "__str__") or "pnk://" not in str(parsed). Every object has __str__, so the first clause is always false and the second only checked a dataclass repr — the test would have passed even if ParsedUri had grown a full URI renderer. Replaced with a precise structural assertion ("__str__" not in ParsedUri.__dict__, present on PnkUri) that names the static guarantee as primary. Lesson: a green test asserting a tautology is worse than no test — it buys false confidence, and hasattr(x, "__dunder__") is almost always one.

MEDIUM — a docstring claimed more than had been verified. ids.py said python-ulid rejects the ambiguous Crockford letters I, L, O, U; only I and U had actually been probed. All four are now probed and the claim is stamped with the time of that probe. Lesson: when writing "verified X", the set being claimed must be the set that was run — a partially-probed claim reads identically to a fully-probed one.

MEDIUM — two except Exception blocks wrapped calls whose only expected failure was InvalidIdError, so a TypeError from a future refactor would have been re-raised as "this is not a valid KB ULID". Narrowed, and ruff's BLE ruleset enabled so the class of mistake cannot be written again.

LOW — internal helpers were public. parse_kb_id_for_uri/parse_doc_id_for_uri took an odd (raw, segment) pair and had no business in the module's API; renamed to _kb_segment/_doc_segment.

LOW — the scheme is matched case-sensitively while the self sentinel is not. Deliberate (URIs are machine-written, self is hand-typed) but undocumented; now stated in the module docstring and covered by a test.

I3 — Manifest parsing and KB root discovery (20260725 14:25)

MEDIUM — an explicit empty value silently became the default. timezone = "" in [budget] read back as "UTC", because the accessor ended in or "UTC". Same shape accepted name = "" and model = "" outright. All three confirmed by probe. Empty strings are now rejected with a named key, and the default only applies when the key is absent. Lesson: value or default conflates "missing" with "empty", and for user configuration those mean opposite things — one is silence, the other is a mistake worth reporting.

MEDIUM — narrowing by assert. Two call sites used assert value is not None to convince the type checker, and a _require fallback built a Path out of a table name for an error that could never fire. Python strips asserts under -O, so the "guarantee" was a comment with syntax. Replaced with three explicit accessors — string (required, returns str), optional_string, string_or — which give the type checker what it needs without a runtime claim. Lesson: when a type checker needs an assertion, the API shape is usually wrong; fixing the signature beats asserting.

MEDIUM (docs) — the required/optional split existed only in code. [chunking], [retrieval], [rerank] and [budget] are optional with documented defaults while [kb], [sources] and [embedding] are mandatory — a user-facing contract that docs/DESIGN.md §2.1 never stated. Added there in the same change, per the repo's docs rule.

LOW — a test caught a real error-message defect. Validation errors read [<root>.retrieval]; the table path is meant to name what the user would type. The failing test was fixed in the source, not the assertion.

I4 — SQLite storage, FTS5 triggers, vector loading (20260725 14:45)

MEDIUM — load_vectors peaked at roughly twice the array it returned. It collected every embedding into a list and vstacked it: measured 669 MB for 200k×384, where the result is 307 MB. At 1M chunks that is ~3.4 GB against the ~1.5 GB §3.1 states, so the design's own memory claim was wrong for its only shipping tier. Now counts first and fills a preallocated array, with a test asserting peak < 1.6× the result. Lesson: "load it all into one contiguous array" has two implementations that differ by a factor of two, and only one of them matches what the design promises.

MEDIUM — a non-database file produced a raw sqlite3.DatabaseError. PRAGMA journal_mode is the first statement to touch the file, so it failed before any schema check could run, and the user got file is not a database with no remedy. Opening is now wrapped. Lesson: the friendly check ran second; the pragma that configures the connection is what actually opens the file.

MEDIUM — chunk insertion restarted ordinals at 0 on every call, so a second call for the same document hit the UNIQUE (doc_id, ordinal) constraint. The operation is really a wholesale replace — re-chunking must not leave old chunks searchable — so it now deletes first and is named for that. Both this and the one above were caught by tests written in the same increment.

MEDIUM — pickling collapsed every error subclass to PinakesError. I1's __reduce__ fix rebuilt through the base class, so StoreError came back as PinakesError and any except StoreError on the far side would miss it. Now rebuilds the original class via __new__, keeping message and remedy. Lesson: a fix that makes an object survive a round trip is not the same as one that makes it survive intact — check identity, not just contents.

LOW — DOCUMENT_STATES/LINK_ORIGINS duplicated the DDL's CHECK constraints with nothing tying them together. A test now fails if they drift.

I5 — Sidecars (20260725 14:55)

HIGH — sidecar writes were not atomic. path.write_text truncates before it writes, so a crash or full disk mid-write leaves a truncated sidecar — and the one thing a sidecar carries that cannot be recomputed is the document's permanent ULID. Losing it breaks every inbound pnk:// link, and no later command can repair it, because nothing else knows what the id was. Now writes to a sibling temporary and os.replaces over the target. Lesson: "the truth is in files" makes every file write a durability question; the ones holding unrecoverable identity deserve rename-based atomicity.

MEDIUM — an explicit empty value was silently deleted on round-trip. tags: [] and provenance: {} vanished, because write tested truthiness. This is exactly the lesson I3 recorded one increment earlier — absent and empty are different statements — and I repeated it in a module whose entire contract is "do not lose what the user wrote". The Sidecar now records which known keys the file carried. Lesson: a recorded lesson only helps if it is re-read while writing the next thing that could break the same way; the pattern to watch for is any if value: guarding output.

MEDIUM — a hedged test assertion. assert made.title == "my research notes.md" or made.title == "my research notes" accepted either answer because I had not checked which one Path.stem gives. Two plausible values means the test asserts nothing about the one that is correct. Pinned to the real value. Lesson: an or in an equality assertion is the tautology smell from I2 wearing a different hat.

LOW — KNOWN_KEYS could drift from what write emits, which would be silent data loss for a key the module claims to understand. A round-trip test now asserts every known key comes back.

I6 — Structural chunking (20260725 15:10)

HIGH — overlap could push a chunk past max_tokens. The carried-over tail was prepended unconditionally, so overlap = 9 with max_tokens = 10 produced 12-token chunks (probed). Those are exactly the chunks the model truncates at encode time, silently — the failure §4.6 exists to prevent, reintroduced by the feature meant to improve context. The carry is now dropped when it would breach the limit, and a parametrised test sweeps the whole (max_tokens, overlap) matrix. Lesson: the earlier test used overlap=5, max_tokens=15 and passed; one comfortable ratio proves nothing about the boundary. Sweep the matrix when two parameters interact.

HIGH (design, found by a test) — heading text landed in no chunk at all. Headings were consumed as pure structure, so a word appearing only in a heading was unsearchable: the FTS index sees chunk text and nothing else. heading_path looked like it covered this, but it is a separate column that v0.1 never searches. Headings are now part of the first chunk beneath them, and docs/DESIGN.md §4.6 says so. Lesson: "the information is still recorded somewhere" is not the same as "the information is still retrievable"; check which column the query path actually reads.

MEDIUM — sentence splitting silently gave up on text without punctuation. A long run with no .!?;: produced one oversize piece that was emitted whole. Now falls back to words, then to characters for a genuinely unbroken run (a hash, a base64 blob).

LOW — piece offsets came from a running total that finditer's empty end-of-string match could desynchronise from the source. Now taken from match.start(), so spans are exact by construction rather than by accounting.

LOW (test-design) — a stand-in counter can be wrong in the direction that hides the bug. The word-counter says a 400-character unbroken run is one token; every real tokenizer disagrees. The character-cut path needed a token-dense counter to be exercised at all.

I7 — Embedding backends and reranker (20260725 15:35)

MEDIUM — a fake that could never disagree. The test backend was registered as FakeBackend(section.dim), so it reported whatever width the manifest claimed — making the dim-mismatch check, the one guard against storing incomparable vectors, impossible to test. Pinned to a fixed width. Lesson: second time in three increments that a stand-in was wrong in exactly the direction that hides the bug (I6's word-counter was the first). A fake that derives its answer from the input under test asserts nothing.

MEDIUM — an assertion guessed at a real model's behaviour. count_tokens("retrieval augmented generation") > 3 was written before any model had been run; the real BPE count is exactly 3, and the test failed the moment weights were cached. Rewritten to assert the relationship (longer text → more tokens) rather than a number I had invented. Lesson: when a test crosses into a real dependency, assert properties, not remembered values.

Verified against real weights (not inferred): fastembed's BAAI/bge-small-en-v1.5 gives dim=384, max_seq_length=512 derived from the tokenizer's truncation config, normalised float32 vectors (self-cosine 1.0), and it wrote to ~/.cache/huggingface/hub rather than $TMPDIR/fastembed_cache. BAAI/bge-reranker-base ranked a relevant passage above an irrelevant one, -0.28 vs -7.85.

LOW (reference, matters for I14) — reranker scores are raw logits, not probabilities. They came back around -0.28 and -7.85, not in [0, 1]. The illustrative thresholds in §2.1 (low_below = 0.31, high_above = 0.62) read like normalised scores; calibration must either fit against the logit scale or squash it first. Recorded now so I14 does not quietly fit thresholds to the wrong scale.

I8a — Sync pairing core (20260725 15:50)

MEDIUM — a model-test guard checked the wrong half (found in I7, surfaced here). The model-marked tests skipped when weights were absent, but not when the backend was absent. Model weights live in a shared machine-wide cache, so a worktree without the light extra installed still saw them, ran the test, and failed with BackendMissingError instead of skipping. Only noticed because a second worktree had a different install set. Now both halves are checked. Lesson: a skip condition is a claim about the environment, and machine-wide state (a shared cache) is not evidence about the local one.

Design note, not a defect — DuplicateIdsError raises rather than returning an action. plans/20260725_1317-v0.1.md lists it among pair()'s return values. Raising is better: the condition is fatal for the whole run, and an action every caller must remember to inspect is one a caller will eventually forget. Recorded because it is a deliberate divergence from the reviewed plan.

What the exhaustive table bought. Writing one test per §6.4 row, then the compound cases the table cannot express, is what forced the two decisions the design left implicit: a sidecar disagreeing with the index wins (docs/ is truth, the index is derived), and a whole-picture rule must be order-independent — asserted directly by pairing the same snapshot walked forwards and backwards.

I8b — pnk sync, locking, and the rebuild swap (20260725 16:20)

HIGH — the rebuild swap left the old database's -wal/-shm behind. Design pass 2 fixed the missing checkpoint; this is the other half nobody had noticed. SQLite names the companions after the path, not the inode, so after os.replace they sit beside the new index claiming to be its write-ahead log — the exact corruption the checkpoint was added to prevent, reintroduced by the rename that followed it. They are now removed after the swap. Lesson: an atomic rename is atomic for one file; a WAL database is three files with correlated names, and correctness arguments about "the file" quietly skip the other two.

MEDIUM — a read-only SQLite connection creates -wal and -shm itself. This masked the bug above: the test read the index before asserting the companions were gone, and the read created them. Worth knowing beyond this test — §4.7 says the MCP server opens the index read-only "so it cannot write", which remains true of the data, but the server does create files in .pinakes/. Any future check that treats "no companions present" as evidence of a clean shutdown is wrong.

MEDIUM — a leaked connection inside a test helper. The helper was a generator, and a caller using next() left it suspended, so its finally: close() never ran. The symptom appeared in an unrelated assertion about rebuild. Now returns a list and closes immediately. Lesson: a generator with cleanup in finally only cleans up if it is exhausted; for a fixture-shaped helper, return the list.

LOW — ty caught a loose test shim pyright had been told to ignore. The monkeypatched __import__ took *args: object; pyright was silenced with an inline ignore, ty flagged the real mismatch. Typing the shim to __import__'s actual signature satisfied both and deleted the suppression. First time the "fast pre-check" found something the gate had been told to skip — noted, since I1's decision assumed ty would only ever be faster, not different.

I9 — Retrieval pipeline (20260725 16:55)

MEDIUM — vector search padded its candidate list with zero-similarity passages. argsort returns every row, so a query sharing no direction at all with a passage still ranked it, and with nothing better available those passages reached the user. Real models rarely produce an exact zero, so this would have hidden until a sparse or domain-shifted corpus hit it. Non-positive cosines are now dropped. Lesson: "return the top N by similarity" is only sane while similarity means something; N is a cap, not a quota to fill.

MEDIUM (design gap, decided here) — the design said the filter set included "date", and no date column existed. Documents carry mtime; a sidecar's created is optional, and filtering on an optional field silently excludes every document that lacks it — worse than having no filter. The filter is now mtime, and §4.1 says so. Lesson: when the design names a filter dimension, check which column actually holds it for every row, not just the well-formed ones.

LOW — sqlite3.Row hands back Any, which erased types through the whole hydration path. Rows are now narrowed once into a small frozen dataclass instead of being cast field by field at each use — pyright strict was the thing that made this visible.

Worth recording: FTS5 escaping is not optional. AND, OR, NEAR, *, " and a bare apostrophe are all parser syntax; a user typing it's would otherwise crash the query. Quoting each word as a literal and joining with OR keeps recall — an implicit AND drops a passage for one missing word, which is exactly the recall the vector half is there to provide.

I10 — pnk init, the notes template, pnk search (20260725 17:15)

MEDIUM — warnings were being printed and ignored. Turning filterwarnings = ["error"] on immediately produced two real problems that had been sitting in the summary: importlib.abc. Traversable is deprecated and removed in Python 3.14, so this project would have broken on the next Python it claims to support; and several tests leaked SQLite handles, surfacing as ResourceWarning raised from wherever the garbage collector happened to run — never the test that leaked. Lesson: a warning nobody has to act on is a warning nobody reads. Making them errors cost one afternoon of cleanup and bought a Python upgrade.

MEDIUM (process) — I committed with a failing gate. The last edit before committing introduced a reportUnusedFunction on an autouse fixture; I had run pyright before that edit, not after. Fixed in the retrospective commit. Lesson: the gate belongs immediately before git commit, not "recently" — and "recently" is exactly what it felt like at the time.

LOW — the test fixture violated a manifest invariant I had written myself. Setting max_tokens = 60 while leaving overlap = 64 was rejected by I3's cross-key validation. Pleasant confirmation that the check earns its place: the first thing it caught was its own author.

Design note — the template ships [retrieval.confidence] commented out. plans/20260725_1317-v0.1.md had already decided this, and building it made the reason concrete: pnk init cannot know anything about the corpus the user is about to add, so any threshold it wrote would be a number with no provenance. confidence: unknown until they fit their own is the only honest default.

I11 — pnk doctor (20260725 17:35)

MEDIUM — ty found a second real defect hiding behind a pyright: ignore. I had typed a dict as dict[Path, object] and silenced the resulting argument-type complaint rather than fixing the annotation. That is the same shape as I8b's finding, and it is now a pattern: an inline suppression is where a type error goes to be forgotten. Every pyright: ignore in src/ has been removed as of this increment; the two that existed were both hiding something real, not appeasing a checker that was wrong.

Worth stating — doctor is where several design promises stop being rhetorical. §3.1's linear scan ceiling, §6.2's link-coverage ceiling, §4.2's unknown confidence, §6.4's orphan reporting and §6.5's lock are all "we will tell you rather than pretend" commitments. None of them is honest unless something actually prints them, and until this increment none of them did. The test that says every non-OK check must carry a remedy is the one enforcing the spirit of it: a report that says "problem" without saying "do this" is just anxiety.

Design note — an uncalibrated KB is a WARN, not a FAIL. Reporting confidence: unknown is the honest behaviour the design chose, so a KB doing exactly that is not broken; it is uninformative, and the warning says so with a pointer to §4.2 and §7.

I12 — pnk install-hooks (20260725 17:50)

Confirmed end to end, with a real commit: the three-hook split does what design pass 6 claimed. docs/note.md and docs/note.md.pnk.yaml land in the same commit, and git status is clean afterwards. That was the whole argument for splitting the hooks, and it is now a test rather than a paragraph.

LOW — the pre-commit half needs no embedding backend at all, which only became obvious when a subprocess pnk ran without the test's fake registered and the sidecar half still worked. That is the right shape: minting an id is cheap and belongs before the commit; embedding is slow and belongs after it. Worth recording because it means a KB whose backend is not installed can still commit documents with correct, permanent ids — the failure is deferred to indexing, exactly where §4.5 says a core-only install should feel it.

LOW (test hygiene) — a "tree is clean" assertion failed on files the fixture never committed. The hooks were fine; the setup was. An assertion about cleanliness is only meaningful from a clean starting point.

HIGH (process) — I committed with red gates for the second time, and now understand why. The pattern uv run pyright 2>&1 | tail -1 && git commit … reports the exit status of tail, which is always 0. Both checkers were failing and the commit went through looking green. This is not carelessness that can be fixed by resolving to be careful: the shell was reporting success. Added check.sh, which runs every gate under set -e, and pointed CLAUDE.md at it. Lesson: if a safety check is a pipeline, the thing you are checking is the last command in the pipe. Make the gate a script that exits non-zero, and the mistake becomes unavailable.

I13 — pnk serve (MCP) (20260725 18:15)

The boundary is testable, so it is tested. §4.7's claim is that an agent cannot reach outside the KBs the server was pointed at. Three tests hold it: pinakes_get refuses a path, a traversal string and an unknown ULID identically; a KB that exists on disk but was not passed on the command line is unreachable and the error says arguments select by name or ULID never by path; and a document deleted since it was indexed cannot be fetched.

MEDIUM — stat()-based staleness detection works, and the test proves the thing the design argued about. After a --rebuild swap the server returns the new documents. The design's reasoning (an open handle pins the old inode, so meta.build_id read through it would report the old build forever) is now backed by a test that would fail if someone "simplified" it back.

LOW — pyright strict flags decorator-registered functions as unused. @mcp.tool() returns something pyright cannot tie back to the name, so all three tools looked dead. Rather than suppressing it, they are now registered in an explicit loop — which also makes the set of exposed tools one readable line instead of three annotations. A suppression would have been the third one this session that turned out to be hiding something; making the code say what it means was cheaper.

LOW — another leaked connection, caught by warnings-as-errors from I10. A Server built inline inside a pytest.raises block was never closed. This is the third leak that setting only found; before I10 it would have been invisible.

I14 — Demo KB, golden set, eval harness, calibration (20260725 19:00)

HIGH — the two error rates were a flattering zero, and it took looking at real numbers to see it. The first eval run reported false_abstain: 0.0 and false_confidence: 0.0. Both were vacuous: the demo KB had no fitted thresholds, so confidence was always unknown, so neither error could ever be counted. A CI gate on false-confidence would have passed forever, and passed loudest exactly when calibration was missing. Added confidence_coverage to the metrics, and made a drop in it a regression in its own right. Lesson: a perfect score on an error rate is a claim that deserves the same suspicion as a failing one — check the denominator before believing the ratio.

HIGH — the first threshold formula made low unreachable. low_below = min(answerable) came out at -9.885 on real logits, a floor almost nothing falls below, so the system could essentially never abstain and false-abstain was zero by construction. Both thresholds are now fitted from the unanswerable distribution — the only outcomes known absolutely — with low_below its median and high_above a high percentile. Only visible because the fit was run against real reranker scores rather than a fake's tidy 0-to-1 range.

The measured cost of the confidence heuristic, stated plainly: false-confidence is 0.25. One no-answer question in four still gets a confident answer, because the score distributions genuinely overlap (answerable -9.9..7.9, unanswerable -8.3..-2.7). §4.2 promised this would be measured rather than assumed; it is now in docs/DESIGN.md's risk table with its date and models. Two caveats are recorded with it: eight no-answer questions is a small sample, and the thresholds are fitted on the same set they are scored against, so it is a floor rather than a measurement.

MEDIUM — the test fixture copied .pinakes/ and ran the 64-dimensional fake against a 384-dimensional index. I4's stored-vector width check refused it, which is exactly right and briefly baffling. Generated state is now excluded from the copy. A good sign for the guard: the first thing it caught was a developer, not a user.

LOW — ruff caught assert ... or True in my own test. The tautology lesson from I2, third appearance, this time found by a linter rather than by reading. SIM222 earns its place.

I15 — CI, packaging, 0.1.0 (20260725 19:30)

MEDIUM — the version lived in two places. pyproject.toml and __init__.py each carried it, which is one place to forget on every release. Hatch now reads it from the module, and the release workflow refuses a tag that disagrees with it — a mismatched tag is the kind of thing nobody notices until an install pulls the wrong thing.

The wheel smoke test earned its place immediately. pnk init reads its template through importlib.resources, so a packaging mistake is invisible in the source tree and total after install. Running pnk init from the built wheel — and asserting the manifest, the golden-set stub and the .gitignore all appear — is the only check that would have caught it.

Publishing is left as a human step, on purpose. The release workflow runs on a v* tag and nothing else, and PyPI trusted publishing has to be configured in the PyPI UI first. Neither the tag nor the publish happens automatically from a merge: an irreversible, outward-facing action should need someone to mean it.

§8's v0.1 sentence, walked item by item: 17/17 present. The plan asked for that walk explicitly at this increment rather than trusting the accumulated sense that everything got done.

---

Post-v0.1 release housekeeping (20260727 15:35)

Not an increment — a session that merged the graph research, closed out the v0.1 plan, and cut the releases that had never been cut. Recorded because four of its findings are the kind that cost more to rediscover than to write down, and one is a mistake made in this session by the person writing this.

The 0.1.0 release existed in every artifact except the ones that matter. __version__ said 0.1.0, the CHANGELOG had a dated [0.1.0] section, and its footer linked releases/tag/v0.1.0 — while git tag -l was empty, no GitHub release existed, and PyPI returned 404. The release workflow fires only on a v* tag, so nothing had ever been built or published. Lesson: a version number is a claim, and a claim in a CHANGELOG is the easiest kind to believe. Verify a release the way a stranger would — git tag -l, gh release list, the package index — never by reading the file that asserts it.

A docs-only merge turned main red. ruff format --check . formats Python fenced blocks inside Markdown, so an igraph snippet in a research doc failed the Format gate. The instinct that a documentation change cannot break the build is wrong in this repo, and the gate is the only arbiter. Now stated in CLAUDE.md and in the README's Development section.

Merging from inside the feature worktree silently does nothing, and the tag lands off-main. Running git merge --ff-only <branch> while cwd is that branch's own worktree merges the branch into itself — "Already up to date" — and the subsequent git push origin main reports "Everything up-to-date" because the local main ref never moved. The git tag that followed pointed at a commit reachable only from the branch, so v0.1.2 existed, was pushed, and was not an ancestor of main. Both commands succeeded; nothing failed loudly. Lesson: merge from the primary checkout, and before creating a release assert the lineage — git merge-base --is-ancestor vX.Y.Z main.

The README was the only surface that lied. An audit against the running CLI found four claims contradicting the code: pnk ask --deep described as existing (it is a v0.4 plan), a budget ledger described as tracking spend (nothing writes one), install lines pointing at a PyPI package that 404s, and the headline KB diagram built on a .pdf — the one file type v0.1 cannot ingest. cli.py and the CHANGELOG were scrupulous in the same places, both saying "planned for v0.4". Lesson: prose drifts toward the design and away from the build, because the design is what the author is thinking about. The check that works is running the commands the README shows. The documented [light] install had the same shape: co-equal in the README, broken at the first sync because pnk init always stamps sentence-transformers.

A promise in a section with no increment number belongs to nobody. plans/20260725_1317-v0.1.md asked for a CI grep gate keeping paid-API clients out of src/ under "Verification of the whole". No increment owned it, so it never shipped — while the invariant it guards is the one CLAUDE.md calls non-negotiable. Now enforced, and verified in both directions: it passes on the current source and catches a planted import openai. Lesson: every promised check carries an increment number and a path, or it is a wish.

Planning v0.2 (20260727 17:00)

A review pass is a change, and a change needs its own review. Adversarial pass 2 over plans/20260727_1543-v0.2.md returned 5 HIGH — and four of the five were created by pass 1's own fixes, not survivals of pass 1's findings. Pass 1 correctly rejected a per-page cost estimate as an order-of-magnitude under-reservation against whole-document requests; the shape it introduced instead was quadratic in input and stopped fitting the model's context window at ~166 pages, so a 100-page document reserved ~$375 and the feature failed closed with a refusal no user could satisfy. Lesson: never implement from the revision that a review produced. Two passes are the floor for a document of this size, and each pass reviews the previous pass's fixes, not only the original.

A threshold that only exists in tests/ protects nobody. Both fitted floors — the one that triggers paid re-extraction and the one that stops a paid run on an already-healthy PDF — were committed to tests/pdf-corpus/baseline.json, which no wheel installs, while three runtime consumers on a user's own KB depended on them. The fail-closed rule was also stated for one floor and not the other, so the guard against paying to re-extract a healthy PDF was silently disabled everywhere. Lesson: a value a user's runtime reads is package data, and every fail-closed rule is stated and tested once per consumer, not once per document.

An append-only ledger needs a way to say "this never happened". Every paid call wrote a reservation before the call and a reconciliation after, with an unreconciled reservation counted at its reserved amount. A call that raised — timeout, 429, 5xx — therefore left spend that nothing could ever close, in a file CLAUDE.md forbids editing: a handful of transient failures would lock a user out of a monthly budget permanently. Lesson: a two-state protocol over an append-only log needs a third state. Reserve → reconcile or void.

Timestamps were composed instead of read. Four "verified on 20260727 17:34" claims were written at 17:00 — 34 minutes in the future. Session context carries a date but never a time, so any HH:MM not read from the clock is invented, and an invented one lands in the future about half the time. A timestamp exists to say how fresh a verified claim is, so a fabricated one is a false evidence claim rather than a formatting slip. Lesson: run date "+%Y%m%d %H:%M" and paste the result; one call covers a batch of edits. Now a rule in CLAUDE.md.

Planning v0.2, pass 3 (20260727 20:23)

Three review passes, and each one's largest source of new HIGHs was the previous pass's fixes. Pass 2 found that four of its five HIGHs were created by pass 1's repairs; pass 3 found the same of five of its twelve. Reviewing a plan is a change to the plan, and a change needs its own review. Lesson: budget for the review of the review. A plan is not ready because the last pass was clean — it is ready when a pass over the current text is clean.

What broke the cycle was method, not effort. Five of pass 3's twelve HIGHs were found only by checking the plan against src/ and against arithmetic — never by reading the plan, however adversarially. --rebuild discards the index and reads its "before" snapshot from the empty new database (sync.py:231-241), so a guarantee recorded only in index.db was invisible to the one command that most needed it. A trace test asserted "exactly one chunk contains this offset" while chunk.py:239-269 prepends the carried overlap and takes start from the carried piece, so chunk n+1 begins inside chunk n whenever a block splits. A filter test named tags as a column; it is json_each over a NOT NULL DEFAULT '{}' field, so the assertion was true by schema for every row and would have passed on a corpus with no tags at all. A raster gate promised "a moved word must fail" against a threshold its own arithmetic puts about four lines of text away. And the docs sweep enumerated README additions while four sentences already in the README are falsified by the release — the same defect, and the same count, as the audit at 0.1.2. Lesson: a consistency pass cannot find a claim that is internally consistent and externally false. Run three narrow passes instead of one wide one — code-reality (resolve every claim about src/ against the file), arithmetic (recompute every stated number), and promise-ledger (walk every enumerated bound, flag, floor, gate and amendment asking "which increment makes this true, and which test proves it"). Each found HIGHs the wide pass missed.

A threshold you cannot fit yet is not a threshold — ship the metric, defer the loop. Two passes each tried to fit the completeness audit's floor, and each picked a pair that was not the pair it is applied to, because the applied pair needs paid model output that does not exist at fitting time. The second attempt would have fitted over a population including a fixture the reader contract requires to raise, landing the floor near zero and leaving the audit inert on every installed copy. Lesson: when a threshold's correct fitting data cannot exist yet, that is the finding. Ship the measurement, report it, and let the release that produces real data fit the number.

"Bypasses module X" is a claim about an import graph, and it was false. The paid extractor was documented as bypassing layout.py — while calling normalise(), which lives in it. The version constant covering that module was then deliberately excluded from the paid fingerprint, so a whitespace or ligature change would have missed every free cache entry and silently hit every paid one, with the coherence check unable to see it. Lesson: when two consumers share one stage, version that stage separately rather than arguing about which of them "really" runs it.

Planning v0.2, pass 4 — three narrow passes (20260727 20:57)

Splitting one wide review into three narrow ones is what finally found the structural defects. Passes 1–3 were each a thorough adversarial reading, and each one's largest source of new HIGHs was the previous pass's fixes. Pass 4 ran instead as a code-reality pass (resolve every claim about src/ against the file), an arithmetic pass (recompute every stated number), and a promise-ledger pass (walk every enumerated flag, floor, gate, constant and amendment row, asking only "which increment makes this true, and which test proves it"). Together they returned 19 HIGH — and almost none of them were things a fourth general reading would have caught. Lesson: a reviewer reading for coherence finds incoherence. Defects that are internally consistent and externally false need a pass that leaves the document: to the source, to a calculator, or to a checklist. Run those as separate passes, because each is a different kind of attention.

A review pass's own fixes are the highest-risk text in the document. Four of pass 2's five HIGHs came from pass 1's fixes; five of pass 3's twelve came from pass 2's; and two of pass 4's came from inside pass 3's fixes — a pnk budget ordering bug fixed by assigning the edit to an increment that lands earlier, and a missing-amendment-row sweep that added a row for one copy of a sentence and missed two others. The arithmetic pass caught the sharpest case: pass 3 replaced a raster tolerance that could not detect a moved word with a different tolerance that also could not detect a moved word. Lesson: never ship the revision a review produced. Re-review the fixes specifically, and prefer a method that cannot be fooled by the same reasoning that produced them.

"The code already does X" is a claim, and this project keeps getting it wrong. Verified against src/: write_sidecar runs only for newly minted documents, so a plan built on "sync writes the sidecar" had no write path for the case it cared about; pnk init --ci was designed in DESIGN and never built, while an increment was written to modify it; CI installs --extra light and runs the model tests in the check job, so a new matrix leg would triple them rather than skip them; and --index-only, which the hooks run, is contractually forbidden from writing into docs/ at all. Lesson: every plan sentence about existing behaviour carries a file:line, or it is a guess.

A threshold needs enough data to have chosen it. The running-head threshold T was documented as fitted over a stratum of 3-page fixtures — where per-document recurrence can only be 1/3, 2/3 or 1, so every T in (1/3, 2/3] reproduces the corpus identically. The value was real, the fit was not. Lesson: state a fitted threshold's resolution alongside its value. If the corpus cannot distinguish 0.4 from 0.6, "fitted" is a claim the data does not support — enlarge the fixture or call it a chosen constant.

I1 — extras, the extractor seam (20260727 22:28)

An explicit, textual exit criterion is not met just because the surrounding tests feel thorough. The plan's own words were "the filterwarnings probe and both marker predicates get named tests" — the probe got tests; pdf_runnable()/paid_runnable() did not, and nothing caught it until the review grepped for their names and found only their own definitions. Twenty-odd other tests passing made the increment feel covered. Lesson: when a plan states a test-coverage exit criterion in so many words, grep for the named thing before calling the increment done — a feeling of thoroughness is not the same claim as a named test existing.

A synthetic probe path that doesn't mirror the real call site's resolution semantics passes on the easy case and is silently wrong on the common one. doctor._could_match_pdf checked whether sources.include could ever match a PDF by testing patterns against a probe path prefixed with the root's own name ("docs/__pdf_probe__.pdf") — but walk_sources applies each pattern via root.glob(pattern), where root is already resolved, so a pattern is relative to it, never to the KB root. The bug was invisible in-repo because the one test written for it used **/*.pdf, which happens to match regardless of the extra prefix; a bare *.pdf — an equally ordinary manifest — silently reported OK: not installed (no .pdf in include) on a KB that would, in fact, fail its very next pnk sync. Lesson: a synthetic probe is only as good as its fidelity to the real resolution path, and one test shape that happens to tolerate an error is not evidence the error doesn't exist — test the shape closest to the literal documented example, not only the most generic one.

The docs-in-the-same-commit rule missed two files because neither is named a "user-facing surface." DESIGN.md and CLAUDE.md were amended correctly in the same commit that made CI a three-leg matrix; README.md's and the Makefile's own make install comments, both reading "as CI does," were not — the exact class of drift a 0.1.2 audit already caught once for this project (above). Neither file is a flag, a manifest key, or a --help string, so neither felt like it was in scope. Lesson: "describes CI/build behaviour in prose" is as much a user-facing surface as a CLI flag — grep README.md and the Makefile for the thing that changed, not only cli.py and DESIGN.md.

I2 — the synthetic PDF corpus (20260727 23:43)

HIGH — a gate rendered at a resolution that made its own threshold unreachable. The scanned stratum's tolerance is "more than 300 pixels differing by more than 32 levels", derived for a page rastered at 150 dpi. The comparison called page.render(scale=1.0) — pdfium's default of 1 px per point, i.e. 72 dpi — which downsamples the stored 150 dpi image ~2× before diffing. That shrinks the page from 2,105,025 px to 485,316 and a moved word's delta from several hundred pixels to well under a hundred: the gate would have passed exactly the change it exists to catch, while its docstring claimed a 2× margin. The docstring also named A4 (1240×1754) for a corpus that is US Letter throughout, so both factors in the derivation were wrong and they partly cancelled, which is why the number still looked plausible. Lesson: a tolerance is meaningless without the resolution it is measured at — state both, and assert the comparison runs at the fixtures' own. When a derived constant is checked, re-derive it from the code that consumes it, not from the prose that introduced it.

HIGH — textwrap invented a hyphen the ground truth then rendered as a phantom space. textwrap.wrap(..., break_long_words=False) leaves break_on_hyphens=True, so it split the existing compound "spine-out" across two lines; the ground truth joins lines with a space, yielding "spine- out" — a string no correct extractor could ever produce, in the one file whose entire job is to be what a correct extractor produces. The same word appears correctly in 16 other places in the corpus, so the corpus contradicted itself. Lesson: when a helper's default silently edits content, the edit shows up wherever the content is re-joined by a different rule. Hyphenation is exercised deliberately here by fixtures that place their own hyphens; it must never arrive by accident.

HIGH — the soft-hyphen fixture's ground truth dropped the first four words of its own page. The expected text began "cooperation agreement…" while the page reads "The clerk filed the coopera- tion agreement…". Written by hand while thinking about the joined word, not about the page. Every automated check still passed: the fixture had a ground truth, the counts matched, the bytes regenerated. Lesson: nothing in a corpus's own test suite can tell you a ground truth is wrong — only reading it beside the page can. Budget for that reading explicitly; "the tests are green" is not evidence about the one thing tests cannot check.

MEDIUM — two claims with no enforcement, in a file full of enforcement. "Pillow is dev-only, never core, never an extra" was stated in the commit message, in conftest.py and in a test docstring — and adding Pillow to [project.dependencies] left the whole suite green, while the structurally identical claim for pypdfium2/anthropic is tested. Separately, pdf_runnable()'s three-part predicate had a test walking false→false→false→false→true that never turned the corpus clause off once both libraries were on, so deleting that clause entirely still passed. Lesson: for an N-part predicate, assert each part is individually load-bearing by turning it off from the all-true state — a monotonic walk up to true proves only the last flip mattered.

LOW — the same quantity was stated twice, differently, and neither figure was right. The commit message said ~440 KB (that is du's block-rounded disk usage of the whole directory), the CHANGELOG said ~370 KB (unreconstructable), and the budgeted quantity — the PDF bytes test_byte_budget sums — is 266 KiB. Lesson: when a number appears in two documents, both are guesses unless one of them was measured; measure once and paste the same figure.

I3a — extraction core, pure: chars to ordered, de-furnished text (20260728 00:52)

HIGH — column clustering compared each candidate to the wrong reference point, letting drift accumulate past its own threshold. reading_order grouped blocks into columns by comparing each new block's x0 to the last-placed member of the current column, not the column's start. Sorted by x0, each step can individually stay under _COLUMN_GAP while the column's accepted range walks steadily rightward — so a genuine third column, far enough from the first to be its own column, could still merge into the second's cluster one small step at a time. Caught with a reproduction script laying out three real columns and reading the wrong (merged) order back. Lesson: "cluster by gap" needs a fixed anchor — the cluster's start, not its most recent member — or the gap check bounds a single step while saying nothing about total drift.

HIGH — y-band clustering by round() put a hard wall at every half-integer. strip_running_heads grouped running-head candidates into y-bands with round(block.y0). Two renderings of one genuine running head at 750.4 and 750.6 pt — sub-point jitter, far smaller than any real layout difference — round to 750 and 751: two distinct, non-recurring signatures, each individually under the suppression threshold even though the line recurs on every page. Fixed with tolerance-based clustering (shared anchors, abs(y0 - anchor) <= _RUNNING_HEAD_Y_TOLERANCE) matching _LINE_TOLERANCE's own approach elsewhere in the same file. Lesson: round() is a clustering method with an invisible discontinuity at every .5; anything claiming "the same, allowing for rendering jitter" needs a tolerance compare, never a shared rounding function, or the false-boundary cases won't show up until real PDFs hit them.

HIGH — the import-purity test only recognised import X, not from X import Y. _imported_names walked ast.ImportFrom nodes and recorded node.module only. from pinakes.extract import layout — the exact style layout.py itself already uses for its own dependency on ExtractedText — resolves to the module name pinakes.extract, plus the separately recorded name layout; the check for "extract.layout" matched neither, so textpolicy.py could have imported layout.py this way and test_textpolicy_is_pure_and_does_not_import_layout would have stayed green. Fixed by folding ImportFrom.names into fully-qualified names (f"{module}.{alias.name}") alongside the bare module. Lesson: an import-graph test written against ast.Import habits misses ast.ImportFrom entirely unless it's built and then attacked with the exact style the file under test itself uses.

MEDIUM — a page's dominant font size was voted on by character, not by line, so a verbose heading could out-vote the body size it was meant to be measured against. _mode_font_size originally took one entry per character in blocks_from_chars; a short body line has few characters, a heading with a long title has many, and counting per-character let a sufficiently wordy heading tip the "mode" size to its own, inverting line_size > body_size for the very line it should have flagged as a heading. Fixed to take one entry per line ([max(c.font_size for c in line) for line in lines]). Lesson: "most common value" needs its unit stated explicitly — voting by the wrong unit of measurement produces a plausible-looking answer that is wrong in exactly the cases with more text, which are also the cases most likely to be headings.

MEDIUM — a symmetric rule was checked on one side only, twice, in different functions. join_hyphenation skipped a block as a join source when it was suppressed, but not when it was itself a heading — so a heading ending in a hyphen could be joined into as a source, even though the same function already refuses to join into a heading as a continuation. Separately, assemble silently produced a truncated document if a block's page_index ever fell outside range(len(pages)) — a caller bug that should be loud (I3b's future pdfium adapter is the only caller that will ever construct page_index), rather than a quietly shorter ExtractedText with no error at all. Both fixed in the same pass: the heading check now runs both ways, and assemble raises RuntimeError on an unplaced block. Lesson: when a function enforces "never X across a boundary," check both directions explicitly — a docstring that says "either side" is a claim, not a guarantee, until both sides have a test — and prefer a loud failure over a silently smaller correct-shaped result whenever "silently smaller" is a shape invariants alone can't distinguish from correct.

LOW — fixing the "no filesystem access" gap introduced a fragile substring check, caught before it shipped. Extending the import-purity test to also assert no os/pathlib/io import used the same marker in name substring style already used for PDF libraries ("pypdfium2" in name). Re-deriving what that check would actually match against layout.py's real imports first — rather than trusting that it passed — showed "io" in name matches typing.Optional and collections.abc.Iterable (...t-i-o-n...), neither of which touches a filesystem; the check would have false-positived the moment either was ever imported. Fixed to match on the module boundary (name == module or name.startswith(f"{module}.")) before it was ever committed. Lesson: two-letter module names are not safe substring needles — os/io collide with ordinary English inside almost any longer identifier — so "does this file import X" must match on the dotted-name boundary, never bare containment.

I3b — the pypdfium2 adapter, extraction-quality metrics, and the two fitted floors (20260728 03:06)

HIGH — an empty page list is not an empty request; it is pypdfium2's spelling of "every page." slice_pages(path, first, last) clamped last to the document's own last page but never validated that first still fell before it. Whenever first > last after clamping — a reversed range, or a first entirely beyond the document — range(first, last_clamped + 1) is empty, and an empty list is falsy in Python: pypdfium2's own import_pages treats a falsy pages argument identically to pages=None, its own spelling of "import every page." Verified directly against the real 12-page baseline-12p.pdf: slice_pages(5, 2), slice_pages(100, 200) and slice_pages(20, 30) each silently returned all 12 pages, no exception. slice_pages is stated as I7b's future paid-path request unit; a future off-by-one computing a page window (the last window of a document whose length isn't a multiple of the window size is the obvious candidate) would have silently sent the entire document to a paid API instead of a small slice — a cost-control failure, not merely a wrong answer, in a project whose one hard invariant is that the free path stays free and spending is never accidental. Fixed with explicit validation (first >= 0, first <= last_clamped) before the range is ever built, raising ValueError. Lesson: an empty collection is not automatically "no items requested" to the function receiving it — some APIs (documented or not) treat empty/None identically as "unfiltered," and a range-clamping function must validate the range is still non-empty itself, not merely non-negative.

MEDIUM — "wide relative to the page" and "spans multiple columns" are not the same fact, and only one of them is safe to test for. reading_order's spanning-block detection was measured against exactly one fixture (two-column-b.pdf's caption, 79% of the page's content span, against a 42% maximum for any genuine column line) and shipped as a fixed fraction, _SPANNING_WIDTH_FRACTION = 0.6. An independently-constructed asymmetric layout — a narrow sidebar beside a much wider main column, a real and common shape, not a contrived one — put the main column's own lines at 77% of the page's content span with nothing in it actually overlapping the sidebar at all; the width-fraction test misread every line of the wider column as spanning and interleaved the two columns line by line, silently, with no error. Fixed by replacing the global-width test with a geometric one: a block is spanning only if its own x1 reaches at or past the next column's own x0 — genuinely bridging into that column's territory, which the caption does (its x1 passes the right column's x0) and the wide sidebar-adjacent column does not (there is nothing to its own right to reach into). Both the original caption case and the new asymmetric-column case are now committed regression tests. Lesson: a measurement taken from one fixture is a fact about that fixture, not evidence the derived threshold generalises — check whether the underlying mechanism the threshold approximates (here, "does this block's own geometry actually overlap another column's") can be tested directly instead, before shipping the approximation.

MEDIUM — a fitting function that raises on a missing upper bound but silently guesses at a missing lower bound is not applying one policy, it is applying two, only one of which is stated. fit_running_head_threshold raised loudly when no true-positive recurrence was ever observed, but silently fell back to max_true_negative = 0.0 when no true-negative was ever observed — and a lower fallback threshold makes strip_running_heads more aggressive, so this fallback was assuming the best case (no decoy content ever recurs) with no evidence for it, dormant only because the current corpus happens to have 76 true negatives. Made symmetric: both empty cases now raise. Refactored the pure midpoint arithmetic out of the corpus-walking function (threshold_from_fractions, taking fraction lists directly) specifically so both raise paths are covered by a direct unit test, not only reachable in principle through a synthetic corpus directory. Lesson: check every "if empty, fall back to X" for whether X is a measured true value (0.0 non-whitespace characters is the true yield of a page with no native text layer — the sibling floor's own fallback, left alone) or merely a plausible-sounding guess standing in for missing data — only the former is safe to leave silent.

MEDIUM — every extraction-quality metric whitespace-flattens its input by design, which means a duplicated-newline regression is invisible to the very gate meant to catch regressions. The \r/\n-character fix (dropping embedded line-break characters that were duplicating assemble()'s own inter-block separator) shipped with zero regression coverage anywhere in the suite: reverting it and re-running every test file, plus the real make pdf-eval gate end to end, produced zero failures, because score_document's own documented design whitespace-flattens both extraction and ground truth before scoring any of the five metrics. Added a structural test asserting the raw, unflattened extraction contains no "\n\n" — verified to actually fail against the reverted code before being trusted. Lesson: "every metric flattens whitespace by design" is a correct, deliberate choice for what those metrics should measure, and also a standing blind spot for anything whose only symptom is whitespace — a fix in that category needs its own structural test, in a file that doesn't flatten, or it ships permanently unguarded.

LOW, bundled — three findings from the same review, each small alone. slice_pages with a negative first leaked a raw pdfium.PdfiumError instead of the module's own ExtractionError (resolved by the same upfront validation as the HIGH finding above, which now catches it before pdfium is ever reached). test_check_script.py's guard test asserted three substrings existed somewhere in check.sh, which stayed green even after deliberately replacing the real make pdf-eval call with a no-op while leaving the explanatory comment above it untouched — rewritten to match the actual if/then/else/fi block and assert where each string falls (inside then, absent from else), verified to fail against the gutted version before being trusted. Rate's numerator/denominator were typed float though every call site produces an int (character, word, and pair counts, and sums of the same) — corrected to int, the type they actually are, with no behavioural effect found. Lesson, shared: "does this test still pass if I break the thing it claims to guard" is a cheap, five-minute check worth running on every new test before trusting it — two of these three would have shipped a false sense of coverage without it.

Cross-platform scanned-fixture rendering — main CI-red since I2, unnoticed for three pushes (20260728 08:13)

HIGH — ./check.sh passing locally was silently substituted for "CI is green," and nobody was checking which one had actually been verified. I2's merge, I3a's merge, and I3b's merge each pushed to main believing the suite was green, because each one was green — on macOS, the only platform any of them had run on. gh run list (prompted by the user asking to check GitHub Actions, not by any check of my own) showed all three runs had actually failed on check (light pdf) / check (light pdf claude), with the identical signature every time: test_scanned_regeneration_within_tolerance — scanned-clean: 8006 pixels differ by >32 levels. Three consecutive merges landed on a red main and every one of them was reported to the user as successfully shipped. Lesson: "the local gate is green" and "CI is green" are different claims — one is evidence for the other, not a substitute for it, and the project's own standing rule (check actual CI status before building on top of a branch) applies with equal force to checking it after pushing to one.

Root cause, confirmed empirically rather than assumed. Every text fixture referenced /BaseFont /Helvetica with no embedded font program (pdfwriter.py, since I2), relying on the PDF reader's own substitution for a font it doesn't have. pypdfium2 ships platform-specific prebuilt binaries; macOS has a real Helvetica installed, ubuntu-latest does not, so pdfium substitutes a different font on each — metrically compatible (identical word-wrap, identical line breaks) but with different glyph outlines. The scanned stratum rasterizes baseline-12p through pdfium at fixture-generation time, baking whichever platform generated it into the committed PDF, so CI's own regeneration (on a different platform) could never match. Confirmed, not theorized: a Docker ubuntu:24.04 container reproduced CI's exact number (8,006 px) on the first attempt, and a diff heatmap of the two renders showed every changed pixel sitting exactly on a glyph edge — same text, same word positions, same line breaks, different anti-aliasing. Measuring all ten scanned pages found cross-platform noise ranging 507-8,262 px depending on how much text and how much the contrast reduction already suppressed it (scanned-low-contrast came in far lower than scanned-clean, consistent with the tolerance test's own documented noise-floor behavior).

The obvious fix (raise the tolerance) was rejected on evidence, not instinct. The measured noise ceiling (8,262 px) sits far closer to the test's stated detection target — a single moved word, "a small fraction of a page" per the test's own docstring — than to the documented whole-page-shift signal (33,451 px) the 300 px threshold was originally sized against. Raising MAX_CHANGED_PIXELS above the noise ceiling would have unblocked CI immediately, but a real single-word regression is plausibly smaller than 8,262 px, meaning a tolerance wide enough to absorb cross-platform noise would very likely also have absorbed exactly the class of regression this test exists to catch — silently, with no way to tell from a green gate. Presented three options (raise the tolerance / scope the test to one platform / embed a real font) with honest pros and cons rather than picking unilaterally, since it's a public-repo, shared-CI-affecting change to a previously "verified" threshold. Fixed at the root instead: pdfwriter.py now embeds a subsetted TrueType font (tests/pdf-corpus/fonts/LiberationSans-Subset.ttf, SIL OFL 1.1 — Liberation Sans specifically for its Helvetica/Arial metric compatibility, so none of generate.py's hand-placed coordinates needed to change) instead of a bare base-14 name, so every platform rasterizes the same outlines regardless of what fonts it has installed. Re-running the identical Docker reproduction after the fix measured 0 pixels changed across every scanned page — not merely under tolerance, bit-for-bit identical. Lesson: when a tolerance-based test's noise floor turns out to be closer to its detection target than expected, re-measure what the tolerance would need to become and check that against the smallest real regression it's meant to catch, before touching the number — a gate that still passes is not evidence it still catches anything.

MEDIUM, from independent review — the fix itself shipped with no regression test. An adversarial re-verification (independently reproducing the pre-fix failure and the post-fix 0-pixel result in its own Docker run, and re-deriving the subset font byte-for-byte from a genuine Debian package to confirm tests/pdf-corpus/fonts/README.md's recipe is real) found that nothing in the diff would have caught a future revert to bare base-14 fonts: the only guard was the pre-existing tolerance test, whose whole failure mode is passing locally and only failing on a second platform — exactly what let this bug live unnoticed for three merges. Fixed with a platform-independent test (test_text_fixtures_embed_a_font_program) asserting every non-scanned fixture's committed bytes contain /FontFile2 and never a bare /BaseFont /Helvetica — verified to actually fail against 89d4fb5's committed fixtures before being trusted, and deliberately written with no pypdfium2 dependency so it runs even on the [light]-only CI leg. The same pass found _font_widths' first_char hardcoded at 0x20 with no symmetric downward extension (unlike last_char), which would have indexed _ASCII_WIDTHS negatively for a hypothetical differences code below 0x20 — unreachable today (grepped every Font(...) call site) but fixed to be symmetric regardless. Lesson: "the existing test would have caught this eventually, on some other platform" is not the same claim as "this increment shipped its own regression test" — a fix for a bug that was invisible on one platform needs a check that doesn't depend on which platform runs it.

I4 — the extraction cache (20260728 10:28)

MEDIUM — a filename collision between a real cache entry and its own in-flight write. Every scanning function (survey, total_stats, clear_all) globs *.json; _write's atomic-write temp file was originally suffixed .json too (.tmp-<random>.json, meant to land beside the final name before os.replace). Verified directly: pathlib.Path.glob("*.json") matches dot-files, unlike shell globbing, so that temp name was scanned as a real entry by every one of them. The window only opens on an uncatchable kill (SIGKILL, OOM, power loss — _write's own except BaseException already cleans up anything else), but inside it a stray file could be double-counted in pnk doctor's totals forever (unreachable via the keyed entry_path() lookup, so a fresh real entry gets written alongside it, never replacing it) and, worse, misclassified as a paid orphan if the abandoned write happened to carry an operation_id — read by the one thing this module exists to protect. Fixed by suffixing the temp file .tmp, never .json, so it can never match the glob regardless of the leading-dot question. Lesson: when several functions all key off "does the filename match this pattern," a temp/staging file used by the same module needs its own, deliberately non-matching pattern — matching the final name's extension by habit (.json in, .json out) is exactly how a temp file becomes indistinguishable from a real one.

LOW-MEDIUM, no live trigger today — a JSON round-trip validated key structure but not value types. _read's reconstruction of per_page_provenance did dict(page) for page in provenance with no check that a page's values were actually strings, unlike page_spans three lines above (int(span[0]), int(span[1])) — the same rigor wasn't applied to both fields reading from the same untrusted JSON. Verified directly: a hand-written entry with {"confidence": None} was accepted as a clean cache hit, silently degrading ExtractedText.per_page_provenance's declared Mapping[str, str]. No live writer currently populates a non-empty per_page_provenance (pdfium.py and the fake backend both rely on the dataclass default), so this was unreachable in practice — but the cache exists precisely to survive untrusted/older/hand-edited files, and I5/I7b are exactly the increments that will start writing real provenance. Fixed with _string_mapping, validating every key and value before trusting the entry. Lesson: "no code currently writes the bad shape" is a fact about today's callers, not a property of the format — a cache that reads its own JSON back should validate every field it reconstructs the same way, not just the ones a current caller happens to populate.

LOW — two verified-true claims shipped with no regression test. A cache-write failure (chmod 0o500 on the cache directory, reproduced directly) correctly returns the extraction result without raising, matching the contextlib.suppress(OSError) comment's claim — but nothing asserted it. Two documents sharing one content_hash within a single KB correctly keep their shared cache entry after one of them is deleted (eviction keys on the hash, which is still claimed by the survivor) — only the cross-KB duplicate case had a test. Both were already correct; both now have a test, one of which (the write-failure case) was verified to actually exercise the except OSError branch by checking no cache file was created afterward, not merely that no exception propagated. Lesson: an already-true claim without a test is one refactor away from becoming a false one with no signal — "this works" and "this is tested" are different sentences even when both are honestly true today.

Reviewed and found correct, not a defect: pnk sync --clear-cache (bare, no argument) deleting paid cache entries along with everything else is the intended I4 behavior, not a gap — the plan's own text ("removed only by an explicit --clear-cache=paid") describes a narrower, paid-preserving variant that lands with I7c's ledger reader, which can price what it would be destroying; building selective removal before that reader exists would mean guessing at a cost nothing can yet compute. Confirmed live: an injected paid orphan was removed by pnk sync --clear-cache --yes along with every other entry, consistent with clear_all's own docstring and docs/DESIGN.md's description.

I5 — PDF chunking, page provenance, and a backend-aware sync (20260728 13:59)

CRITICAL — decision 9's "never silently downgraded" guarantee held for a same-path sync and for --rebuild, and silently did not for everything in between: a rename, or a document's first sync on a machine that never extracted it. The initial design protected a paid extraction two ways: pairing.py's own comparison when a document keeps its path, and --rebuild's copy-forward from the old index. Neither covers Adopt/Rename outside a rebuild, which instead fell through to _extract_for_index's only remaining signal — an extract/cache.py hit. A cache miss was then read as "content changed", which is a false equivalence: a --clear-cache immediately before a rename, or the first sync of a KB whose paid PDFs were extracted on a different machine (docs/ committed, .pinakes/ gitignored — the ordinary shape of a clone), miss the cache identically without the file having changed at all. Confirmed live, both ways: (1) paid-index a PDF, --clear-cache, rename it with its sidecar travelling, sync — landed a PaidExtractionRequiredError falsely claiming changed content, and left the index describing the old, now-nonexistent path, since the failed transaction rolled back before the rename could be recorded; (2) paid-index a PDF, rm -rf .pinakes (index and cache, simulating a fresh clone), first sync — identical false failure on a document that had never changed at all. An independent adversarial review (a fresh subagent, unanchored on this increment's own design reasoning) found and reproduced both.

Fixed by moving the "has this changed" decision off the cache entirely: the sidecar's own provenance.extraction gained a fourth field, content_hash — the file's hash at the moment of that specific paid extraction, distinct from the general change-detection hash docs/DESIGN.md §2.2 already refuses to store, since this one changes only when a fresh paid extraction runs. Comparing it directly against the current file's hash answers "changed?" without consulting any cache or index at all. Getting the actual text without paying again is a separate question, now answered in a fixed order: this same sync's own connection (covers a rename — the row is still there under its old path), the old index during a --rebuild (unchanged from the original design, now keyed on doc_id rather than content_hash+path, since a rename's action only ever carries its current path and the old index's row still has the old one), then extract/cache.py. Only when none of the three has an answer does it fail — and now with a new, distinct error (PaidExtractionUnavailableError) naming the file unchanged but its text simply not present on this machine, never conflated with "content changed" (PaidExtractionRequiredError) again. The cross-machine case (2) is not solved by this — there is genuinely no durable, shared home for a paid extraction's text yet — but the failure is now honest about which situation it is, which docs/DESIGN.md §9 records as an accepted, disclosed limitation rather than a silently discovered one. Lesson: a mechanism justified as "the answer must not depend on the cache" (the --rebuild + --clear-cache case this increment was explicitly designed around) needs that property checked against every path that can reach the same decision, not just the one path the design started from — pairing.py's own decision table has four ways into "paid, free-effective, unchanged", and only one of them was ever traced all the way through.

HIGH — --rebuild turned "a paid document's content changed" into "the document vanishes from the index", where a normal sync leaves it stale but searchable. pair()'s PaidExtractionRequired action (decision 14) can only ever fire against a populated before, so it can never fire during a rebuild at all (before is empty by construction) — meaning a changed-hash paid document reaching --rebuild fell through to _extract_for_index's ordinary raise, caught by _apply's generic exception handler, which never inserted a row for it in the first place. Confirmed live: paid-index a PDF, change its bytes, pnk sync --rebuild — report.ok correctly False, but the document was gone from the rebuilt index entirely (zero chunks), not merely flagged. A normal (non-rebuild) sync hitting the identical case leaves the old row, chunks and embeddings untouched instead. Fixed by extending the rebuild copy-forward to the changed-hash case too: the stale row is copied forward at its old content_hash regardless of whether the current file still matches, and a failures entry is recorded only when it does not — so a rebuild now reaches the same outcome a normal sync already gives this exact case, rather than a harsher one purely as a side effect of which command happened to run. Lesson: --rebuild is documented as "free, deterministic, cron-safe" — exactly the description that invites treating it as interchangeable with a normal sync. Any new failure mode this increment gives normal sync a considered answer for needs the identical question asked of --rebuild explicitly, since its empty before makes "the same case" arrive by a structurally different path that is easy to reason about in isolation and forget to reconcile.

MEDIUM — a fresh paid-provenance write left the very next sync one RefreshMetadata cycle away from settling. _index_document decides sidecar_hash from the walk, before it rewrites the sidecar with fresh provenance.extraction a few lines later in the same call — so the hash it writes into documents.sidecar_hash was already stale the moment the write happened. Confirmed live: three consecutive syncs of a freshly paid-extracted PDF produced embedded=1, then refreshed=1 (unexpected — nothing the user did changed), then finally skipped=1. Fixed by recomputing sidecar_hash from the file just written, whenever a write happened, before it reaches the documents INSERT. Lesson: any function that both decides a hash-like value from an earlier read and has the power to invalidate that exact read later in its own body needs the value recomputed after the write, not carried from before it — "I already have this value" stops being true the moment code between the read and its use can change what it should have been.

Caught live, during the adversarial review itself, not by this increment's own author: two different documents can share one content_hash with only one of them ever paid-extracted (a second document minted later for identical bytes gets its own ordinary free extraction). The original rebuild copy-forward keyed its survivor lookup on content_hash alone, which would have let the free twin's own rebuild incorrectly inherit the paid one's chunks, embeddings and backend label. Found and fixed inline by the reviewing subagent (re-keying on (content_hash, path), later superseded by the doc_id keying above once the rename/clone findings required a broader redesign), with its own regression test. Lesson: a value that is "usually unique in practice" is not a key — content_hash was never meant to identify one document, only to detect whether one had changed; this table's actual primary key was sitting right there the whole time.

I6a — budget core, pure (20260728 17:52)

HIGH — every test of the timezone conversion that is this module's entire reason to exist would have passed with the conversion deleted. window.py aggregates ledger records into day/month totals in [budget] timezone, converting both now and each record's reserved_at before comparing. Every test — including the midnight, month-end and DST-transition trio written specifically to exercise attribution — constructed both values already in the target zone, where .astimezone() is a no-op. The adversarial review mutated local_now = now.astimezone(timezone) to local_now = now, and separately the per-record conversion, and both mutations passed all 35 tests. The real case is ordinary, not exotic: a ledger storing UTC timestamps (the obvious choice for I6b) read under a non-UTC [budget] timezone — 23:30 UTC on the 15th is 00:30 on the 16th in Berlin, so either mutation silently files the spend under the wrong day. Fixed by adding a test that aggregates a UTC-stamped record against a Berlin window. Lesson: a test whose fixture is built in the same units the code converts to cannot detect a missing conversion. Exercising a transformation requires input where the transformation actually changes something — three tests that carefully varied the clock while holding the timezone constant proved nothing about the timezone at all.

MEDIUM — an exception hierarchy copied from a sibling module inherited the wrong exception types. prices.py was deliberately modelled on extract/floors.py, including its except (TOMLDecodeError, KeyError, TypeError, ValueError) around parsing. But floors.py parses with float(x), which raises ValueError on bad input, while prices.py parses with Decimal(str(x)), which raises decimal.InvalidOperation — not a ValueError subclass (InvalidOperation → DecimalException → ArithmeticError). A single-character price typo (a European "5,00", an unfilled "TBD") therefore escaped as a bare InvalidOperation instead of the named PricesMissingError the module's own docstring promises, and the test claiming to cover this only ever exercised a TOML syntax error. Lesson: when mirroring a module's error handling, the except clause travels with the parsing call it was written for. Changing float to Decimal changes which exceptions are possible, and an inherited except tuple is a claim about the old code.

MEDIUM — validation absent at the one boundary where a wrong sign inverts the guarantee. estimate_document accepted any pages/pages_estimated: pages=0 produced requests=0 and made per_request_eur raise on a zero division, and a negative pages_estimated produced a negative total_eur. Every other failure mode in the module — unknown model, missing prices, stale prices, oversize request — had a named error, but the one that would make a budget guard understate spend had none. Not reachable from any caller in this increment, since nothing calls it yet; guarded at the source rather than trusting a caller that does not exist to be written correctly. Lesson: for a component whose whole job is bounding a number, the input validation that matters most is whichever one lets the number move in the safe-looking direction.

LOW, worth keeping — three assertions that were true but untested, in the same shape. reserve_document's "every blocked window is named" was only ever tested with all three windows breaching at once (so "always names all three" would also have passed); reserve's "first breach in order wins" was only tested where a single window could breach (two of three caps were set to a generous 100 in every case); and confirm_above_eur's strict > boundary was asserted only incidentally. All three were verified correct by hand and none was a defect — but a regression in any would have been invisible. Lesson: a passing test suite says nothing about the claims it never puts under tension; the boundary cases worth writing are the ones where two plausible implementations disagree.

Also fixed: Table.decimal()'s default path returned before its own minimum check, so a below-minimum default would pass silently — integer()/number() avoid this for free by sharing one code path between default and parsed value, which is why the bug was invisible by analogy. ContextWindowExceededError's remedy told the user to lower a [chunking]-equivalent slice size that does not exist as configuration (K is a fixed constant). Every fix in this increment was confirmed to fail against the pre-fix code before landing.

I7a — the paid-path allowlist gate (20260728 19:25)

HIGH — the gate found two live paid-client imports on the free path, in code that had shipped. doctor._extraction reported whether the configured extractor was available by calling load_extractor(backend), and sync._missing_pdf_extra did the same to decide whether a skipped .pdf still needed an extra. The registry's factory imports the client (extract/__init__.py's _import), so on a KB whose [extraction] backend is claude-vision, both pnk doctor and pnk sync pulled anthropic into a free-path process — doctor on every run. Neither was reachable from any test, because every test KB is configured for pypdfium2. Nothing could actually spend (the extractor is an I7b stub), so the cost was an import rather than a charge — but the invariant CLAUDE.md calls non-negotiable was already false when the gate arrived to check it. Lesson: "does the free path import a paid client" is a question about a running process, not about the source text. A grep over src/ was green the whole time both leaks existed, because neither leak is an import statement — it is an ordinary function call that reaches one.

HIGH — the gate's own paid KB was silently never paid. The free-path runner configures a second KB for claude-vision, since that is the only configuration where the two probes above fire. It did so with text.replace('backend = "pypdfium2"', ...) against a manifest template that has no [extraction] section at all — so the replace matched nothing, the KB stayed on the free backend, and the gate would have passed whether or not the leaks existed. Caught only because the run printed pdf extractor: pypdfium2 importable for a KB that was supposed to say claude-vision. Fixed by appending the section rather than replacing, by making every manifest rewrite a _replace_once that raises when it matches nothing, and by loading the manifest back through the real parser and asserting extraction.backend is what was asked for. Lesson: str.replace returns the string unchanged when it matches nothing and reports it to no one — the perfect way to build a test fixture that does not test what its name says. A fixture whose whole purpose is to be in an unusual state must be read back through the parser and checked.

MEDIUM — two of the first six mutations survived, and both survivals were the finding. Mutating is_backend_installed to return True passed the entire suite: its only two callers are tested with it monkeypatched, so nothing exercised the function itself. And test_a_directory_entry_fails_gate_1 asserted only a non-zero exit — which gate 2 produced anyway by reporting the planted import, so the test stayed green with gate 1's directory branch deleted. Fixed with three direct tests (including one whose probe module raises on import, so a regression to importing errors rather than fails quietly) and by asserting gate 1's specific message. Lesson: a function whose every caller stubs it out has no test, only agreement; and an assertion on an exit code cannot tell which of two gates produced it.

MEDIUM — the runtime checker would have flagged google.protobuf as a paid client. Matching sys.modules names by root (name.split(".")[0] in {"anthropic", …, "google"}) makes every google.* module a hit, and protobuf arrives transitively with onnxruntime, grpc and much of the ML ecosystem. Not triggered on this platform today — verified, zero google.* modules in a free-path run — which is exactly what makes it dangerous: it would have fired first on some other CI leg, on a change having nothing to do with money, and the obvious repair for a safety gate that cries wolf is to weaken the gate. Now matched on a dotted-prefix boundary against google.generativeai in full. Lesson: for a gate whose failure mode is "someone turns it off", the false-positive direction is as load-bearing as the true-positive one — and a latent false positive is worse than an active one, because it lands on an unrelated change.

LOW, worth keeping — gate 2 cannot see a dynamic import, and that is deliberate. extract/__init__.py calls __import__("anthropic", …) inside the claude-vision factory, which no import-statement grep will ever match, and it is not on the allowlist. Exempting the registry would exempt the one file where an accidental static import is most likely; the dynamic call only runs when a caller has explicitly selected a paid backend, which is an allowlisted entry point. The limit is written into the gate's own docstring, and gate 4 is what covers the direction gate 2 cannot: no spelling of an import hides from sys.modules. Recorded so the next reader does not "fix" gate 2 by widening the allowlist.

Also: pnk doctor no longer proves a paid backend's adapter constructs, only that its library is locatable — the necessary price of not importing it, and worth remembering at I7b, when _load_claude_vision stops being a stub. Every fix above was confirmed to fail against the pre-fix code: 10 mutations planted, 10 detected, including both fixes made during this review.

I6b — budget I/O: the ledger, pnk budget, hooks that cannot spend (20260728 23:27)

HIGH — the accountant handed out a PaidCall object, undoing the one guarantee the ledger module exists to provide. budget/ledger.py's whole argument is that a void may only be written when no response was received, and that this is enforced by paid_call being a context manager rather than a convention someone remembers. Accountant.open_call then wrote the reservation and returned the object, putting both the void/unknown decision and the closing write back in the caller's hands — and the caller it was written for is I7b's retry loop, the most branch-heavy code in the release. Its own test left a permanent unknown outcome behind and asserted nothing about it. Now a context manager delegating to ledger.paid_call. Lesson: an invariant enforced by a control-flow construct is only enforced where that construct is actually used. A convenience wrapper one layer up is exactly where it gets quietly opted out of, and "the module below guarantees it" stops being true the moment a caller can hold the handle.

MEDIUM — a single bad character in the ledger could take pnk budget down entirely. Every euro figure is cost_usd / usd_per_eur, computed in a property called long after parsing, from inside the summing loop. A line with usd_per_eur of "0" therefore raised DivisionByZero out of a read-only reporting command — defeating the malformed-line counting whose entire purpose is that no one bad line can do that. The parse-time checks covered type (a JSON number is refused, so no float gets in) but not domain. Rates are now validated positive at parse time, where a failure is a counted malformed line. Lesson: validation placed at the parse boundary only protects what is computed at the parse boundary. A derived value computed lazily elsewhere needs its inputs checked where they enter, not where they are used.

MEDIUM — fsync on the file does not make the file's name durable. The reservation is written before the call precisely so a crash during the call cannot lose it, and each write is fsynced. But creating a file and fsyncing its contents leaves the directory entry unsynced, so the very first reservation a KB ever writes — the one before its first paid call — could vanish on a crash while every later one survived. The parent directory is now fsynced on creation only. Lesson: durability claims have to name what is durable. "The write is fsynced" is a claim about bytes; "the record survives a crash" additionally requires the file to still have a name.

MEDIUM — both hook checks read root/.git/hooks directly, so they were blind in a git worktree. Inside a worktree or submodule .git is a file pointing elsewhere; hooks.hooks_dir has resolved that since I12, and doctor never used it. Every hook read as absent, so pnk doctor reported "0 of 3 installed" and I6b's new machine-driven-spend check would have reported "no hooks installed, so no automatic sync runs" on a KB whose hooks were installed and running. Worth more than its size here: this project's own CLAUDE.md mandates a worktree for every change, so the layout the check is blind on is the one it is developed in. Lesson: a helper that already handles a case is not the same as using it — the second reader of .git reintroduced the bug the first one had solved.

LOW, worth keeping — one report, two clocks. pnk budget computes its windows in [budget] timezone and printed the recent-operations list in the machine's local zone, unlabelled. On a KB synced from two machines the same operation renders at two different times, and the day a call is filed under would not match the timestamp beside it. Recorded because the fix is trivial and the class of bug is not: a report that derives one number from configuration and another from ambient state looks consistent on the machine it was written on.

LOW, second pass — two flags that read as the opposite of what they do. pnk budget listed the five most recent operations and stopped, with no line saying how many it had not shown — the silent-cap failure this plan's own ground rules name ("if a workflow bounds coverage, log() what was dropped"). And --clear-cache's bare form parsed to the value free, which reads as "clear only the free entries" when both spellings clear the whole cache; argparse validates const against choices, so a private sentinel is not available and the bare form has to be a real, honest word. It is now all, and the value names what is being authorised rather than what is removed.

LOW, third pass — the notice init --ci printed described a file it had not written. FREE_BACKEND_NOTICE began "hooks run pnk sync --extract=pypdfium2 …" and was printed by two callers: install-hooks, which writes hooks, and init --ci, which writes a workflow. So pnk init --ci announced .github/workflows/pinakes.yml and then explained what the hooks do. Every test asserted the flag and the phrase "can never spend", both of which were present and correct. Found only by running the command the docs tell a user to run — the constant is now subject-less and each caller supplies its own ("each hook …", "it …"). Lesson: a shared string with a subject baked in is correct for exactly one caller. The tests all checked the part that was shared; the part that was wrong was the part no assertion mentioned.

Also: doctor printed cost_eur — a Decimal division — with a bare f-string, putting all 28 significant digits into a health-check line; the --resolve record's operation field was documented as a value it never takes. Every fix above was confirmed to fail against the pre-fix code: 14 mutations planted over the implementation (14 detected), and 5 more reverting each review fix in turn (5 detected), the last of which found that the formatting fix had no test at all until one was written for it.

---

I7b — the paid Claude-vision extractor (20260729 00:24)

HIGH — the reconciliation recorded the reserved amount, which makes the whole reservation/reconciliation protocol a no-op. _billed_call closed each successful call with cost_usd=reserved_eur * usd_per_eur — the estimate again, not what the response said it cost. The shape was perfect: a reservation, then a reconciliation superseding it, exactly as I6b's protocol requires, with a ledger pair per call and every test about pairing passing. What it superseded the reservation with was the reservation. Every window would have charged worst-case forever, pnk budget would have reported an estimate as spend, and the reconciliation record's presence is precisely what would have made it look settled. Fixed with actual_cost_usd, derived from the response's own usage and the model's price. Lesson: I6b's tests could only ever check that a reconciliation exists and supersedes; that it carries the right number is a claim only the increment that produces the number can make. A protocol test and a value test look alike and are not — and every mutation I planted over the retry logic passed straight through this, because the bug was in the one line none of them touched.

HIGH — one bad PDF would have crashed a 1,000-document sync. TransportError and RequestTooLargeError were plain Exceptions. sync isolates each document behind except (PinakesError, OSError, ValueError), so an exhausted 429, a 500, or an oversized page would have escaped that handler and taken the entire run down — the exact opposite of the per-document isolation §6.4 promises and pnk sync's own "one broken PDF cannot block a 1,000-document corpus". Not caught by any test, because every test called extract_slice directly and asserted pytest.raises(TransportError), which passes identically whichever base class it has. Lesson: an exception's type is part of its contract with a caller several layers up, and a test that catches the exception it just raised cannot see that contract at all.

MEDIUM — two mutation survivors, both the finding. A cap check hoisted out of the transport retry loop survived because every attempt inside that loop voids at zero: nothing the loop does moves the total, so the omission looks harmless. It is not — between a 429 and its backoff another process syncing the same KB can spend the headroom, and the retry would go out anyway. And the per-slice semantic budget survived because every test used a single slice, where "per slice" and "per document" are the same number; that was a defect I had found and fixed while writing the loop and then never put under tension. Lesson: a bound that is only ever exercised at N = 1 is not tested, it is agreed with.

MEDIUM — the module imported pypdfium2 at module scope, which §4.4 cannot afford. The fingerprint path reaches this module on every query, on whatever install the user has, so a top-level pdfium import made a coherence check on a claude-vision KB fail outright on a core-only install. Caught by test_coherence_never_imports_a_paid_client — a test written in I2 for a different reason, which happened to be the exact shape of this mistake. Recorded because it is the second time this project has been saved by an import-graph test that nobody wrote for the case that caught them.

LOW, worth keeping — the gate refused the commit, which is the gate working. .paid-path-allowlist shipped empty at I7a specifically so that its first real entry would be earned, and it was: the commit creating claude.py failed until the line was added. Its test then turned out to assert only "0 exempt paths", which would have passed on any addition at all, so it now pins the expected contents — widening an allowlist is how a gate like this one dies. Two dead exception classes (TruncatedResponseError, RefusalError) were also removed: the stop_reason branches replaced them and nothing ever raised either.

LOW, second pass — --estimate-only demanded an API key from a KB with nothing to estimate. The transport was built before the walk, so a KB with no PDFs failed on a missing key instead of reporting nothing. Built on the first PDF now. Also cleaned up in the same pass: an unreachable except RequestTooLargeError around slice_pages (only build_request raises it, and by then the call is committed — the size question belongs before anything is built), and a repeated "claude-vision" literal where the module already imports the constant.

HIGH, fourth pass — the paid fingerprint omitted the model, so changing it reused another model's text. The plan states the inputs as "(backend name, model id, prompt version, schema version, request-shape version…)" and I wrote every one of those except the model. The cache key is <content_hash>-<fingerprint>, so editing [extraction] model would have hit an entry a different model wrote, with no miss, no warning and no stale marker — the §4.4 machinery intact and looking at the wrong key. The plan names two tests for exactly this (test_changing_the_model_misses_the_cache, test_changing_k_misses_the_cache) and I had written a placeholder for the first that only asserted two unrelated fields were non-empty, which is how it stayed invisible through three review passes and fifteen mutations. Fixing it meant threading [extraction] model through the registry's FingerprintInputs contract — two real callers, sync and the §4.4 coherence check — plus a test that the free backend's key is unperturbed by the new parameter, or every existing free KB's index would have gone stale the day it was added. Lesson: a test written to a name from the plan, but not to the claim behind the name, is worse than a missing test — the plan's checklist reads as satisfied. The tell was there in the placeholder's own body: it asserted things that could not fail.

MEDIUM, fifth pass — three of the plan's named tests had no implementation at all. Auditing the plan's test list against the file (prompted by pass 4, since the same failure mode was in play) found the multi-slice document, the oversize-slice split, and the no-floor-installed refusal all missing — the first being where the slice-window arithmetic and the short final slice actually meet a real PDF, and where an off-by-one either drops pages or sends the whole document to a paid API. Writing them found nothing broken, which is the useful outcome to record: the code was right and unwitnessed. Two of them also had to be sized from the fixture rather than from a constant — a hard-coded byte threshold either never splits or recurses straight to the single-page failure, and which one it does is a property of the corpus, not of the code. slice_windows and slice_bytes lost their underscores in the process: they are the two things a reader most wants to check.

LOW, sixth pass — nothing drove pnk sync itself. Every test exercised a piece: the slice loop, the ledger pairing, the cache's join key, the accountant's windows. The pieces were wired together across four modules, and "each part works" is not the claim "the parts are connected" — which is the seam an increment is most likely to get wrong. One end-to-end test now runs a real pnk sync over a paid KB and checks the whole chain: the document indexed under claude-vision, the ledger's call reconciled, and the cache entry carrying the operation_id/call_ids that §6.3 left null until this increment. It passed first time — but writing it introduced the pass's one real defect, and only the [claude] leg saw it: the test swapped the registry entry and then called unregister_extractor, which deletes. There is no undo for a name the package registers at import, so two unrelated tests later in the session lost claude-vision entirely. Fixed with a registered_entry accessor and a re-register in the finally. Lesson: a test that mutates process-global state needs to restore the previous value, not remove its own — and a suite that only ever runs on one CI leg will not show you which.

MEDIUM, seventh pass — the [light] leg's green was partly an artefact, and I had not really run that leg at all. uv run --extra light does not prune extras a previous --extra pdf --extra claude installed, so "all three CI legs pass" was one leg run three times. A real uv sync --frozen --extra light — what CI actually does — then showed the paid suite failing on its own while passing in a full run: tests/test_extract.py leaves a fake pypdfium2 in sys.modules, and two --estimate-only tests were quietly relying on it. The underlying cause was mine: _estimate_only imported page_count before the walk, so a KB with no PDFs demanded the [pdf] extra to be told it had nothing to estimate — the same defect as pass 2's transport, one line above where I had fixed it. Lesson: uv run --extra X is not uv sync --extra X; verifying a matrix means reproducing what the matrix does, not asking for the same set three times. And a suite that only ever runs after its neighbours cannot tell you what it depends on.

LOW, third pass — markdown emphasis reached a terminal. --estimate-only's help text carried **A network call**, which argparse renders as literal asterisks: the emphasis was written for CLI.md and pasted into a surface that has no renderer. Now checked against every command's rendered --help output rather than argparse's internals — the artefact a user actually sees. Backticks are deliberately allowed, since [extraction] backend reads fine in a terminal and is the convention this CLI already used; flagging those too would have been a style crusade over pre-existing text rather than a defect.

Also: stubs/anthropic.pyi joins stubs/pypdfium2.pyi, because the strict type gate runs on the [light] leg where the package is absent. It records the one relationship easy to get wrong from memory — APIConnectionError is a sibling of APIStatusError, and APITimeoutError a subclass of the former — because checking them in the wrong order classifies every timeout as a plain connection failure, which is the difference between recording €0 and admitting a possible charge. 18 mutations planted in total, 18 detected once the survivors got tests.

---

I7c — the completeness audit, staging, all-or-nothing (20260729 02:38)

MEDIUM — three of ten mutations survived, and all three were the same shape: a rule with no test that it fires, only that it exists. Clearing staging after the complete entry is written had no test that staging is ever cleared at all — and stale staging is not litter, because its key is <content_hash>-<fingerprint>: a later run of the same document would find it, skip slices, and serve text from a superseded extraction silently and for free. Stopping the corpus at the first cap breach had no test that drove sync; the on_exceed tests build a SyncReport by hand, so they can check what a stop means but never whether the loop stops. And keeping staging out of the cache root had nothing asserting that survey, total_stats and clear_all cannot see it. Lesson: the tests I wrote were about the rules I was thinking about while writing the code, which is exactly the increment-shaped blind spot this project keeps rediscovering. Mutation found all three in one pass because a mutation asks the question the test forgot: not "is the rule stated" but "would anything notice if it stopped being true".

MEDIUM — the audit's own fixtures measured nothing. word_coverage tokenises on [a-zA-Z]+, so forty words of f"{seed}{index}" collapse to one distinct word: every page scored 1.00, the dropped-content test found no outlier, and the fixture could not have failed whatever the code did. Caught only because that test failed for a different reason first. Lesson: a fixture built for a metric has to survive that metric's own tokeniser — and "the test passes" would have hidden this completely if the assertion had been slightly weaker.

MEDIUM, second pass — the audit made the very mistake it was written to avoid, one branch later. A page above the yield floor whose text holds no significant words — a table of figures, which is an ordinary PDF page, not an exotic one — gives word_coverage a denominator of zero. I scored it 1.0: full preservation claimed for something never checked, and it drags the median up, making the genuine outliers look less unusual. Six lines earlier the module argues at length that a scanned page must be exempt rather than zero, for exactly the same reason in the other direction. Now both are exempt. The fix then exposed a second worthless fixture: _significant_words keeps only words of four characters or more, so prose("a")'s three-letter words were never measurable either — that test had been passing purely on the 1.0 default it was supposed to be testing around. prose now asserts its own output is measurable. Lesson: a default that makes "unmeasurable" indistinguishable from "perfect" hides broken fixtures as effectively as it hides broken code — and I wrote the argument against it into the docstring of the function that did it.

LOW, second pass — one feature, two standards for the same question. _is_budget_refusal identifies a refusal by exception type, with a comment saying an error string is prose and prose gets reworded. Twelve lines later SyncReport.ok decided which failure was the budget's by comparing that same prose. It now matches on the path, which is structural.

LOW — on_exceed had been parsed, validated and read by nothing since v0.1. A manifest key with a choice() validator, a default, a template comment and a documented meaning, wired to no behaviour at all. It now decides whether a budget stop is a failure. Recorded because validation is what made it invisible: a key that round-trips through the parser looks implemented from every angle except the one that matters.

LOW — the interruption I first scripted was not one the caller isolates. The test double raised AssertionError when its script ran out, which sync does not catch, so it escaped the per-document handler and failed the test rather than the document. Replaced with a timeout — a real interruption, and one the caller actually handles. A test double's failure mode is part of the test: if it fails in a way production cannot produce, the path under test never runs.

LOW, and the second time in one session — I composed a timestamp instead of reading one. This entry's heading was written 03:04 moments after date had printed 02:38. CLAUDE.md's rule is already explicit ("read the clock; never compose a timestamp"), and knowing the rule is evidently not the same as running the command: the failure is that composing feels like recalling. Caught both times only by diffing the two numbers on screen.

Also: another agent, working independently, found that I7b's own docs contradicted themselves — STATUS.md said "claude-vision is a real extractor" in one row while the prose eight lines below still explained that nothing can spend because it is a stub. I had updated the table and left the paragraph justifying it. Seven review passes over I7b did not catch it. 15 mutations planted here, 15 detected once the survivors got tests.

---

---

The eval harness: three defects under one green suite (20260729 03:23)

Found while planning the links and graph releases, whose whole gate rests on this harness. All three were live on main and all three passed every test.

HIGH — the multi-hop class measured nothing about hopping. Outcome.hops_followed was computed for every scripted question and read by no metric — not recall_at_k, not by_kind, nothing compare() looks at. Deleting the hop loop outright left by_kind["multi-hop"] bit-identical. A multi-hop question was a single-shot search of its last hop's query wearing a label. The one guard was assert any(outcome.hops_followed > 0 …) — an any() over five questions, on a field that fed nothing.

HIGH — and that hid a defect in the golden set itself. Three of the five questions named their last hop's document in expect; two named their first. So the scorer ran a query about brittle-paper conservation and demanded the annual report. Nothing could catch the disagreement, because hops fed no metric that could notice it. The fix makes expect exactly the union of the hops' documents and asserts it for the committed set.

The numbers moved because the scorer was wrong, not because retrieval changed. recall@5 0.8788 → 0.9091, MRR 0.7737 → 0.8116, rerank precision 0.7273 → 0.7576, by_kind["multi-hop"] 0.80 → 1.00. Stricter scoring, higher score — because the two inverted questions had been asked about the wrong document all along. A metric that improves when you make it stricter is telling you it was measuring something else.

MEDIUM — compare() wrote by_kind into every baseline and never read it back. A change lifting one class and dropping another by the same amount moves the aggregates by almost nothing; CI was green through it. The question count had the same shape: written, never compared, so a golden set that silently lost its hard questions would have scored better.

MEDIUM — the "cheap deterministic embedder" was not deterministic. HashingBackend hashed each word with hash(), which Python randomises per process for str unless PYTHONHASHSEED is set — and nothing sets it, nor can a conftest.py, since the value is read before the interpreter starts. Which words collided in the 64-dimensional space changed run to run: one failure in 40 runs before, zero in 60 after switching to zlib.crc32. It surfaced only because a newly written test tripped over it once. A fake that cannot reproduce itself cannot tell a real regression from its own noise (v0.1 rule 5).

The transferable lesson. All three survived because the tests asserted that the machinery ran, never that it could detect. The mutation pass is what caught them: four mutants — hit ignoring hops, the by_kind comparison, the question-count check, and the golden-set consistency assertion — were introduced deliberately and all four killed a named test. Green proves the tests ran; only breaking the code on purpose proves they can see.

Shared-file contention tooling (20260729 04:06)

HIGH — git status --porcelain's leading space is significant, and a helper doing .strip() on the whole output silently ate it. The overlap gate's git() wrapper returned proc.stdout.strip(), which is correct for merge-base and symbolic-ref and wrong for status --porcelain: a modified file is reported as M CHANGELOG.md with the status in columns 0–1, so stripping the output removed the first line's leading space and the path parsed one character short — HANGELOG.md. It matched nothing, and the gate reported "no overlap" with total confidence. Exactly the one failure a contention gate cannot have.

Two things about how it was caught, both worth keeping:

• The tests drive real git against real temp repositories, not a mocked subprocess. The gate is almost entirely a set of claims about git's behaviour — what diff A...B means, which commit merge-base picks, how status --porcelain spells a rename — and a mock asserts the author's belief about each of those rather than the behaviour. A mocked test would have returned " M CHANGELOG.md" from a fake and passed with the bug present.
• The mutation pass re-introduced this exact bug deliberately and confirmed the right test fails. git() is now documented as trailing-newlines-only, with the reason, because the next person to "tidy" it back to .strip() will find nothing obviously wrong.

MEDIUM — a clean auto-merge is not a correct merge, and only the loud half of that was being managed. Three parallel branches edited CHANGELOG.md, docs/STATUS.md and docs/DESIGN.md inside one hour on 20260729. CHANGELOG.md conflicted and was resolved by hand; the other two merged silently, because the edits landed on different lines. Git merges edits that do not overlap textually, never edits that agree — so two agents can leave one document contradicting itself with every command reporting success, and no conflict resolution however careful would surface it.

The response is deliberately in two layers, because one does not cover the other's cases:

• changelog.d/ and retro.d/ remove the cause for the two documents every change must write to — separate files cannot conflict, so for those the class stops existing.
• tools/shared_file_overlap.py reports what remains, which is the living documents (docs/STATUS.md, docs/DESIGN.md) that fragments do not suit, because they are edited in place rather than appended to.

MEDIUM — splicing produced two ### Added headings under one ## [Unreleased], and only cutting a release revealed it. The tool inserted each rendered ### Category block under the anchor without looking at what was already there, so a section that already carried an ### Added from unmigrated prose ended up with two. Keep a Changelog expects one heading per category, and a reader scanning for "what was added" stops at the first. Fixed by merging into an existing heading when there is one, bounded to the anchor's own section so a shipped release's ### Added is never written into.

Worth keeping for the reason it was missed: test_apply_leaves_existing_unreleased_prose_exactly_ where_it_was was written deliberately, and it passed — leaving existing prose alone is correct. The case it did not imagine is that the existing prose has its own category headings. A test written by the reasoning that wrote the code inherits its assumptions, which is the same increment-shaped blind spot CLAUDE.md already names; here the escape was dogfooding, not mutation testing, because the mutation pass only ever perturbs cases somebody already thought of.

Separately, and pre-existing: [Unreleased] had accumulated seven category headings by hand over several days — two ### Added, three ### Changed, two ### Fixed. Consolidated when cutting 0.3.0, with a check that every non-heading line survived the regrouping.

LOW — the fragment tool takes --repo so its tests can drive the real artifact. Importing a tools/ script from a test needs sys.path surgery that pyright and ty then cannot resolve — ty failed the build on exactly that. Running it as a subprocess follows the precedent tests/test_paid_path.py set for tools/paid_path_gate.py, and tests the same artifact check.sh runs, argument parsing included.

I8 — Page citations on both surfaces, and pnk doctor's text yield (20260729 04:55)

HIGH — pinakes_get on a PDF crashed, and no test could have caught it. document() read the source with read_text(encoding="utf-8") inside a try guarded by except OSError. UnicodeDecodeError is a ValueError, so the guard never applied and the traceback escaped through the MCP surface. It survived since v0.1 because no test ever called pinakes_get on a PDF — the serve suite's KB is two markdown files, and every PDF test lives in a module that never builds a server. A gap between two test modules is invisible to both.

HIGH — the plan's page_start == p assertion is wrong, and would have failed on correct code. The I8 draft specifies that every chunk covering the traced offset "reports page_start == p". A chunk that straddles a page break starts on the earlier page, so a word on the later one legitimately sits inside a chunk whose page_start is smaller — which I5 explicitly allows and which the citation renders as p1-2. The trace asserts page_start <= p <= page_end instead. The draft had already corrected "exactly one chunk" to "at least one" for the same fixture; the page assertion needed the same correction and did not get it.

MEDIUM — the stale_extraction row understated its own gap by half. DESIGN §4.7's pending amendment said the marker "today reaches the CLI's Passage but stops there", so I8 would carry it to the agent surface. It reached the CLI's Passage object and was then dropped by the CLI renderer too — computed in search.py, surfaced nowhere. A field that exists in a dataclass reads, at review time, like a field that is displayed.

MEDIUM — the free per-page yield lived inside the only module allowed to import anthropic. survey_free_yield measures what pypdfium2 got out of a page; nothing about it is paid. But it sat in extract/claude.py, so pnk doctor — a free command — could not consume it without importing the paid path to ask a free question, against CLAUDE.md's own "never probe a backend by loading it". Moved to extract/pageyield.py. The alternative, a second per-page loop in doctor.py, would have been a second definition of a measurement that decides whether to spend money.

A dead statistic in a shipped template. The notes template's [budget] comment told every new KB that "no shipped code path spends money" — written when that was true, still shipping three releases later. docs/GUIDE.md said the paid extractor was "built but in no release yet". Both are the same failure as the four README claims found at 0.1.2: prose drifts toward the design, because the design is what you are thinking about while writing it. Neither was in the increment's scope; both were found by reading the files the increment touched for other reasons.

Mutation testing found the test whose name was stronger than its assertion. Twelve of thirteen mutations were detected. The survivor deleted pnk doctor's unmeasured-document tally, and test_a_swept_cache_entry_is_counted_as_unmeasured_rather_than_as_a_pass stayed green — because that test sweeps the whole cache and reads a branch that counts documents rather than the tally. The mixed case, where some documents measure and others do not, is the one the tally exists for, and it had no test. Its name claimed the general property; its body tested the degenerate one.

The review pass over I8's own diff

Three defects, all in pnk doctor's new check, all found by reading it adversarially rather than by any test:

HIGH — the health check crashed on an unhealthy KB. is_paid_backend raises BackendUnknownError on a name it does not recognise, and the check passed it every PDF's recorded backend. A KB indexed by a newer Pinakes, or with an extra since uninstalled, would make pnk doctor itself raise — the one command someone runs because their KB is in a state they do not understand. §4.4's coherence check has carried the identical guard, with the identical comment, since I5; the new code was written beside it and did not copy it.

MEDIUM — a KB whose PDFs are all paid-extracted got a permanent, unclearable warning, with a remedy (pnk sync) that on those documents spends money. The check deliberately skips paid-extracted documents, then reported the resulting empty measurement through the branch meant for a swept cache. Skipped-on-purpose and lost look identical to a counter.

LOW — a single out-of-range page bound was reported as a backwards range. page_start=5 on a two-page document read "pages 5-2 is not a range within it", because the bounds were validated after the omitted one was defaulted. It describes a range the caller never asked for, and reads as Pinakes' mistake rather than a bad argument. Found by running the tool, not by reading it.

What the tests could not have caught. All three needed either a KB state no fixture builds (an unknown backend name, wholly paid extraction) or a human reading an error message. The increment's own tests were green throughout, and so was a sixteen-mutation pass — mutation only perturbs cases somebody already thought of, which is the same limit that let the fragment tooling ship a duplicate-heading bug at the 0.3.0 release.

I9 — Auditing the verification table (20260729 05:40)

HIGH — the table that verifies everything verified nothing. plans/20260727_1543-v0.2.md ends with 98 rows, each promising a property and naming the test that holds it, under a preamble reading "a promise in a section with no owner is a wish". 61 of the 98 test paths did not resolve. Not because the properties went untested — nearly all are tested, usually under a better name than the plan guessed — but because the paths were written before the tests existed and implementation renamed them.

The failure is not the renaming. It is that nothing ever read the table, so it could drift a row at a time for nine increments with every gate green. A table of test paths is prose until something executes it, and prose about tests reads exactly like tests. The fix is not a better table: it is tests/test_verification.py, which resolves every reference in docs/VERIFICATION.md and fails on the first one that does not exist. The document can now go stale exactly once — in the commit that breaks it.

The audit found a real gap on its first run, which is the argument for doing it. test_every_v02_check_appears was assigned to I8, named in the table, and never written. Writing it (as test_every_doctor_check_is_exercised_by_a_test) immediately found five pnk doctor checks with no test at all: template, reranker, model cache, extensions, links. Link coverage is a §6.2 promise; the reranker check exists so a health check does not download weights. Both had shipped untested since I11.

MEDIUM — I wrote the exact CI assertion the plan warned against, and only running it caught it. The plan says the core-only wheel smoke must use "a core-only KB that does not need embeddings, because today's smoke KB fails on sentence-transformers long before it reaches an extractor, so the assertion would prove nothing". I built a PDF-only KB believing that satisfied it, and it does not: the embedding backend loads before any extractor, so pnk sync on a PDF-only core install still fails on pinakes[st]. My grep -q 'pinakes\[' passed — against the wrong extra. pnk doctor is the only surface that reaches the extractor question on a core-only install, because it reports a failing backend as a check and carries on. Reading the plan's warning was not enough to avoid the thing the plan warned about; running the command was.

A plan is a historical record, and correcting it would have destroyed the evidence. The temptation was to fix the 61 paths in place. That would have erased the only proof that predicted test names drift — and with it the reason tests/test_verification.py needs to exist. The plan keeps its predictions under a dated supersession note; the resolved mapping lives in docs/.

A sidecar that would not parse was replaced by a freshly minted one (20260729 07:26)

HIGH — the one failure the design says is unrecoverable, shipped since v0.1 and live in 0.4.0 on PyPI. walk_sources dropped a sidecar it could not read (except PinakesError: continue, sync.py:385) so that one bad file would not stop the walk. That was right. What it did not account for is that the document then matches DESIGN §6.4's "new path, no sidecar" row — and the mint path wrote a freshly minted sidecar over the file still holding the document's permanent ULID. Every inbound pnk:// link points at the id that was destroyed, and there is no migration machinery by design.

Three things made it invisible:

• pnk sync reported success. report.ok was true, failures empty, 1 indexed.
• pnk doctor afterwards reported sidecars: N readable, duplicate ids: none, failures: none — every check green, because the unparseable file no longer existed. The skip site's own comment said "reported by pnk doctor"; that safety net could never fire, because syncing repairs the symptom by destroying the evidence.
• The module that owns the risk had already named it. sidecar.write's atomic-rename comment calls ULID loss "the one failure in this module that no later command could repair" — and then handed the file to a caller that overwrote it deliberately. A guard written against a torn write says nothing about a deliberate one.

How it was found, and what that says. Not by a test — by hand-authoring L1's partner corpus with one deliberately unresolvable link, syncing it, and noticing that pnk doctor reported 10 links where the density gate had just counted 13. The discrepancy was three links, all on one document, and that document's sidecar had a new ULID and a created stamp from the sync. A second, independent count of the same population is what exposed it; every check that read only the post-sync state agreed with itself. L7 requires the gate's number and doctor's number to be the same population for a different reason — so that a user and CI cannot disagree — and this is the argument for computing both at all.

The fix, and one guard that was removed for failing its own mutation test. Minting goes through a new sidecar.create, which refuses where a file exists; the refusal lives at the write rather than in the caller, because "the only caller that reaches it" is a property of today's code. A matching guard added to the --index-only branch of _mint proved undetectable by mutation — deleting it changed no observable behaviour, only which of two SidecarErrors was reported, because the indexing path re-reads the sidecar for its metadata and that read refuses first. It was removed rather than kept, and _mint's docstring records why, so a later reader does not "restore the missing check". A guard that cannot be mutated is not a guard; keeping it would have been the kind of decoration this project's mutation step exists to catch.

The adversarial pass found the bigger half. The fix as first written covered only the case where the document is absent from the index — a fresh KB, a fresh clone, a --rebuild. For a document already indexed whose content is unchanged, pairing yields RefreshMetadata, and that branch sits outside _apply's per-document try, so _refresh_metadata's re-read of the sidecar raised straight through _apply, the action loop and sync(). One hand-broken file aborted the entire corpus: no failures row, no set_meta, no commit, and every document after it unprocessed — contradicting this module's own opening promise and docs/CLI.md's "failures are recorded, the run continues". That is the likeliest route in: edit a link by hand, re-sync. Three paths existed for one cause (Mint, Reembed, RefreshMetadata) and each behaved differently; they now report identically. The lesson is about where the first fix stopped: it was written against the reproduction, and the reproduction was a fresh KB because that is what a corpus author happens to have. A fix aimed at a repro covers the repro's path.

Two smaller things the same pass caught, both about honesty rather than correctness. The refusal said only "already exists, so a freshly minted sidecar cannot be written over it" — which reads like a Pinakes bug (of course it exists) and says nothing about the character the user mistyped, while DESIGN, the changelog and the commit message all claimed it named the parse error. The walk has to swallow that error to keep walking, so the mint path now re-reads the one file to recover it. And the remedy said "repair the file rather than deleting it — it holds the permanent ULID", which is false for the second shape the tests deliberately parametrise over: id: not-a-ulid has no ULID to repair to, and a user in a blocked pre-commit was being told not to do the only thing that unblocks them.

What the tests are parametrised over, and why. Two unrelated parse failures — a malformed link URI and a malformed id. The defect is any PinakesError from read_sidecar reaching the mint path, and a test written only against a bad link would have gone quiet the moment link parsing moved.

L1 — The partner corpus and the density gate (20260729 08:47)

HIGH — the gate did not gate the one shape it was built for. degrees was keyed by basename (path.name), so two documents sharing a filename in different folders collapsed to one key and the later-sorted one overwrote the earlier. Demonstrated against the shipped gate: a degree-6 hub — 50% above the cap of 4 — behind docs/aaa/policy.md exited 0 and was reported as "worst degree 1 (policy.md)". Density alone permits one hub wired to everything; the degree cap exists separately precisely to catch that, and a basename key is the single way it cannot. Now keyed by path relative to the KB root. The committed corpora are flat, so nothing in this repo would ever have exercised it — the fixture had to be built to find it.

HIGH — the gate counted sidecars where it meant documents, and was wrong in both directions. An orphaned sidecar (which pnk sync deliberately keeps) inflated the denominator: 8 of 10 real documents linked read as 27% and passed a 35% cap. A document whose sidecar had not been minted was invisible, so the gate reported nonsense on any KB where sync had not run. Documents now come from [sources] include, which is what the word means.

HIGH — two documents kept a privacy claim the increment made false. README.md still said "The sole KB here is a small synthetic corpus" and CLAUDE.md "The only KB here is the synthetic demo corpus", while DESIGN.md was updated in the same commit to "the two synthetic corpora". The repo contradicted itself about what had been committed, in the section a reader consults because they are worried about exactly that. The audit-the-neighbourhood rule exists for this and I applied it to DESIGN alone — the file I was already editing — which is the failure mode the rule describes.

A claim that was true by coincidence, stated as if by construction. The gate's docstring, check.sh and the changelog all said it "counts the same population pnk doctor reports". The link counts are the same population, by construction. The document counts are not: doctor counts indexed documents, the gate counts files matching include. They agree on the committed corpora and nothing makes them. The wording now says which half is guaranteed.

Two gates and no test that either still exists. L1 added a check.sh gate and a CI job and asserted neither, so deleting either left the suite green — in a repo that already has test_check_sh_declares_the_pdf_quality_guard, written for that exact failure. The convention was there; I did not apply it. Both are pinned now, and CI's negative step additionally greps for the message rather than accepting any non-zero exit, since a crash, a missing corpus, or uv itself falling over all satisfy the weaker check.

What the corpus taught about relations. counterpart was used both as a reciprocated 1:1 pairing (inward loan ↔ outward loan) and as a loose association (courier requirements → outward loan). A later increment reading counterpart as a pairing would be misinformed by its own fixture. Now governs.

The self-form fixture is not a fixed point of the product's own writer. sidecar.write resolves self to a ULID on write, so anything that reads and rewrites that file destroys the trap L2 needs — and pnk link (L6) writes exactly that key. The test catches it, but a long way from the cause, so the hazard is named in the test rather than left to be rediscovered.

Mutation, twice. Seven targets before the review, seven after the gate was rewritten. Two mutations appeared to survive and were worth more than the ones that failed: the first had not applied at all — a str.replace searching for 'self' where the source says "self", the exact no-op conftest._rewrite refuses, met in my own mutation harness, which now asserts the substitution happened. The second was real: nothing asserted that the report prints the cap in force rather than the module default, so --max-density 0.1 printed "27% of the 35% cap" and then failed the corpus in the next line.

And the verification gate caught its own author. Renaming test_the_committed_split_is_what_pnk_doctor_counts (misnamed — it never consulted pnk doctor) turned tests/test_verification.py red until docs/VERIFICATION.md was updated with it. That is I9 working on the first increment after it shipped.

L2 — Reverse-scan (20260730 16:51)

One root cause behind all three HIGH findings: I bypassed manifest.load and kept none of what it was doing. The bypass is right — a partner may run a newer Pinakes whose manifest mentions keys this one has never heard of, and refusing to read a neighbour's inbound links over that would make every connected KB a version dependency of every other. But load is not only a parser; it is also the place that rejects an absolute [sources] roots, rejects .. in one, and validates the include patterns. Reading the TOML directly removed all of it and replaced none of it, and the partner's manifest is input this KB does not control. Every one of the three failures below is that same sentence.

A partner renaming its own docs/ silently deleted every inbound row it had. A roots entry that is not a directory was a quiet continue, so the walk yielded zero sidecars, reported complete=True, and the caller did exactly what it is written to do with a complete walk: delete and replace. Reproduced — rows 1 → 0, link_scan empty, last_scan stamped fresh, so the retry was suppressed for a full window too. This is precisely the mass deletion the complete flag exists to prevent, arriving through the one door the flag was not watching, and no "successful walk" test could ever have caught it because they all leave the partner's sidecars where they are. A missing root is now a walk failure with a reason.

The partner's exclude was ignored, while a comment claimed otherwise. sidecars_under's docstring said a document "whose document was excluded" contributes nothing; it read only roots and include. The shipped notes template stamps exclude = ["**/drafts/**"], so this is not an exotic configuration — it is the shape of every KB pnk init creates, and the scan was recording inbound links from documents the partner's own KB does not contain.

A partner's manifest could crash pnk sync on a git hook. The sidecars_under call sat outside the try, and Path.glob raises on patterns manifest.load would have rejected — NotImplementedError for a non-relative pattern, ValueError for an empty one. Both escaped sync() entirely. The module's central promise is "nothing here raises", precisely so a partner that is merely broken cannot block a commit; the one call that could raise was the one left outside.

Two tests that could not fail, both of mine. test_the_partner_is_never_locked asserted the partner had no .pinakes/ — on a fixture where the partner was never synced, so the directory had never existed. It proved nothing was created and nothing whatever about locking; it now holds the partner's real SyncLock while the local sync runs. And a test asserting no SQLite connection was left open re-asserted pre-existing sync() behaviour (_run's finally: close() always releases it), so no L2-shaped defect could have made it fail. Deleted rather than kept: a test that cannot fail is worse than no test, because it is counted.

A failed local run blamed the partner. known_documents is read from the index, so a document that failed to index this run is absent from it — and a genuine inbound link was then reported as pointing at a document this KB does not have. It does have it; it failed to index it. The local picture is now passed as None on a failed or budget-stopped run, which suppresses that check without touching the rows, since the rows come from the partner and owe nothing to our state. _run already guards active_content_hashes on report.ok for the same class of reason — the precedent was there.

Dead code that credited itself with someone else's work. ScanResult.delisted and the known_kb_ids parameter were computed every sync, complete with a docstring explaining that the rows "are removed" — by a function that never read either. The sweep is store.forget_reverse_links, which takes the manifest's ids directly. Removed, along with the SELECT DISTINCT that fed it.

Mutation: 11 targets before the review, 5 more after, all detected. The one apparent survivor was equivalent code rather than a gap — taking src_kb_id from the declared id instead of the partner's own is indistinguishable wherever a row is written, because the mismatch guard refuses first. The guard is what carries the weight, so the test asserts what makes the assignment moot: a mismatched id writes no rows and no kb_refs entry.

And a test premise of mine was wrong, which the failure said plainly. _replace_links only runs for a document that gets an action, so the reverse-then-authored ordering needs the document to actually change — a second sync skips everything and rewrites nothing. Worth keeping because it is a fact about when authored links are re-asserted at all, not just about this test.

L3–L4 — The traversal core and pnk links (20260730 18:06)

Four HIGH findings, all in the properties the increment's own prose claimed loudest. That is the pattern worth keeping: the module docstring argued at length for double-capping, precedence and server-side clamping, and each of those three was where the defect was. Writing the argument down appears to have substituted for checking it.

The response was half-capped. max_rows and token_budget gated neighbours; the frontier was appended to unconditionally. Measured: a caller asking for one row received 1,000 frontier entries — and the frontier is the part an agent parses to decide what to ask next. Now capped, and ordered so that entries about nodes you did not get come first: capping without that ordering let the depth notes of accepted nodes fill the whole budget and crowd out every rows note, so a caller asking for 2 of 5 was told nothing about the 3 it missed.

"Every bound is clamped server-side" was true of two of the four. max_rows=10**9 returned 3,660 rows with an empty truncated. Three documents said otherwise. Once this is reachable over MCP the caller supplying max_rows is the untrusted party, so the sentence was not merely inaccurate.

A frontier entry contradicted the answer beside it. A node dropped by fan-out at one hop and reached at another kept its fanout entry — while sitting in neighbours and having been expanded. FrontierEntry's own docstring says "discovered and not expanded". Stale drops are now retracted at return; terminal and depth are kept, because those describe accepted nodes deliberately not expanded, which is the contract rather than a contradiction of it.

Half the stated precedence was inverted. The row and token checks ran before terminality was consulted, so a terminal neighbour dropped by the row cap reported rows — inviting a retry with a smaller request, which cannot help. Of the ten pairs the declared order implies, five were backwards and exactly one was tested: the one the code happened to honour.

The gate had three separate ways of being vacuous, and its docstring was an essay about gates that cannot fail.

• It passed against a traverse() that returned an empty Result — every check was one-sided, so zero neighbours satisfied all of them. Now equalities.
• It imported MAX_DEPTH and MAX_ADJACENT_K from the code it gates and compared them with themselves. Raising the caps to 10 and 150 moved the walk and the gate still passed, while docs/MANIFEST.md went on promising 64. The documented numbers are now literals in the gate — a second copy, which is the only thing that makes a silent change show up.
• It had no negative check, in a repo where the immediately preceding increment added one to its sibling job and a test that guards it. Added, with a --expect-depth override so CI can drive the gate into failure on purpose and assert the stated reason.

Two more the same pass found. The row cap truncated by parent-expansion order while ranking was per-parent, so a top-ranked neighbour behind a low-ranked parent lost to a worthless one in front of it — the same mistake as truncate-then-rank, one level up. And node-level row dedup silently dropped a second distinct relation to the same target, in a module whose contract is that a fact about the graph is returned rather than dropped; rows are now deduped per edge while expansion stays per node.

A dead sort term with a docstring defending it. _rank sorted by (-weight, distance, node_key) and explained that a nearer neighbour of equal weight ranks higher. _rank is called with one hop's candidates, so distance was constant in every sort. Removing it changed nothing — which is how it was found, and is the argument for deleting rather than believing prose.

A test that could not hold its name. test_depth_counts_logical_hops_not_physical_edges had no hub in its fixture and its own docstring conceded the core never sees one; it was a second copy of the clamp test wearing a larger claim, and docs/VERIFICATION.md cited it for a promise it could not carry. Renamed to what it actually checks. The logical-hop promise belongs to the provider that composes hubs.

And new behaviour shipped without tests. [retrieval] adjacent_k and _toml.integer(maximum=) had none — the commit message's claim that a value above the cap is refused rather than clamped was asserted and never executed, against this project's own rule that tests ship in the increment that introduces the behaviour.

A process failure worth recording separately. L4 was built in L3's worktree while L3's adversarial review was still reading it, so the reviewer found the tree dirty with a parallel increment's work and had to run every probe against a copy. It cost the review nothing this time because L4 added files rather than editing traverse.py, but that was luck. One increment, one worktree, and the review finishes before the next one starts.

Four silent str.replace no-ops this session, one of which spliced a new import into the middle of an existing one and produced a nonsense symbol. It is the same failure conftest._rewrite exists to refuse, met in editing rather than in a fixture. Non-trivial edits now go through a tool that errors when its anchor does not match.

L5 — pinakes_links on the MCP surface (20260731 11:29)

The defect was in the field nobody thought to assert

L5's own mutation pass killed all three of the targets the plan named. An adversarial review then mutated eight more payload fields and watched every one survive the full 887-test suite. Two of those were real defects, not merely untested:

• direction was keyed by node, while a row is (node, rel). Given a --related--> b and b --cites--> a, asking about a reported the citation as running from a — the opposite of what someone wrote. Shipped in L4, copied verbatim into L5, wrong on both surfaces. The provider's own docstring argued the case for the key it used: "Keyed by node, because a node reached both ways is still one neighbour" — true of the node, irrelevant to the row.
• DIRECTIONS was defined and never enforced. edges_of tests in ("out", "both") and in ("in", "both"), so direction="outbound" ran neither query and returned a confident empty answer with a "no links from here" hint. argparse choices covered the CLI; the MCP surface, the one an untrusted model types into, had nothing.

The lesson that generalises: a field with no assertion is a field that can be a constant. Ask of each one, "which mutation would this catch?" — not "is it correct?". scored_by_query, the field L3's docstring calls load-bearing, could be frozen to True; unresolved, whose contract says "returned, never dropped", could be frozen to [].

A tidy fixture defeats a mutation test. Three fields survived even after tests were written for them, because the KB-backed fixtures were too clean: the fake embedding backend's vectors are orthonormal, so every cosine is exactly 1.0 and deleting round() changes nothing; nothing hit a response cap, so truncated could be frozen empty; no frontier entry sat past distance 1. The fix was to build the dataclasses directly and take the fixture out of the question.

Two copies of one payload had already drifted — the MCP frontier carried a distance the CLI's did not, scored_by_query reached only one of them, unresolved dropped a kb_id its sibling lists carried. Neither failed, because nothing compared them. They now share pinakes.graph.present, and a test asserts both surfaces project the same keys.

Calling a tool is not the same as exercising it. The free-path gate was strengthened to invoke pinakes_links rather than only list it — but the fixture KB had one document and no links, so the whole neighbour projection never executed. A raise SystemExit planted in that loop never fired. The fixture now authors one intra-KB and one unreachable-KB link, and the same probe fires.

The fix for a wrong answer produced a differently wrong answer, and the tests written with it could not see that either. Keying directions by (node, rel) was right; merging to both across expansions was not. directions accumulates over the whole walk, so an edge discovered while expanding an unrelated parent rewrote a row already emitted from the start — and a row's direction then changed with --depth, to exactly the untruth the fix was written to remove. Both new direction tests ran at depth=1, where the start is the only parent, so neither could reach it. A second adversarial pass found it by varying the one parameter the tests held fixed.

The generalisation: when a fix adds a rule, test the axis the rule is defined over. The rule was about which expansion a direction came from, and every test pinned a single expansion.

A third pass found no new defect in the traversal itself, and four in what surrounded it. The (node, rel) scheme was probed against a reciprocal pair, a mutual same-rel pair, each direction, a self-loop, a 3-cycle, a node reached at two different hops by different relations, a node reached by two parents in one hop with opposite directions, and a node dropped by fan-out then re-reached — all correct. What was still wrong sat one layer out: an assignment nobody asserted, a message worded from the wrong end, a branch ordered ahead of a better one, and an assertion satisfied by a substring.

Two of those are worth naming as patterns:

• assert "-> related: b" in output passed on <-> related: b. A substring assertion over rendered text will match a longer glyph containing the shorter one, so dropping the outbound arrow entirely left the test green. Match whole lines when asserting on human output.
• Splitting f(x, scores=s) into f(x); f.scores = s moved a value out of the type checker's reach. The construction was covered by the tests that built providers directly; the assignment was covered by nothing, and deleting it disabled query ranking with every gate green. When a refactor turns an argument into a mutation, the mutation needs its own assertion at its own call site — and there were two call sites.

Left for the graph release (L3 core, predating this increment, found while probing): a node dropped by fan-out at hop 1 and re-reached at hop 2 is emitted with distance: 2 although it is one authored hop from the start; and a self-loop (a --sameas--> a) is dropped entirely — not a neighbour, not unresolved, not on the frontier.

A fix applied to one surface is half a fix. Round 3 gave pinakes_links the rule that a narrowed walk reports the narrowing before it reports dangling links — and left pnk links branching on unresolved alone, in the same commit, so the CLI told a user their links "resolve to nothing" about a document with a live neighbour one dropped --rel away. Both the docs and the changelog described the MCP behaviour as though it were both. The two surfaces now share present.is_filtered and present.arrow, which is the only way this stops recurring: the rule has to live in one place, not be applied twice.

A remedy in an error message is a claim, and it was false. The dangling-links hint sent the caller to pnk doctor — but doctor._links inspects only the destination side of local sidecar rows, so when the missing endpoint is the link's source (a deleted document whose outbound rows survive the soft delete) doctor reports links: OK and contradicts the message that sent you there. Dropped the clause; extending that check belongs to L7, which owns doctor's link coverage.

Four review rounds, each finding real defects in the previous round's fix, then converging: 11 findings, then 11 with one HIGH, then 7 with none, then 5 with none. What the last two rounds found was never the traversal — it was the layer around it: an assignment nobody asserted, an assertion satisfied by a substring, a message worded from the wrong end, a branch ordered ahead of a better one, and a rule applied to one of two surfaces.

The rule two rounds were spent getting right had no test that could detect its inversion. Round 5 found the shipped behaviour correct on both surfaces and the precedence — filter before dangling before "no links" — freely reversible with the suite green. The cause was a fixture that could not make both conditions true at once: --rel narrows provider.unresolved as well as the neighbours (edges_of receives the same rel), so a rel-filtered call leaves unresolved empty and the branch being out-ranked never competes. --direction is the lever that does it — an outbound link that dangles and an inbound one that is live. The assertion that named the defect in its own message ("one dropped argument away from a live neighbour") was the vacuous one.

Test the discriminating case, not the two sides separately. A precedence rule is only observable where both branches are eligible; a fixture that satisfies one at a time asserts the wording of each and the order of neither.

L5b — ruamel.yaml replaces pyyaml in the sidecar (20260731 11:29)

Swapping a YAML library is not a swap

The plan predicted three failures precisely, and all three landed as written — which is worth recording because it is the first increment in this project where that happened. It named the 872nd test ({id: x, : }, which ruamel parses, so the case fell through to the id check and the parse-error branch had been asserting nothing), the free-path gate being red on day one, and the ScalarBoolean coercion being insufficient at one level.

The free-path gate was defeated by its own harness, and I wrote the defect. _author_links in tests/free_path_run.py — added in L5 to close a coverage hole — wrote a sidecar through yaml.safe_dump. That put yaml into the very module list the gate inspects, and it was also the last PyYAML sidecar writer in the repo: a fixture written by one library and read by another, which is exactly the divergence the gate exists to forbid.

existing[:] = keep wipes ruamel's comment metadata outright. Reconciling a sequence by rebuilding a keep-list destroys CommentedSeq.ca.items entirely — every comment in the block, not just the removed entry's — while del existing[index] shifts the survivors. Measured: {} against {0: '# first', 1: '# third'}. Both merge functions had it.

A comment before a sequence entry belongs to the entry above it, exactly as it does for a mapping key. The plan pins the deletion limitation for mapping keys; it is broader than that. After deleting the middle of three commented links, # first stays correct, # second reattaches to the third link, and # third disappears. The surviving links are all correct; the prose beside one of them is not.

"Unobservable" was the wrong conclusion; "observable only where something else is broken" was the right one. The plan's "assign a known key only when its value actually changed" looked untestable — every known-key value is the node read out of the document and written straight back, so a write of an unchanged document is already a no-op. Two attempts at a test passed against the mutated source and I wrote that no mutation of it could fail. A reviewer then removed the rule and the committed corpora stopped round-tripping: the short-circuit was masking the duplicate-link defect below, not proving its own redundancy. Once that was fixed the rule really was unobservable — but the claim was true by accident for two commits, and the difference is exactly what an adversarial pass is for.

-x makes a mutation look like it was caught by the wrong test. Two links mutations appeared to be killed only by an unrelated pre-existing test; without -x both were also killed by the test written for them. Run the mutation pass without early exit, or the report is about test ordering.

A merge key must be the identity the storage layer already uses. Reconciling links on to alone looked sufficient and is undefined the moment two links point at one document with different relations — which _links() accepts and the index stores as two rows, its primary key including rel. Measured on the version that had it: dropping an unrelated third link rewrote the first link's rel to the second's and deleted the second, leaving one row carrying the wrong relation under the other's comment. The index's own PRIMARY KEY was the answer, and it was already written down in store.py.

A recursive rule needs a depth bound as much as a base case. "A key absent from the new mapping is deleted" is required at the top of provenance — or --force leaves a false paid claim behind — and destructive one level down, where with_extraction_provenance builds a plain four-key replacement and the user's own reviewed_by sits beside content_hash. One sentence, two opposite correct answers, distinguished only by depth.

The exit criterion was the thing nobody ran. The plan's one falsifiable sentence — every committed sidecar still round-trips — had no test, and running it by hand found a pnk://self/… entry in partner-kb being deleted, rebuilt without its unknown per-link keys, and moved to the end of its block. _links() expands self on read, so the loaded entry's raw to never equals anything in the reconciliation set. The docs bounded the invariant with "pnk://self/… expansion", which reads as the URI text changes and not the entry is rebuilt — a documented exclusion that quietly covered a defect.

Quoting was applied on the path that was tested and not on the path that ships. Decision 23's predicate reached the merge branch and the mint, but not the branch taken when a key first appears — which is the branch pnk link will follow on a sidecar that has no links: yet, i.e. almost all of them. Three quoting mutations survived the whole suite.

An error message is part of the interface. Three of this increment's breaking changes surfaced as TaggedScalar, ScalarFloat and OctalInt — ruamel class names, from a library the user never chose, with no remedy. The type is not what they need; "quote it, or drop the tag" is.

A fix instruction can carry its own defects, and two of pass 6's did. Keying links on the (to, rel) pair made the pair the entire content of an entry, so no matched entry was ever updated and every rel edit became a delete plus an append — landing straight in the comment misattribution the rule was written to avoid. And "positional fallback among equal pairs" was too vague to implement; what I wrote from it used a set, which collapses two identical entries: three links in, one out. The final rule needed three explicit clauses — resolve before comparing, multiplicity never a set, assign rel in place — each naming the shipped version that got it wrong.

"Exactly the call being protected" was true of the call and false of the argument. The JSON-encodability check ran json.dumps(extra, sort_keys=True, ensure_ascii=False) — the same function store.dumps_metadata calls — over extra alone. _metadata() hands that function {"tags": …, "provenance": …, **extra}. A uniformly int-keyed {1: a} sorts perfectly on its own and becomes mixed the moment the string keys join it, so pnk sync still crashed. Checking the parts is not checking the whole, and the docstring asserting otherwise is what made it look done.

Two tests could not observe what they claimed, for the same reason. A plain read-write of an unchanged document short-circuits before the merge runs, so a wanted that deduplicates survived test_two_identical_link_entries_both_survive and a set-based collapse was invisible. Any test of reconciliation has to change something first, or it is testing the short-circuit.

A warning is not an error, and a library that downgrades one is changing behaviour. A reused anchor name raised ComposerError — a YAMLError — before the swap, and after it the document loads, every alias resolves to the last anchor of that name, and the only signal is a ReusedAnchorWarning on stderr. Three consequences, none visible in a passing suite: the value silently changes; read()'s except YAMLError never sees it, because a Warning is not one; and under this project's filterwarnings = ["error"] it escapes as a bare warning traceback rather than a named error. Promoting it at the load makes the outcome independent of whatever warning filters the calling program happens to have set — which is the right place for a property of the file format to live.

An exclusion list is a set of claims, and claims rot. Every bound on the byte-identity invariant — indentation, !! tags, anchors, CRLF, BOM, document markers — was prose in a table until it was pinned by a test. Writing those tests measured two behaviours that were not on the list at all: a plain (non-recursive) anchor on an empty value is destroyed, where the list named only the self-referential case; and a file with no trailing newline gains one. Both are byte changes to a file nobody edited, which is exactly what the invariant claims does not happen. A bound stated only in prose cannot notice the library moving under it, and cannot be wrong out loud.

It also falsified a changelog line I had written: "!!int, !!float, !!bool, !!seq and !!map keep working — verified". Verified of loading; the tag itself is dropped on write, so !!int 3 comes back as 3. True of the value, false of the invariant, and the word "verified" is what made it read as covering both.

A gate that has never been shown to fail is a claim too. The AST scan now proves it sees all four shapes of a planted import — including the function-scoped one an import walk cannot reach — and does not fire on any of the four legal ruamel forms, every one of which contains "yaml". The stub signature test proves it can fail: writing transform into the expected set failed it, because transform belongs to dump rather than __init__.

"PyYAML left the runtime" is true of what Pinakes declares and false of what a user's machine has. Measured on a built wheel: bare, yaml is absent; pinakes[light] has it, transitively from huggingface_hub. starlette and uvicorn list it too, but only under an extra they do not pull. So the CI assertion is correctly scoped to the bare wheel — Pinakes never asks for PyYAML — and the consequence is the part worth remembering: import yaml will succeed in a real install, so a stray import in src/ would quietly work instead of failing loudly. That is what makes the AST scan load-bearing rather than a second belt.

The worst defect in this increment was a rule the plan itself wrote. "One instance, reused rather than reconstructed per call", justified by 282 µs against 399 µs. ruamel keeps the %YAML directive from the last load() on the instance and applies it to every later load and dump: read a sidecar carrying %YAML 1.1, then write an unrelated one that never did, and it comes back with the directive injected and country: NO rewritten to false. The exact corruption this increment exists to remove, reintroduced across documents, in exchange for 117 microseconds — and freshly minted sidecars are contaminated the same way. Nothing softer fixes it: resetting version after the load still emits the directive, pinning it up front is overwritten by the next load. A performance justification measured in microseconds should be read as an argument that the optimisation does not matter.

A gate that never reads the artifact it guards is checking a copy. The stub-signature test listed the symbols in a hand-written Python dict, checked them with hasattr, and compared against hardcoded signature supersets — so a stub declaring a parameter ruamel does not have was green under pytest and pyright, which is the single failure decision 20 exists to catch. It parses the .pyi files with ast now.

A fixture can be right for the wrong reason and hide the defect it was written for. The two-links-sharing-a-to test edited the first entry, and entries are walked in descending index order — so the single-pass form it was meant to catch happened to produce the correct answer. Editing the second entry is the discriminating case: its fallback claims the link the first was owed exactly, and both relations end up swapped under the wrong comments. Two of this increment's tests have now needed the specific case rather than a representative one.

main was red for four merges, and local check.sh could not have known (20260801 06:05)

Four tests written across L6 and L7 passed on macOS and failed on CI, so 2314dea (L6) and ed01b00 merged onto a red main and stayed there until L8's verification step 1 looked.

Two causes, one shape: a test that cannot build its precondition does not skip — it asserts the wrong thing.

• chmod(0o000) is not a portable way to deny a read. Three fixtures built an "unreadable directory" that CI read anyway: pnk link reported no pinakes.toml there where the test demanded Permission denied, and 'docs/locked/x.md' is not a document in this KB where it demanded cannot be read.

The first fix was wrong, and its wrongness is the lesson. It probed whether permissions are enforced and skipped when they are not — reasoning that CI runs as root. CI is not root: the probe reported permissions enforced, did not skip, and the run failed identically. Whatever that runner does with a mode-000 directory, is_file() neither succeeded nor raised EACCES.

Skipping was the wrong shape regardless. It disables the guard exactly where it broke — the environment the test could not model is the one that most needed testing. The refusal is now injected: Path.is_file raises PermissionError, which is precisely what the guard exists to catch, on every platform and with no filesystem semantics in the way. A test for "an OSError becomes a PinakesError" should raise an OSError, not arrange for the operating system to. - Two more of the same shape, found only by pushing the fix and watching CI again. A 300-character filename asserted to produce ENAMETOOLONG — the length at which a filesystem says that is a property of the filesystem, and on CI the name was simply not a document. And an embedded NUL in an include pattern asserted to raise from resolve() — on CI it raised nothing, so the test asserted a problem that never occurred. Both now raise the error directly.

Every one of these tests asked the operating system to produce an error, and then asserted on the answer. That is a test of the platform, not of the guard. The guard's contract is "an OSError/ValueError from this call becomes a PinakesError" — so the test should raise one.

• pathlib's wording is not a contract. A test asserted Unacceptable pattern: PosixPath('.'), which CPython renders as Unacceptable pattern: '' on other versions. The increment's promise is that the pattern the author wrote is named and the other include entries survive; that is what it asserts now. Third instance in two increments of asserting a phrase where the property was meant.

The process failure is the larger one. ./check.sh was green before each merge, and green on one developer machine is not green on the three-leg CI matrix — different OS, different Python patch, different privileges. The project rule already says to check whether the latest run on the default branch actually succeeded; it was not checked after either merge, and the second merge landed on top of the first failure without noticing it.

gh run list --branch main belongs in the merge sequence, after the push, not in the next increment's verification step. A red default branch blocks the release either way — finding it two merges later only makes the bisect longer.

G1 — Is the eval reproducible? (20260801 00:52)

Decision 15 said measure before fixing, and the measurement paid for itself twice: once by finding a real defect, and once by contradicting the fix that was nearly written for it.

HIGH — the eval was reproducible by luck, and the luck was invisible. Running the golden set, editing a document, re-syncing, rebuilding and comparing per-question outcomes: the real [light] models agreed everywhere. A low-dimensional fake disagreed on one question in 41 between an incremental sync and a --rebuild. Both facts are the same fact — 384-dimensional cosines almost never tie exactly, and every tiebreak underneath resolved to chunks.id, the rowid, which store.py's own schema comment says has no identity across rebuilds. A property that holds because the corpus never exercises it is not a property, and G5's sign test was about to be built on it. The fix is total ordering on (documents.path, chunks.ordinal) at three sites plus a stable argsort; no measured number moved, which is the right outcome for a change that only breaks ties, and is why this increment rewrites no baseline and needs no amendment to L8 step 5.

HIGH — the fixture was the algorithm, in the one place that judges the algorithm. The first version of the tests used eight dimensions, reasoning "fewer dimensions, more ties". Swept against the genuine pre-fix code, eight and sixteen dimensions both reported zero differences across all four perturbations: collapse the space far enough and every candidate ties, so the ordering underneath stops reaching the top-k at all. The relationship is not monotonic — 32 caught two perturbations, 64 caught one, 128 caught two. Had the sweep not been run, G1 would have shipped a green gate, a green test suite and a live defect, all three agreeing. The sweep is recorded in the gate's own DIM docstring rather than the conclusion alone, because the next person's intuition will be the same as this one's.

HIGH — the mutation harness deleted the thing it was measuring. The first mutation run restored each mutation with git checkout -- <file>. The fix was uncommitted, so the first restore reverted it; every later mutation then failed to apply to code that no longer had the target, and the suite ran green against original code four times while reporting "0 failures" as though the mutations had been survived. It read exactly like a well-tested change. Two rules fall out. Restore from a copy of the mutated-from state, never from HEAD — git checkout restores to the last commit, which is a different thing from "undo my mutation" whenever the work is uncommitted. And a mutation harness must assert that its mutation applied: the rewritten one fails loudly if the target string is not found exactly once, which is what turned three silent no-ops into an error.

MEDIUM — a stable sort needed its own test, and the obvious one was vacuous. kind="stable" changes nothing a repeated run can observe: on a fixed input array NumPy's introsort is deterministic. It changes what happens when the array grows, because partitioning depends on the whole array — measured at 500 of 500 random tie-heavy arrays reordering their original entries. The first test written for it used four tied chunks and passed under the mutation, because NumPy uses an insertion sort below roughly sixteen elements and insertion sort is stable whatever kind says. A fixture can be too small to contain the behaviour it is named after.

What the increment ended up asserting, and at which level. Three end-to-end tests state the property G5 needs, over the committed corpus and questions; four site tests each drive one ordering decision directly and are the mutation targets — one mutation, one failing test, verified for all four. The two levels are not redundant: the end-to-end tests can only observe ties the corpus happens to contain, which is precisely how the defect survived three releases.

The adversarial pass over G1's own diff (20260801 01:25)

Six findings, four of them real defects in work that was already green.

HIGH — a gate advertised a field it had retired. _plant rewrote the reranker's model name and left [retrieval.confidence] fitted_for naming the real one, so _confidence short-circuited and all 41 questions scored unknown. Both the gate's docstring and the tests claimed to compare the confidence label. Naming the reranker was not enough either: the committed thresholds were fitted on a real cross-encoder's logits and sit below every score the fake can emit, so the label became a constant high — still unable to move. Thresholds inside the fake's range give 35 medium / 5 high / 1 low, and the field is finally live. The class of defect matters more than the instance: a fixture that rewires half of a calibrated pair silently disables the thing it was calibrated for, and nothing fails.

HIGH — the plan still asserted what the measurement disproved. Decision 15 says a final tiebreak would be "a provable no-op" because cross-document ties are totalised by documents.path and rowid order is ordinal order. Both premises are true about writes and irrelevant to the output: documents.path cannot separate two chunks of the same document, and an incremental sync by definition does not rewrite the files it did not touch, so rowid order stops matching corpus order at the first re-chunked file. The plan is an executor doc; leaving that cell intact would have licensed a G2–G5 executor to skip a tiebreak for a reason this increment measured to be false.

MEDIUM — half the gate's sweep has never observed anything. Of its four perturbations, added and removed report zero differences against the genuine pre-fix code at every width swept (8/16/32/64/128), while edited and renamed bite. --inject-difference cannot reveal this: it corrupts all four alike. The gate now states it. A gate's own justification is a claim like any other — this one said "it sweeps four ways where the tests exercise one", and two of the four were along for the ride.

MEDIUM — the contract's file table was checked against the wrong question. It compared the two tracks' owned files and never asked what a new gate touches. Every gate edits check.sh, ci.yml and tests/test_check_script.py, which both tracks append to at the end of the same regions; and G1 necessarily edits search.py and store.py, which the table lists under neither track, because reproducibility is a property of core retrieval. Widened, with the reason.

LOW, and recorded rather than fixed — making the BM25 cut total costs a join: +11.5 ms on a 50k-chunk corpus where every chunk matches every term. That is the worst case a planner can be given, the correctness is not optional, and the number now sits in docs/STATUS.md so a later change can argue with it.

What the pass confirmed, having tried to break it: bm25() still resolves with the alias present and returns byte-identical rows; the join multiplies nothing (both sides unique); the load_vectors reordering costs nothing measurable; graph/provider.py, the other caller, reduces to a per-document max and is order-independent; the four site tests each fail against pre-fix code; and the artifact paths, cache keys and macOS wheels in the cross-machine job all resolve.

L6 — pnk link (20260801 01:41)

Every review commit on this increment found defects in the one before it, and most of them found the previous commit's own fix or claim rather than something it had missed — 3ce150e (review 1's containment fix had traded one defect for two), 986faf3 (review 2's fix was right; its stated justification was not), 7b3f0a3 (the escaping-error class sat one line above the try added for it), 9c8f667 (the totality fix re-anchored the walk on the working directory), cdee8d8 (the test for an untested branch entered it, but its assertion held either way), dbebd8b (a severity asserted, not measured), and the last three, which took four goes at one containment rule.

No total is given, deliberately. Three drafts stated one and all three were wrong, because it changes depending on whether 8b and 9b count as rounds of their own — and the last wrong figure was introduced by the commit correcting the one before it. git log main..HEAD is the answer, and it cannot go stale. What follows is the state after all of them, not a log: the rule is to rewrite to the current state rather than layer corrections, and earlier drafts of this fragment broke it four times — describing a concurrency scenario a later round had disproved, calling every self-link a typo after the fix for the other case existed, counting the rounds that had happened when it was written rather than the ones that had, and asserting a safety property ("Path.resolve() is safe at both call sites") that was wrong twice over: strict=False suppresses OSError, not the ValueError an embedded NUL raises, and there are six Path.resolve() sites across the two modules rather than two.

One defect class, six instances, and why fixing it at the call site produced them

HIGH. cli.main catches PinakesError. Anything else is a traceback on a user's terminal — or on an unattended post-commit hook. Six calls in this increment's blast radius raised something else:

1. Path("~nosuchuser/x.md").expanduser() raises RuntimeError, on <source> in link.py. It bought nothing either: a ~ that does expand lands in $HOME and is refused by the containment check on the next line. Copying a call across a boundary copies its justification too, and linkscan's need for it (a [[links.kb]] path may be ~/kbs/partner) did not survive the trip.
2. Path.is_file() ignores ENOENT, ENOTDIR, EBADF and ELOOP and raises everything else — so an unreadable parent directory (EACCES) and an over-long name (ENAMETOOLONG) on the same source path.
3. The is_file()/is_dir() pair one branch over, in _via_alias: a partner KB directory this user cannot read raised PermissionError.
4. resolve_path, on the line immediately above the try just added for (3).
5. The same three, in the module link.py calls into. linkscan.scan_one's docstring promises "Never raises: every failure comes back in issues", and all of them sat in the three lines that ran before any handling did — so pnk sync on a hook became a traceback. There since L2.
6. resolve_path again, bare in scan()'s freshness branch — which plain pnk sync takes, so a partner path that stopped resolving crashed every git commit inside the TTL. The branch had no test at all.

Fixes 1–5 each wrapped the instance in front of them and stopped. What closed the class was moving the guarantee into resolve_path itself — a guarantee three call sites each have to remember is a function with the wrong contract — which then removed the wrappers fixes 4 and 5 had added.

The first version of that fix introduced a worse defect than the one it closed, and this is the part worth keeping. resolve_path was made total: on text no filesystem call accepts it returned Path(raw), the declared text, so an error could still name what the author wrote. That value is relative, and five consumers use it as a filesystem base — (path / MANIFEST_NAME).is_file(), why_not_a_kb, partner_sources, sidecars_under, _doc_id_of. So the walk silently re-anchored on the process's working directory: the precise thing resolve_path's own first paragraph says it exists to prevent, reintroduced four paragraphs below by the round that wrote it.

With a directory of that literal name in the CWD holding a readable pinakes.toml, pnk sync walked the decoy, found nothing, stamped the scan complete — and replace_reverse_links deleted every inbound row the real partner had, with report.ok true and no issue raised. That is the real consequence, and it is silent data loss.

Round 8 also claimed pnk link would write the decoy's ULID into the real sidecar, permanently. It would not, and round 9 reproduced the refusal. _document_in compares an absolute joined.parent.resolve() against the relative root, which can never be is_relative_to, so it fires before any sidecar is read — 'docs/one.md' is outside `partner`, which tells the user the path they typed correctly is wrong and names neither the KB path nor the expansion failure. A message defect, not corruption.

Three things kept that overstatement alive for a round. The true account was already in the tree — link.py's own comment describes the misleading refusal — so the increment carried both versions at once. The regression test's docstring asserted the severe reading, and under the round-7 mutation it failed on its first assertion, so the two that encoded the severe claim were never reached: a test that fails proves the mutation is caught, never that it is caught for the stated reason. And the claim was written from the mechanism (a relative base re-anchors the walk → therefore the walk completes) rather than from running it. That is the same "prose written from the design" failure as the two documentation defects below, in a commit message and a retrospective — the two places where being wrong is hardest to notice later, because nothing executes them.

The answer is None, not a fallback value: text that names no path yields no path, and pyright makes every caller say what it does instead — a type-checked obligation rather than a remembered one, which is the same lesson one level up. The declared text is still what the message names; it was always available as linked.path, which every caller already held. A total function is not automatically a safe one — totality only moves the failure from a raise to a return value, and a return value that is the wrong kind of thing is harder to notice than an exception.

Four tests fail against the round-7 shape, verified by mutation — including the two written for it, though one of those for a different reason than its docstring gave (above).

A defect class is not closed until it has been searched for, and the search is mechanical: list every call in the module that touches the filesystem and ask of each which errno it swallows.

Path.resolve() belongs on that list and was wrongly excused twice. strict=False suppresses OSError; it does not suppress the ValueError raised for an embedded NUL, which tomllib accepts in a manifest and pathlib will not open. Enumerated rather than excused, there are six sites: _document_in (link.py:298) resolves a path built from user text and is now guarded and tested; resolve_path (linkscan.py:178) is the fix above; and sidecars_under has four — anchor, the roots entry, the pattern probe and the per-candidate check — all inside the caller's except (OSError, ValueError, NotImplementedError, PinakesError).

The enumeration is the point: "safe at both call sites" named neither the number nor the reason, so it could not be checked without redoing the work — whereas a count with line numbers is wrong the moment it drifts, and says so. It has drifted twice already: round 8 corrected an earlier version that called two of the sidecars_under sites partner-controlled when one is not, round 10's own fix added the fifth site, making "four" stale in the same commit that relied on it; and round 13's added the sixth the same way.

The containment check took three spellings, and the first two were each wrong in one direction

HIGH. _document_in decides whether a path names a document in this KB.

• joined.resolve() — the original — follows the final symlink before checking, so a symlinked document was refused as "outside this KB", with a remedy repeating the path the user had typed correctly. pnk sync indexes such a file, pnk doctor calls its sidecar readable and pnk links traverses it; only pnk link said it was not there, and nothing could link it in either direction.
• os.path.normpath — round 1's fix — follows nothing, so a symlinked directory under docs/ passed containment: the write went out of the KB through it, and in the other direction minted a permanent pnk:// to a ULID this KB will never index, because Path.glob does not recurse a symlinked directory. It simultaneously refused a legitimate absolute path whose ancestor is a symlink — the ordinary shape on macOS (/tmp → /private/tmp) and behind any symlinked checkout — because manifest.load resolves the root, so a verbatim comparison could never match it.
• joined.parent.resolve() / joined.name is right in both directions. The directory chain is followed, so an escape through it is caught and a symlinked ancestor lands inside; the final component is left alone, so the document's own symlink is irrelevant — which is correct, because Path.glob does yield a symlinked file.

normpath must also not run first: it collapses docs/link-to-elsewhere/../x.md to docs/x.md textually, turning an escaping path into one that looks contained. resolve() on the parent collapses .. after following the links it sits behind.

The docstring was wrong for longer than the code. Two drafts justified the check with "what decides membership is the path under [sources]" — a rule the check has never implemented, since it compares against the KB root. Round 2 quoted that sentence as the lesson and left it in place; round 3 found it still there. The residual it was papering over is now stated instead: a document inside the root but outside [sources] can be linked, and the link will not resolve until that document is ingested. Answering the [sources] question properly means re-implementing walk_sources including its globs, and refusing a "link it now, ingest it next" order of work that costs nothing.

Two documentation claims the code contradicted, in prose written from the design

HIGH. The new pnk link section told the reader that a pnk:// URI pointing at a KB not on this machine is fine because "pnk doctor reports a dangling target; pnk links lists it under unresolved". Neither happens. doctor.py filters its dangling list to this KB — the cross-KB check is L7, the next increment — and provider.py's unresolved carries a docstring explicitly refusing to widen: "a cross-KB target cannot be checked from here without the other KB, and reporting one as unresolved on that basis would be asserting something this index has no standing to know." A reassurance was invented for the one case the section was telling the reader not to worry about, and half of it described something the design had already declined to build.

The replacement prose then made the same mistake twice more, which is the finding worth keeping. Round 1's fix illustrated the missing lock with a post-commit hook firing a paid extraction — a scenario hooks.py structurally prevents. Round 2's fix replaced that with "the one sync that rewrites an existing sidecar is a paid extraction", which sync.py's --force-plus-free---extract override falsifies, and which this increment's own edits to DESIGN, MANIFEST and CLAUDE.md all name the carve-out for. A correction is a diff and earns the same verification as the line it replaces; three rounds of unverified prose about the same paragraph is what happens otherwise.

Fixtures that were representative rather than discriminating

MEDIUM, four times. A test can be green because the code is right or because the input never reaches it, and the two look identical from the outside.

• test_no_line_outside_the_links_block_changes_when_a_link_is_added used a sidecar with a populated tags: list, which write() short-circuits as unchanged and therefore never touches. It could not have failed. Meanwhile tags: and provenance: written with nothing under them were being rewritten to tags: [] and provenance: {} on every pnk link — two lines changed outside the block, in the increment whose test says none are, against a promise stated as byte-identity. Reachable before L6 only from a paid PDF extraction, which is why L5b's sweep missed it; pnk link reaches it on a first link, the common case. The sibling test_a_known_key_with_a_null_value_does_not_crash_the_writer parametrises exactly these three keys and asserts only "id:" in text: it pins the absence of a crash and nothing about the value.
• The embedded-NUL test put its NUL in the filename — docs/a\x00b.md — where only the parent is resolved, so it never reached the guard it was written for and passed against the ordinary "not a document" refusal. Moved into a directory component. Caught by mutation, not by review.
• assert "outside" in message against a fixture named outside.md, and assert "partner" in message against a tmp_path ending in /partner. Both were satisfied by the interpolated path, so the reason could have vanished from the wording with the test still green — proven by rewording the error and watching all 29 pass. Fixtures renamed, phrases asserted.
• A fixture stops reaching its guard when a later fix gets there first, and nothing says so. The ordering test for the containment check was retargeted twice — once when the static refusal was added, once when that learned to resolve the prefix — because each fix caught its input earlier, leaving the test green and its guard unexercised. Both times the mutation found it and the reading did not. Re-run the whole mutation battery after every fix, not only a mutant for the fix itself: a fix can silently disarm a test written for something else.
• The test written for the freshness branch — the branch a finding had just called untested — asserted only report.ok, which holds whether that branch runs or not. Proven by forcing is_stale to return True: the branch never ran and the test still passed. A skipped-fresh row carries no issue, so link_scan is the assertion that discriminates. Found by a reviewer, not by the round that wrote it, in the commit whose message called its other two fixes mutation-verified — "mutation-verified" is a per-assertion claim, not a per-commit one.

A docstring claiming a safety property its function cannot have

MEDIUM. _doc_id_of's owner argument was documented as preventing the pnk://self/… retargeting defect. It cannot: only .id is returned, so owner never reaches an observable — measured both ways, the mutation is caught by no test and the output against a partner sidecar carrying the exact retargeting shape is byte-identical. The protection is real but lives in linkscan.scan_one, which keeps the links it reads. A plausible rationale attached to the correct line is harder to catch than a wrong line, because reviewing it means re-deriving the claim rather than reading the code.

Mutation testing: a killed run poisons everything after it

HIGH, methodological. The first mutation run blew a two-minute timeout and was killed mid-mutation, so its finally never restored the source. The next run's pattern then failed to match the already-mutated file, reported "pattern not found", skipped — and that guard stayed disabled for all ten mutants that followed. The signature is unmistakable once known: one unrelated test failing on every mutant, including mutations that cannot reach it.

Two things made it recoverable: the disabled guard had its own test, so the failure was loud, and ./check.sh had been green minutes earlier, which dated the contamination. The fix is a baseline snapshot taken before the first mutation and asserted after every restore — not git diff --quiet, useless in the increment's own worktree where the source is legitimately dirty. Scope the run to the modules under test, too: the full-suite run is what blew the timeout that caused this.

Every fix was mutation-tested against the test written for it, and three escaped, each in a different way that "green" could not distinguish. The NUL guard had a test whose input never reached the line. The containment-ordering test stopped reaching its branch twice, when a later fix caught its fixture earlier. And review 14's next() guard had no test at all: the "" and "." written for it raise at the glob() call, not at the step, so the guard one line down was never executed — which review 14's own commit message called "eleven mutants, each killed by the right test", and this paragraph called "all but one".

The method is now: mutate every behaviour in the function, not the ones the diff touched. That standard was stated before it was met. The sweep that first claimed it covered three behaviours; an independent 47-mutant pass over the whole of sidecars_under and scan_one killed 33 and left 14 alive — 2 provably equivalent, 12 unpinned. Every one of the 12 was checked and the code is right in each, so they are coverage rather than defects, and they are listed rather than closed: the two halves of the exclude disjunction (a deliberate mirror of sync._excluded), the continue that bounds a pre-walk escape, .resolve() on anchor and on base, the is_file() and sidecar-suffix skips, the two sorted() calls, partner_sources raising, and LinkTargetMissingError's count.

Naming them is the point. A number for the battery is unverifiable afterwards — the runs leave no artefact — but which behaviours are unpinned is checkable by anyone who repeats the sweep, and that is what a later reader needs.

One mutant is genuinely equivalent: substituting the locally declared [[links.kb]] id for the partner's own when writing an alias target changes nothing, because the refusal above has already established the two are equal. Saying so is part of the result — the rule is enforced by that refusal, which is caught, and the docstring records it so nobody simplifies the variable away on the grounds that they are the same.

Green expires at the next keystroke

HIGH. ./check.sh ran green, then a docstring was reworded to 101 characters, then the increment was committed. Under set -e a failing ruff check means the eleven gates after it never ran either: the increment's own verification stopped at gate two, unnoticed, because the earlier green run was still in mind. The rule already says green-before-review; what this adds is that the run has to be the last thing before the commit, including after an edit to a comment.

A containment rule argued in prose and implemented for half its inputs

HIGH. sidecars_under reads a partner's [sources], and its docstring says why that input is untrusted: "without the same check here, a partner manifest could point the walk at any directory on this machine, and roots = ["/"] would be an unbounded walk on a post-commit hook." The check existed for roots. include is exactly as partner-controlled and had none, so include = ["../../outside/*.md"] walked out of the partner KB and this one recorded inbound links from files the partner does not own — complete true, so sync persisted them.

The line that looks like the guard is not one. candidate.relative_to(root) ran on every match, and relative_to is purely lexical: docs/../../outside/planted.md is relative to the root as a string, returning docs/../../outside/planted.md rather than raising. A .. is only collapsed by resolving, which is what the roots branch does one block above and this one did not. Two spellings of the same rule, ten lines apart, one of them not implementing it.

The fix then took four goes, and every wrong one came from spelling the rule differently from the place that already had it right. link._document_in resolves the parent and leaves the final component: the directory chain is followed, so .. collapses and an escape through a symlinked ancestor is caught, while the document's own symlink is irrelevant. That is the rule. Each attempt reinvented it:

1. Per candidate, after globbing. Correct about what to refuse, but it refuses the results: glob has already enumerated and stat'd the whole tree by the time the first match is inspected, so include = ["../../../../**/*.md"] still walked the machine on every post-commit. And an escape sets complete false, so no last_scan is written and the TTL cannot suppress the retry either — unbounded, forever.
2. Refuse any pattern containing .., before globbing. Bounded, and wrong in the other direction: ../notes/*.md stays inside the KB and the partner's own walk_sources ingests it. This KB called a legitimate manifest an escape and then never refreshed that partner again. Refusing a partner's valid configuration is the same defect as accepting an invalid one.
3. Resolve the prefix before the first glob component. Defeated by a pattern that starts with one: */../../../outside/**/*.md has an empty prefix, so the check passed unconditionally and the .. ran inside glob — attempt 1's defect, reachable again. It also refused a fixed pattern naming a symlinked document, because with no glob component the "prefix" is the whole path and resolving it whole follows the final symlink: include = ["alpha.md"] refused while include = ["*.md"], reaching the same file, was accepted.
4. Join the whole pattern and apply _document_in's spelling to it. A glob component is just a name that does not exist, which resolve() collapses lexically, so one resolve() answers it with no enumeration. Ten patterns measured — a .. staying inside, a directory genuinely named a..b, a literal bracket, two escapes, one behind a leading glob, a symlinked directory under a glob, a symlinked document by both spellings, and an absolute — all correct, escapes refused in 0.12ms without touching a 3000-file tree. None of the ten contained ** followed by ...
5. Drop ** from the probe. ** matches zero or more components while Path.parts counts it as one, so keeping it let a following .. cancel it and the probe landed one level below where the walk goes. **/../../**/*.md probed inside the KB and walked the directory containing it, recursively — linear in the outside tree, and silent, because an escape is only noticed once a candidate is yielded and that pattern matched none. Dropping it is exact rather than merely conservative: each component ** expands to is one a following .. then pops, so the zero-expansion is the highest the walk can reach.

Attempt 4's measurement was real and its ten patterns were all correct. It was the sampling that was wrong — ten hand-chosen inputs, none of which combined the two tokens whose interaction is the whole difficulty. A table of cases proves the cases in it, and reads like proof of the rule.

An absolute pattern is refused separately, because glob cannot walk one wherever it points — including at this KB's own docs/. It had been folded into the escape message, which was simply false for that case.

The lesson is not about paths. Three of the four attempts were written by reasoning about the problem afresh instead of copying the spelling from the function twenty lines away that had solved it. A rule implemented twice is a rule with two behaviours; the fix was to make the third implementation textually identical to the first two and say so in all three.

Each attempt was found by mutating its predecessor, and three of them disarmed an existing test: a fix that catches its input earlier leaves the older test green with its guard unexercised. By the last round two guards written in earlier increments had gone dead this way — L2's roots containment check, whose test was satisfied by the substring "outside the KB" that the new per-candidate check also emits, and scan_one's own except around the walk, whose only input (include = ["/etc/**/*.md"]) the absolute branch now answers first. Both were found by mutating behaviours this increment never touched.

So the rule is stronger than "re-run the battery after every fix": the battery is over the whole function, not the diff — and a promise worth a guard is pinned directly (here, by making the walk raise) rather than through an input that some later fix can intercept. Running it that way in the next round found three more unpinned behaviours: the sidecar existence check, which predates L6 (7570a69), and the */** boundary and the next() guard, both written two rounds earlier by this increment (425d106). The sharper reading is the second one — the code least likely to be pinned is not the oldest, it is what a recent fix added while attention was on the defect it closed.

The fix for "one bad entry is not the end of the partner" was itself incomplete, one line either side of where it landed. probe.parent.resolve() sat above it unguarded, so an embedded NUL in any but a pattern's final component still raised out of the function; and Path.match("") sat below it, so an empty exclude entry did the same — a case the new guard's own comment cited by name as its reason for being scoped tightly, and then did not handle. Both produced the outcome the commit had just declared impossible. A guard placed by reasoning about one call is a guard for one call; the same mechanical sweep that closes an escaping-error class — list every call that can raise, ask what each does with bad input — is what this needed, and it is the third time in this increment that lesson has been relearned rather than applied.

One more from the same sweep: the containment predicate was fooled by a trailing .., because Path("/kb/..").is_relative_to("/kb") is lexically true. The final component is left unresolved so a symlinked document stays readable, and .. is never a document.

sync.walk_sources has the identical shape for the local manifest and is not fixed here: it is the user's own configuration rather than a partner's, and changing the engine's document walk is not this increment's to do. Reported instead, and now scoped as its own increment and PATCH release in plans/20260731_2128-source-walk-containment.md — which measured a third defect this pass had not: an absolute local include is a raw NotImplementedError out of cli.main, the same escaping- error class L6 spent four passes closing on the partner side.

Smaller things

• pnk link A A wrote a self-loop, which says nothing and would return the document as its own neighbour. Refused now — and worded for both ways of arriving, because only the ULID is known here: the target really is this document, or it is a different file carrying the same id, which is a KB fault in its own right. "would link to itself" told someone who had named two different documents that one of them was itself, and pointed at neither the duplicate nor pnk doctor, which finds it.
• os.replace onto a symlinked sidecar destroyed the link and left a regular file, with the real file elsewhere still holding the old text. create() guards this explicitly; write() did not, and pnk link is the first command a person points at a file of their own choosing. It now writes through the link.
• The error fallback named the local KB root while the comment beside it claimed it named the declared path — reporting an unrelated readable directory for a failure that had nothing to do with it. Neither of the two tests written for that fix caught it: both asserted only the message prefix.
• why_not_a_kb reproduced, one level down, the defect its own docstring exists to record. The three-way split was added because an is_dir() split called an existing regular file "no such directory" — "the one answer a person would check and find false". But the caller's probe is is_file(), so a pinakes.toml that exists and is a directory, or a symlink to nothing, fell through to "no pinakes.toml there" with the file plainly visible in ls. Found in review 9 by reading the docstring's justification against the code beneath it, which is a cheap check worth running on any function whose comment argues for its shape.
• pnk link takes no lock, so a concurrent write to the same sidecar can lose one side's change. Rename-atomicity prevents a torn file, not a lost update, and DESIGN §2.2 now says which.
• STATUS's surface you can use today table had no pnk links row at all, an hour and a quarter after it shipped in 0.5.0 (20260731 11:27; the row landed in b96d247, 12:44) — found while writing the increment by reading the neighbourhood rather than the diff.

L7 — pnk doctor's link checks (20260801 05:40)

The check read a partner's index, and DESIGN §6.2 forbids exactly that

HIGH. The cross-KB check opened <partner>/.pinakes/index.db read-only to ask whether the target document exists. §6.2 rules that out in the sentence that defines reverse links: they come from the other KB's committed sidecars, "not its index, which is gitignored and simply absent in a fresh clone, and which could not be read without holding a second KB's lock" — repeated verbatim in linkscan's module docstring, which is the module the check imports from.

mode=ro is not enough, and this is the part worth keeping. Measured: a read-only connection still materialises index.db-shm and index.db-wal inside the partner's .pinakes/, and cannot checkpoint them away on close. A diagnostic command wrote into a KB it was only asked to look at. Two more consequences fell out of the same choice: a partner cloned but never synced answered "missing" for every target, and a partner whose .pinakes/ is mode 0500 degraded silently with an internal StoreError message that misdiagnosed the cause.

The fix is the machinery L2 already had — partner_sources + sidecars_under + read_sidecar — which is design-conformant, works on a fresh clone with no index at all, and is now tested that way.

The rule was in the imported module's own docstring. Not a subtle design point: a paragraph in the file the new code imports three names from.

The metric's numerator and denominator came from different populations

HIGH. Coverage is COUNT(DISTINCT src_doc_id) / active. sync's SoftDelete sets state = 'deleted' and drops the chunks — it never deletes that document's origin = 'sidecar' rows. So a deleted document still counted toward the numerator while leaving the denominator, and the headline number of this increment reported 2 of 1 documents linked (200%).

A ratio built from two queries is two populations until something makes them one. The join is one line; noticing it needed one was the work.

The declared [[links.kb]] id is not evidence of which KB is at that path

MEDIUM. The check keyed partner document sets on linked.id — the local declaration. linkscan.scan_one refuses that substitution with LinkedKbIdMismatchError, and DESIGN §6.2 rule 1 states it as a rule, because trusting the manifest files another KB's links under this alias. Measured both ways with a manifest declaring X over a partner whose real id is Y: a target that existed in Y was reported unresolved, and one that did not was silently resolved.

Two directions need two tests, and only one of them is obvious. Filtering on the declared id also skips a partner whose real id is the one wanted — a dangling target that goes unreported rather than misreported — and that mutant survived until a test was written for it specifically.

Four remedies could be blanked with the suite green

MEDIUM. The plan required "every new WARN carries a remedy" precisely because the meta-guard (test_every_problem_carries_a_remedy) runs on a fixture where these checks are OK and carry no problem. The helper written to stand in for it asserted is not None — which "" satisfies, while the guard it substitutes for asserts truthiness. Four of five remedies were emptiable.

A stand-in for a guard has to assert what the guard asserts. It now returns the string and each caller asserts a phrase from it.

A test named for a guard, authoring nothing that reaches it

MEDIUM. test_an_unreadable_linked_kb_path_is_a_warning_not_a_traceback was written for the sentence "a diagnostic command reporting a traceback is the one outcome pnk doctor may not have", and named why_not_a_kb's "third caller needing the same try". It authored no cross-KB link -- so wanted was empty, _unresolved_cross_kb returned before touching the partner, and the test pinned the guard in _linked_kbs and neither of the two in the function the review had just added. Both are load-bearing: a partner directory behind a mode-0000 parent raises PermissionError out of partner_sources, and a roots entry carrying an escaped NUL -- which tomllib accepts and Path.resolve does not -- raises ValueError out of sidecars_under.

Third time in two increments that a fixture stopped one step short of its guard, and the shape is always the same: the test sets up the failure but not the demand for it. An unreadable partner is only reached by code that has a reason to read it.

The dangling-link side of the soft-delete interaction had the same gap in miniature -- the fixture that proves the numerator excludes a deleted document already produced 1 dangling inside this KB in the detail it held, and asserted nothing about it. The fix was one line in a test that already existed.

Mutants that were not the logic they claimed

Methodological. Four "blank the remedy" mutants replaced "A cross-KB target…" with "" or "A cross-KB target…", which evaluates to the original string. All four reported SURVIVED, which read as four coverage gaps and was really one broken harness. Rebuilt to replace the whole remedies.append(...) call, all four die.

This is the second increment where a mutant that did not reproduce the real prior logic was briefly taken for a result. The check is cheap: a mutant that survives should be run against the case it claims to break before it is believed.

G2 — The headroom measurement, and what it found (20260801 12:14)

G2 was built to answer one question: is the graph release's gate reachable at all? It is not, on this corpus. The precondition needed at least 7 of the ~18 single-KB multi-hop questions to fail today. One fails. G3 does not start, and the answer arrived before anything bumped schema_version and forced every KB in existence to rebuild — which is the whole reason the measurement was sequenced first.

HIGH — the demo corpus has no tags, and one directory. The plan assumed otherwise. tests/demo-kb is thirty documents in a flat docs/, and not one sidecar carries a tags key. With mentions cut (decision 6), that leaves exactly one structural edge kind that crosses a document boundary: co-located, through a single thirty-way directory hub. shared-tag derives zero edges. sibling, parent/child and in-section are all intra-document and cannot bridge two evidence documents by construction. So the "derived structure" the graph release exists to evaluate is, on the committed corpus, one hub — and G5's own text reasons about "the directory layout and tag vocabulary of tests/demo-kb" as though a vocabulary existed. It does not. Whatever G3 would derive here, a result carried by it is a claim about one directory.

HIGH — a reachability probe on a thirty-document corpus is close to vacuous, and the reason is not the probe. candidates_per_source is 30 and the corpus has ~30 chunks, so the vector channel already returns essentially every document with a positive cosine: the funnel sees the whole corpus on every query and then cuts to final_k = 5. A failing question is therefore almost never a recall failure the channel could fix by reaching further — it is a ranking failure. That is why the probe reports at-seed separately from liftable: two of the three questions the fake backend called liftable were already among the fused candidates and merely ranked below the cut, having traversed no edge at all. A ceiling built from those would have read as headroom and been none.

The numbers, real [light] models, tests/demo-kb at 20260801 12:14. 18 multi-hop questions, 1 failing (mh-withdrawn-collection-register), liftable 1 without authored edges and 1 with, beyond-2-hops 0, membership-only 0. Required: 7 failing and 7 liftable without authored edges. It fails on the first clause by six.

The questions were frozen before the probe ran, and were not re-authored afterwards. Thirteen new multi-hop chains, authored from pairs of documents that between them answer one question and share no vocabulary on the thing that joins them, with the second hop phrased in the first document's words. Seventeen of eighteen are answered correctly. Re-authoring them until seven fail is fitting the question set to the gate — the circularity decision 14 removed by cutting cross-KB questions, and undetectable once done. The honest reading is that a corpus of thirty short, topically disjoint documents cannot produce a hard multi-hop set: picking 5 of 30 is not a discriminating retrieval task, and the pipeline scores 0.94 on it.

MEDIUM — the fake backend and the real models disagree about the shape of the answer, and only one of them is the measurement. Under the deliberately tie-heavy hashing fake the same set shows 9 failing and 3 liftable without authored edges (6 with) — the exact with/without gap the plan predicted L1's hand-authored links would produce. Under the real models both collapse to 1. A measurement taken on the fake would have reported a different failure of the precondition and invited the wrong remedy.

MEDIUM — _score read Outcome objects, so the artifact could not have been re-scored. Every metric is a function of five fields per question, but the scorer was written against the in-memory type. Splitting score_rows(rows) out is what makes the committed artifact checkable offline, and it is what test_the_committed_41_score_exactly_their_pre_growth_values runs on: no weights, no network, and the 41 pre-growth questions reproduce their baseline byte-identically — measured on macOS against a baseline written by CI's ubuntu runner, which is the same cross-machine agreement G1's new CI job independently confirmed the same morning.

LOW — the first --fake run silently asked for real weights. _fake_kb asserted each manifest substitution appeared exactly once; provider = "fastembed" appears twice (embedding and rerank), so the assertion fired and the run aborted — correctly. Loosening it to "replace whatever is there" would have left the rerank provider real and made an "offline" gate download a model. The expected occurrence count is asserted per line, not assumed.

Review pass — MEDIUM, the empty-set skip could swallow a typo. load_questions read raw.get("questions") or [], so a file whose key was misspelled produced an empty list — which the new skip then reported as "a template deliberately ships none" and exited 0. Under the old behaviour that file failed. An absent key is now an error and only an explicit questions: [] skips; the two cases are genuinely different and the skip is only safe because it can tell them apart.

Review pass — MEDIUM, read_outcomes promised more than it delivered. Its docstring said it "refuses a file whose rows are not rows — never a partial read", and a row missing confidence raised a bare KeyError from the middle of the loop. Every one of the five fields reaches a metric in score_rows, so a row missing one cannot be scored; they are now checked by name.

Review pass — LOW, fused_candidates is a stage, not an entry point. It does not run check_coherence, because search does that before calling it. Anything reaching for the new public function directly is querying an index that may have been built by a different embedding model, which returns confident nonsense rather than an error (§4.4). Stated in the docstring rather than duplicated in the function, since search would then run it twice on every query.

Source-walk containment — one rule, three sites, enforced at one (20260801 13:28)

The durable lesson: a containment rule argued in prose beside one of its two inputs is a rule for one input. manifest._sources states that a source root must stay inside the KB and enforces it for roots. include sat two lines away, validated nowhere. The same lexical candidate.relative_to(root) non-guard then appeared at three sites — linkscan.sidecars_under (fixed in L6 review 10), sync.walk_sources, and the sidecar sweep beside it — and the one whose docstring carried the argument was the one that did not implement it.

All three defects were re-measured on 0.7.0 before anything was changed, against a plan that had measured them on 0.5.0 at 900aae7. All three still reproduced, unchanged:

	Before	After
include = ["../../outside/*.md"]	2 indexed, a sidecar minted outside the KB, document keyed docs/../../outside/secret.md	ManifestError at load, naming the pattern and the root
include = ["/abs/path/*.md"]	bare NotImplementedError traceback out of cli.main	ManifestError: "is an absolute path", with its own remedy
docs/escape -> /outside, include = ["*/*.md"]	1 indexed, a sidecar minted outside the KB	0 indexed, the pattern reported, nothing written outside

HIGH — a fourth defect, found by a test that was meant to pin correct behaviour. Layer 1 deliberately accepts a .. pattern that lands inside the KB (include = ["../notes/*.md"] from docs/), because what matters is where a path lands rather than whether .. occurs in it. The test asserting that then failed on the document's key: relative_to is lexical, so it returned docs/../notes/n.md. Measured with roots = ["docs/", "notes/"] and include = ["../notes/*.md", "*.md"], one file on disk produced one indexed document and two failures — "appeared after the walk had already read this directory" — because the sidecar found under one key was invisible under the other, and the unmatched sweep reported an indexed document as unmatched. Nothing in the plan predicted this; it exists only because the legal .. case had never been exercised.

The fix is lexical collapse (posixpath.normpath), not resolve(). Resolving would follow a symlinked directory and re-key every document under it — docs/alias/x.md becoming docs/real/x.md — which on an existing KB is a path change against a permanent identity. Lexical collapse touches only paths containing .., and every one of those is already broken today. Containment does not rely on it: the per-candidate check resolves.

The predicate was copied from linkscan.sidecars_under, not re-derived, and that was the whole point. Reviews 11, 12, 13 and 14 each found a different defect in a different spelling of this one rule: refusing any .. (rejects a valid manifest); resolving only the prefix before the first glob component (defeated by a leading *); resolving the whole path (refuses a symlinked document while accepting the same file via *.md); and keeping ** in the probe (it matches zero components while Path.parts counts it as one, so a following .. cancels it). Re-deriving would have cost that sequence again for nothing.

MEDIUM — the static layer is the bound, and the dynamic layer is the guard; neither covers the other. Checking candidates after globbing refuses the results while still paying for the enumeration, which is what the roots rule exists to prevent — test_an_escaping_pattern_is_refused_without_enumerating_the_tree counts entries pulled from the generator, not resolve() calls, because the cost being avoided is the walk itself. And a symlinked directory has no .. and no absolute path, so it is invisible to any load-time check. The per-candidate test breaks rather than continues, and runs before the is_file()/sidecar skip: a pattern reaching outside that matched only directories or only sidecars hit one of those continues first, so the walk left the KB and reported nothing.

LOW — the default include is safe by luck, not by design. ["**/*.md", "**/*.txt"] does not escape through a symlinked directory, because pathlib's recursive ** skips them. Any user who writes a non-recursive pattern loses that, which is exactly the shape of a guarantee nobody knows they are relying on. Stated in walk_sources' docstring rather than left as folklore.

Mutation round — three survivors, two of them defects (20260801 13:38)

Eleven guards broken on purpose. Eight were caught immediately. The three that survived were worth more than the eight:

HIGH — the per-root skip copied from linkscan was data loss here. sidecars_under does if pattern in escaping: continue, so a pattern known to escape contributes nothing under any later root — correct there, where a dropped candidate costs one inbound link and a partner's [sources] is one statement about one KB. Copied into walk_sources it means something else entirely: the escapes this loop can see are symlinks, which are a property of one directory rather than of the pattern, and a dropped candidate here is a deleted index row and an orphaned sidecar. So docs/escape -> /outside silently stopped */*.md collecting anything under an unrelated second root. Removed, with a test. "Copy the predicate, do not re-derive it" was the right instruction and this was still the wrong thing to copy — the predicate and the policy around it are different decisions.

MEDIUM — the containment check ran before is_file(), and no test could tell. Every symlink test matched a file, so moving the check after the skip changed nothing observable. The case the ordering exists for is a pattern that matches only a directory (or only sidecars): it hits that continue first, and the walk leaves the KB reporting nothing. */* against a symlinked directory containing a subdirectory is that case, and it now has a test.

MEDIUM — the break bounded nothing, because sorted() had already drained the generator. The plan carried break, not continue, on a 360× measurement from linkscan review 12 — where the loop is lazy. Written here as for candidate in sorted(root.glob(pattern)) the enumeration a symlinked escape triggers has already happened by the time the first candidate is inspected, so the break saved only the loop body, and the resolved cache made even that one dict lookup. This is the shape of a guard inherited with its justification and without the property the justification rested on. The loop is now lazy; output order does not depend on it, because walk_sources sorts what it returns and the per-root sort only decided which of two candidates sharing one key won — and they describe the same file with the same hash. Measured: 301 entries enumerated before, 1 after, and both the break and a reversion to sorted() are caught by that number.

Two tooling corrections swept in the same PATCH (20260801 13:52)

tools/link_density_gate.py resolved one of its two bases and not the other, so any non-canonical root — every /tmp path on macOS — exited with a traceback. One root.resolve() at the top of census, and a test driving the tool through a symlinked parent.

tools/fragments.py's duplicate-heading defect was already closed, and the open-corrections entry saying "the tool is unchanged" is stale. Measured rather than assumed: three fragments (two fixed-*, one added-* whose body begins - **Fixed: …**) spliced into a section that already had a ### Fixed produce exactly one ### Fixed, one ### Added, and the category-prefixed entry filed by its filename, which is where the category belongs. Both halves have regression tests already — test_fragments_merge_into_a_category_heading_that_already_exists asserts count("### Added") == 1. Closed by the 0.6.0 release-prep commit, not by this one.

LOW, and the reason to write this down: fragments.py --apply is anchored to the repo it lives in, not the working directory. Testing it by cd-ing to a temp tree spliced this worktree's CHANGELOG.md and deleted its changelog.d/ fragment, reporting success. --repo exists exactly for that and the tool's own test suite uses it. The damage was recoverable only because the fragment had already been committed — which is the same rule the G1 mutation harness earned: commit before running anything that rewrites the tree. A git checkout -- to undo a mutation in the same session then reverted an uncommitted fix in tools/, for the second time in this project, and was caught only by re-reading git diff --stat afterwards.

The reachable-ceiling probe, against a corpus it did not ship with (20260804 04:21)

HIGH — a measurement tool that absorbs malformed input reports a number that looks valid. The finding class, stated once because it generalises past this tool: every defect below is an input the probe accepted, turned into a plausible verdict, and reported with no mark on the output. That is strictly worse than a crash. A crash costs an hour; a number that is quietly wrong is read into docs/STATUS.md, decides whether a schema_version bump is licensed, and is not falsifiable after the fact — nobody re-derives a measurement that already looks fine. Anything that converts input into a number owes its caller a refusal for input it cannot measure, and the refusal must be a named failure, never a diagnostic line a reader has to notice.

The two found by the rehearsal that ran the probe against an external KB — both measured on demo-kb under the offline fake backend, where the corpus reads 18 multi-hop / 9 failing / 3 liftable (the real [light] reading of the same corpus is 18 / 1 / 1, so the real-model impact of each is several times larger):

• A hop expect naming a path not in the index resolved through a lookup that answered "" for an unknown path, so the hop was recorded lands=False, reachable=False — failing and unreachable, identical in the output to a genuine one. One typo took failing from 9 to 10 while liftable stayed 3. On a 200-document corpus converted by hand from a frozen question set, this is not hypothetical.
• A multi-hop question with no hops incremented the denominator and produced no verdict, so it could never be counted failing and appeared in no other figure: 18 became 19, invisibly. The scaffolded template documented id, question, expect and kind and never mentioned hops — the trap was armed by our own template, which is why the fix edits both the tool and src/pinakes/templates/notes/eval/questions.yaml.

MEDIUM — the first fix was narrower than its own commit message claimed, and an adversarial pass found three more of the same class. Worth recording because the pattern is the lesson, not the individual bugs: a guard written against the two known instances validated the thing the bug report named rather than the property the measurement needs.

• A document with no chunks. The guard asked whether the path was in documents. Every node the channel walks is built from the chunks table, so a document with zero chunks — a blank file, a note that is only front matter, a PDF whose free extraction yielded nothing — passes a path check and is still incapable of landing or being reached. It reproduced defect 1 digit for digit (failing 9 → 10, liftable 3), from a path spelled correctly. "The name resolves" is not "the measurement can use it".
• A multi-hop question with exactly one hop, which the guard's own wording called "multi-hop in name only" and let through. This one is worse than the defect it was written for, because it moves liftable upward (3 → 4). Under-counting fails safe against a floor; over-counting licenses the schema bump. A guard on a threshold must be written in the direction that can do harm, and the harmful direction here was the one nobody had an example of.
• An empty hop query, absorbed the same way, and a golden set with no multi-hop question at all, whose entirely-zero report is indistinguishable from a measured one.

MEDIUM — the second review pass found the same hole again, in the key nobody had looked at. question.filters is applied to the last hop and was never validated. A tags, path_prefix or source_type the index does not hold makes that hop unable to land whatever the corpus contains. Measured on demo-kb under the fake backend, against its 9 failing / 3 liftable: one such filter took failing to 10; the same filter across every multi-hop question took the run to 18 failing / 0 liftable, exit 0, unremarked. (The review pass that found it quoted only the second figure for a single question — checking it is what caught the difference, which is the M5 lesson applied to the fix's own write-up.) It is the empty-query defect wearing a different key, it moves both binding clauses, and failing moves upward. Two review passes each found one more instance of a class the first fix was supposed to have closed, which is the actual lesson: the guard has to be written from the list of everything the measurement consumes, not from the list of bugs already reported. What probe() consumes is now the checklist — hops, each hop's query and expect, the document behind that path, and filters. It is validated through search's own _filter_sql rather than a hand-written copy, so the check cannot drift from the semantics it is checking.

MEDIUM — a third pass, and the same lesson a third time: the artifact recorded every setting except the one that moves the number most. retrieval.rerank records the mode (local), never the reranker's provider and model — and lands is expect in the top final_k after reranking. Demonstrated by the reviewer on one corpus, one path, one manifest, with only [rerank] model changed: 9 failing / 3 liftable became 18 / 12, and every identifying field in the two artifacts compared equal. Worse, the commit that added the block claimed it mirrored eval.py::_header — which carries three blocks, embedding, rerank and retrieval, its docstring saying it holds "every setting that can move a row". The copy took two of the three and dropped precisely the one not derivable from the others. When you cite a prior art as the standard you met, diff against it. index_built_at joined the payload at the same time: a corpus edited since its last pnk sync is measured as it stood then, and nothing else would say so.

LOW, and the most human of the findings: one defect, two accusations. The filters check ran before the hop-path check, and filters cannot admit a path the index does not hold — so a mistyped expect under a healthy filters: block produced two problems, the first of them pointing at the wrong line, and a {len(problems)} count that overstated. Ordering between checks is part of a refusal's correctness, not a detail: the message that names the wrong cause costs the same debugging hour the guard was written to save.

LOW — a sentence assembled from parts is a sentence nobody read. The per-kind wording was spliced mid-clause into three messages, and on the branch no test covered — a non-multi-hop question carrying hops, which load_questions allows — it rendered "so this probe never measures this question — only multi-hop — so no figure moves for the query rather than for the corpus — the same silent deflation as a mistyped path": two sos, and a closing clause asserting the deflation the same sentence had just denied. The commit message claimed that wording was fixed; no test exercised it. Each message now ends with a whole sentence, and a test covers the branch.

MEDIUM — a test can pin a claim it cannot falsify. test_the_probe_names_the_kb_it_measured ran only --fake, and --fake measures a copy of the demo KB: every assertion in it was satisfied by a probe that ignored --kb and hardcoded the demo path, which is the very defect the test names. It now runs a real --kb against a KB deliberately not called demo-kb (a small runner script registers the fake backend in the subprocess). A test whose fixture is the default cannot detect "always reports the default".

The same mistake then repeated one layer down, and is worth recording because it is so easy to miss twice: the replacement asserted that kb_root is resolved, using tmp_path — which is already absolute and already resolved, so dropping the .resolve() left the entire suite green. An assertion whose fixture already satisfies the property tests nothing. It now also runs with a relative --kb from the corpus's own parent directory, which is the only shape that can fail.

MEDIUM — naming the corpus is not naming the measurement. The artifact recorded which KB was measured and not what produced the numbers. failing is hop.expect in the top final_k passages, downstream of candidates_per_source, fusion, fusion_top_k, rerank and vector_tier — all per-KB manifest keys. Two artifacts from two configurations of the same corpus were indistinguishable, the exact defect the KB-naming fix was written to close, one level in. eval.py's artifact header already recorded the same set for the same reason; this one now does too.

MEDIUM — quoting a number without its backend. The first commit message and changelog fragment said "measured on demo-kb: failing 9 → 10" without saying the numbers were the hashing fake's; this repository's own retrospective already records that the fake and the real models disagree about the shape of that answer, and the real reading is 18 / 1 / 1. A user-facing fragment reads as the real measurement. Every measured number carries the configuration that produced it, or it is a different claim than the one intended.

MEDIUM — four passes, and the identity question kept moving outward one input at a time. Pass one: the artifact did not name the corpus. Pass three: it named the corpus but not the pipeline. Pass four: it named corpus and pipeline but not the golden set — the input every printed figure is computed from, and the one this branch's own refuse-edit-re-run loop changes most often. Demonstrated on one corpus, one index, one manifest, rewriting only the hop queries into a generic word: 9 failing / 3 liftable became 18 / 9, and every recorded field except reports compared equal. The payload now carries the golden set's resolved path, a sha256 of its bytes and its counts, plus revision on both model blocks — a revision selects weights as surely as a model name does. One correction the sixth pass earned: that "needs no re-sync, so nothing else would move with it" is true of [rerank] revision, which nothing compares against the index, and false of [embedding] revision, which search.check_coherence guards — change it without a re-sync and the run stops rather than drifting. Both are recorded; only one of them could ever have moved a figure in silence. The general form: an artifact must identify every input its numbers are a function of, and the way to find them is to enumerate the function's arguments, not to wait for a reviewer to name one.

LOW — the contradiction moved instead of leaving. The per-kind wording was fixed once by appending the conditional sentence to the end, which left "the hop is recorded failing-and-unreachable. No figure this probe prints moves" — an assertion and its denial, one sentence apart, in a message the previous commit claimed to have fixed. The consequence is now entirely inside the conditional (_consequence), so a non-multi-hop question is told nothing was recorded at all, and the test asserts the class ("no line may claim a hop was recorded") instead of one superseded string.

LOW — one more absorption, found by asking what check_measurable does not compare. Every hop was validated on its own and never against its siblings, so two byte-identical hops passed: MIN_HOPS satisfied, one retrieval written twice, and liftable moved from 3 to 4 on demo-kb — upward again. A YAML copy-paste is the realistic route.

MEDIUM — five passes, and the fixture-satisfied assertion came back twice in one commit. The pass-four commit added the golden set to the artifact and asserted it only under --fake — where the measured golden set is demo-kb's, so hardcoding the demo path and digest passed every assertion and the full suite. That is the identical defect pass two found for kb_root, recorded in this very fragment as "a test whose fixture is the default cannot detect 'always reports the default'", reintroduced by the same author two commits later for the input he had just added. The same commit pinned revision on both model blocks against manifest.<section>.revision — and demo-kb declares neither, so both assertions were None == None. Writing the lesson down does not apply it: the check is mechanical — for every assertion, ask what value the fixture already has, and whether a hardcoded constant would pass. _fake_kb now writes distinctive revisions into its copy, and the golden-set identity is pinned on the real---kb run.

LOW — one more absorption, one normalisation short. The identical-hops check compared (query, expect) byte-exactly, so upper-casing or padding the duplicated query defeated it while the retrieval stayed identical — FTS5 folds case, every backend here splits on whitespace. Measured on demo-kb: liftable 3 → 4, exit 0, the same upward move the check was written to stop. The fingerprint is now case-folded and whitespace-collapsed. A guard on "the same input" must normalise the way the consumer normalises, which is the _filter_sql lesson again in a smaller key.

The fix removes the place the defect could live, not just the symptom. _doc_id is gone; check_measurable validates the golden set against the active documents rows and the chunked subset up front, and probe is handed the resulting map, so an unknown path has exactly one place it can be handled and that place refuses. Validation runs before the backend loads — on a real run that is a model download, and a run that is going to refuse should refuse in a second.

Two smaller defects of the same family. --fake silently discarded --kb, so --kb <corpus> --fake measured demo-kb and reported its numbers as the corpus's; and neither output format named the KB, so two runs against two corpora produced artifacts indistinguishable on inspection — which is exactly what made the discarded --kb survivable. The pair belongs together: a silent substitution is only dangerous because the output is anonymous, and naming the input in the artifact is the cheapest defence a measurement tool has. The closing prose's hardcoded >= 7 was the same error in prose form, a claim about one corpus printed under the numbers of another; the threshold now stays with the corpus's own measurement plan.

On testing a refusal in a subprocess. These tests run the probe against a KB whose manifest names a backend the test subprocess never registered, so a run that got past the refusal fails too — a bare non-zero exit proves nothing. Every refusal test asserts the named message and the offending id/path, and test_a_well_formed_golden_set_is_not_refused is the control that keeps the message attributable to the question rather than the environment. Two assertions were weak for the same reason and were tightened: "hops" in stderr is contained in every refusal's closing remedy, and "--fake" in stderr is satisfied by any argparse error, since the usage line names both flags.

One deliberate over-reach, recorded so it can be overruled. A question-level expect naming a missing document refuses the run although probe() never reads expect — it measures hops. It cannot move any figure the probe prints, and refusing hard-stops a corpus whose frozen question set may not be edited. Kept because a golden set naming a document the index does not hold is broken for make eval, which does read it, and measuring a release precondition against an unchecked question set is not worth the saved minute — but the refusal now says which of its lines move the count and which do not, rather than claiming they all do.

The edge census — a .get default that would have hidden its own defect (20260804 11:25)

MEDIUM — the first draft of the text renderer used report.edges.get(kind, 0), which is exactly the failure class this feature exists to prevent. The whole point of edge_census is that a kind missing from the dict must never read the same as a kind genuinely at zero. _render's first draft indexed the dict defensively, so if edge_census ever regressed to drop a kind — a renamed key, a kind skipped by mistake — the printed table would still show kind 0, silently correct-looking, while the JSON output (built with dict(self.edges), no default) would be missing the key outright. Same bug, two output formats disagreeing about whether it happened. Caught before committing, by asking what a reviewer would ask: "would this line notice its own input being wrong?" Fixed to report.edges[kind] — direct indexing, so a dropped kind crashes loudly in text output too, matching JSON. The instinct to make a formatter defensive is usually right and was wrong here: this formatter's job is to report a fact edge_census promises to supply completely, and a default that papers over the promise being broken is worse than a crash — the same lesson docs/RETROSPECTIVES.md's reachable-ceiling probe retro already drew about the probe's inputs, recurring one layer up, in the probe's own output code.

HIGH — the first edge_census design counted every hub bucket, so co-located and shared-tag could never report zero on a corpus with documents in it, which is exactly the case the feature exists to surface. Found by an independent adversarial-review pass (a fresh agent given the diff and the requirement, not the implementation reasoning), not by the author: a directory holding one document, a tag on one document — plans/20260803_2239-corpus-probe-run.md itself calls this shape "most dirs connect nothing" — still contributed its member count to the sum, because sum(len(members) for members in hub.values()) does not care how large each bucket is. A corpus with real documents and zero shared structure would have printed a large positive co-located, the opposite of what the census exists to show. Fixed by excluding buckets of size one (_spoke_count, tools/reachable_ceiling_probe.py) — a bucket with nothing else in it derives no edge. The reviewer also showed the first reconciliation test could not have caught this: its "independent" expectation was sum(len(members)) over the same kind of total (document count, tag-row count), which is invariant to which bucket each item lands in — a grouping bug and a correct grouping produce the same sum. The fixed test computes its own per-bucket sizes from the raw tables and applies the same size-two-or-more filter, so a fixture now needs at least one real multi-member bucket and at least one genuine singleton bucket (docs/alone/solo.md, deliberately alone) to mean anything.

HIGH — the first reconciliation fixture could not have caught a wrong parent-child formula, twice. 900 words replaced an initial 240: at 240 words per heading section, structural chunking produced exactly one chunk per heading_path, so every groups[a] * groups[b] term was 1 * 1, and a flat += 1 per group-pair passed unnoticed. Raising every group to 900 words moved every group to size 2, not different sizes — and 2*2 == 2+2, so the next plausible wrong formula (groups[a] + groups[b]) also passed unnoticed; the adversarial review caught this one, mutating the multiplication to addition and finding the test still green. Fixed with three unequal group sizes (2, 3, 5 chunks, from 800/1300/2000 words against max_tokens = 510) chosen so no pairwise sum equals the corresponding product. The general form, twice-demonstrated: a fixture built to be "nonzero" is not a fixture built to distinguish the correct formula from the plausible wrong ones near it, and "I already raised the fixture size once" is not evidence the new size clears every nearby coincidence — each candidate wrong formula has to be checked against the actual numbers chosen, not assumed defeated by "bigger."

MEDIUM — the first authored reconciliation counted links rows; edge_census counts unique pairs. graph.authored is a set per document, so two rels between the same two documents (both legal — pnk link refuses only an identical (target, rel) pair, not a second relation between the same target) still count as one edge. The first test's "independent" expectation filtered and counted rows, which happened to agree only because the original fixture had no document pair linked twice. Fixed by comparing unique frozenset({src, dst}) pairs instead of rows, and the fixture now deliberately links one pair twice with different rels to make the distinction observable.

Common thread across all three: an adversarial reviewer with no stake in the implementation asked "would this test's expectation still be right if the code grouped things differently, or if the same edge were recorded twice?" — a question the author, having just written the grouping code being tested, did not think to ask about their own logic.

The docs site — 20260804 11:55

Building a rendered site over docs/ was worth it for one reason before any of the styling: the strict build is a link checker nobody had run over these files. Its first pass reported 31 failures, and they split into two classes worth keeping apart.

Twenty-three were links that leave docs/ — ../plans/…, ../CLAUDE.md, ../tools/record_claude_fixtures.py. Every one of them is correct on GitHub and dangles on the site, because the site only has pages for docs/. These were never wrong; they became wrong the moment a second renderer existed. All are now absolute github.com URLs, which work on both.

Eight were heading anchors, and the fix was not the obvious one. Every failure was a heading containing an em dash or an angle-bracketed code span, and the tempting repair — rewrite the eight links to whatever MkDocs generated — would have fixed the site by breaking the copy people already read in the repo. The three algorithms disagree:

Heading	GitHub	Python-Markdown	pymdownx
## 2.1 The manifest — pinakes.toml	21-the-manifest--pinakestoml	…-manifest-pinakestoml	✓
# The sidecar — <file>.pnk.yaml	the-sidecar--filepnkyaml	✗	the-sidecar--pnkyaml

Python-Markdown collapses the whitespace the dropped em dash leaves behind; pymdownx keeps it but strips anything angle-bracketed as an HTML tag, where GitHub slugs the escaped heading and keeps file. mkdocs_hooks.py implements GitHub's algorithm instead, so the site matches the docs rather than the docs being edited to match the site. The general rule: when a doc is rendered in two places, the renderer that arrived second adapts. Anchors are a published interface.

Two headings carry an angle-bracketed code span today, which is why this is a hook and not two edits — nothing stops a third, and the next one would fail a build with no obvious cause.

Not done, deliberately: PDFScout's 01-getting-started/-style directory tree. docs/ filenames are read by tools/fragments.py, tools/status_header_gate.py, tools/shared_file_overlap.py, tests/test_verification.py, CI and ~150 source comments. The numbered chapters a reader sees come from mkdocs.yml's nav labels and javascripts/section-numbering.js; the directory layout contributes nothing to them.

Renaming the repository broke PyPI trusted publishing — 20260804 12:40

Renaming lucagattoni/Pinakes → lucagattoni/pinakes was checked against everything inside the repo — 64 URLs, the docs site, CI, the gates — and all of it passed. What it broke was outside the repo: PyPI's trusted publisher matches on the exact repository name, and it is registered on pypi.org, where no gate here can see it.

So v0.9.0 tagged, built, smoke-tested, and was refused at the upload:

invalid-publisher — valid token, but no corresponding publisher
repository: "lucagattoni/pinakes"      # the token's claim, post-rename
                                       # the publisher still says "Pinakes"

Nothing was uploaded, which is the only reason this was recoverable: PyPI never allows re-uploading a version, so had the rename landed between two files of the same upload the version would have been burned and 0.9.0 would have had to be skipped. Verified against https://pypi.org/simple/pinakes/ rather than the JSON API, which is CDN-cached and has read as "missing" for a correct upload before (20260729).

The durable lesson: a rename's blast radius is every system that identifies this repo by name, and most of them are not in the repo. Before renaming, enumerate the external identifiers — trusted-publishing publishers, deploy keys and webhooks, any CI in another repo that clones this one, badge and package-index metadata — because a repo-wide grep proves only that the inside is consistent. GitHub's redirect for the old URL is what makes the inside look fine and hides this.

Ordering that would have avoided the outage entirely: update the external publisher first, then rename, then tag. The publisher can name a repository that does not exist yet, so there is no window where neither name works — and the reverse order guarantees one.

Interrupted sync — a TZ test that used the fixture helper would have proven nothing (20260804 12:46)

HIGH — the first draft of the UTC-timestamp regression test called test_sync.py's own run() helper, which hardcodes now="20260725 16:00" for every other test in the file specifically to bypass the real clock. sync.py:709's stamp = now or datetime.now(UTC)... only reaches the real-clock branch when the caller passes now=None — and run(kb, ...) never does, because that fixed string is what makes 60-odd other tests in test_sync.py deterministic. A TZ test built on top of run() would set TZ, call run(kb), read back meta['built_at'], and find it exactly "20260725 16:00" regardless of which clock sync.py actually used — passing identically whether the site under test read datetime.now() or datetime.now(UTC), because neither ever ran. Caught before committing by asking the same question the increment's own instructions insist on for the mutation pass: does the assertion distinguish the fixed from the broken code, or only look like it does? Fixed by calling sync() directly with no now= override in both TZ tests (test_a_real_sync_stamps_utc_not_local_under_a_non_utc_timezone, test_estimate_only_stamps_utc_not_local_under_a_non_utc_timezone — the second reaches its own independent clock at sync.py's _estimate_only, unaffected by the outer now= either way, so it needed no such change but is named here for the record) — and mutation-verified afterward by reverting each site to datetime.now() and confirming the corresponding test failed by roughly the test's chosen offset (Pacific/Kiritimati, UTC+14), not merely failing.

The general rule this confirms rather than discovers: a test helper that fixes an input to make most tests deterministic is exactly the place a new test targeting that specific input must not reuse the helper — the fixture built to remove non-determinism from one property will just as readily remove the property a new test exists to observe. docs/RETROSPECTIVES.md's own advice on claiming a test is mutation-verified — run the mutant, don't just trust the assertion reads correctly — is what caught this one before the mutation pass would have had to.

The progress printer's closing newline assumed the loop always reaches its own end (20260804 13:07)

MEDIUM — an independent adversarial review found that cli._progress_printer's "finished" branch (done >= total) is the only place the printer ever emits its closing newline, and _run's loop (sync.py) does not always reach done == total. A [budget] cap or any early exit stops the loop partway through, so the last progress(done, total) call for that run has done < total, and the printer's \r-prefixed line is left open — no trailing newline — for whatever prints next (print_sync_report, or an error message on an unhandled exception) to land on. Confirmed live: a progress line followed immediately by report text on the same terminal row. No test in the original commit exercised done < total as a run's last call — test_progress_printer_throttles_... always ended at done == total, and the progress-callback sync test never triggered an early stop.

Fixed by splitting _progress_printer() into (progress, finish): progress behaves as before, but also tracks whether a line is open (dirty); finish closes it with one newline if so, and is a no-op otherwise. run_sync calls finish unconditionally in a finally around the sync() call, so it closes the line whether the run finished normally, stopped on a budget cap, or sync() raised. Mutation-verified: made finish() unconditionally clear dirty without printing; test_progress_printer_finish_closes_a_line_an_early_stop_left_open caught it (expected "\n", got "").

The general shape of the miss: a "does this print the right thing" test that only drives the happy path (the loop's last call always being its final call) cannot see a defect that only exists on an early-exit path, because the assertion and the code under test share the same unstated assumption — "the loop reaches done == total" was never itself questioned, only how the printer behaves once it does.

A green ./check.sh only proves the worktree's own venv is green (20260804 13:30)

MEDIUM — test_estimate_only_stamps_utc_not_local_under_a_non_utc_timezone shipped with no pinakes[pdf] skip marker, and ./check.sh was green anyway, because this worktree had [pdf] installed. The test writes a real baseline-1p.pdf and calls page_count on it for real (its own docstring already said so), which needs pypdfium2. The planner's worktree did not have the extra installed and hit AttributeError: module 'pypdfium2' has no attribute 'PdfDocument' at merge time — same commit, same script, different venv. CI's check job runs a three-leg matrix over [light], [light,pdf] and [light,pdf,claude] specifically because core stays torch- and pypdfium2-free by design (CLAUDE.md § Tooling); this would have gone red on the [light] leg after merge, which the merge-time worktree-mismatch is what actually caught here.

The rule this earns: a green ./check.sh run proves the worktree's own dependency set is green, not the matrix — pytest silently skips whatever the installed extras cannot exercise rather than failing loudly, so a test that forgot its @pytest.mark.skipif(not pdf_extraction_runnable(), ...) marker doesn't skip or fail locally when the extra happens to be present; it just runs, passes, and says nothing about the leg where it can't. Before landing any test that touches a PDF fixture, the claude extractor, or a real embedding backend (sentence-transformers/fastembed), check which pdf_runnable()/pdf_extraction_runnable()/ paid_runnable() predicate (tests/conftest.py) it needs and mark it — and separately, run at least the affected file with the relevant extra uninstalled (uv sync --extra light --frozen, run, then uv sync --extra light --extra pdf --extra claude --frozen to restore), because that is the only way ./check.sh passing locally actually predicts the [light] CI leg rather than just restating the worktree it ran in.

G3 — the node model and the edge set (20260804 16:30)

HIGH — a hierarchy derivation that was quadratic in a document's chunk count. parent-child was derived by testing child.heading_path.startswith(parent.heading_path + " > ") over every chunk pair within a document. Measured on one document: 2 000 chunks 0.23 s, 4 000 chunks 0.85 s, 8 000 chunks 3.32 s — so a single 32 000-chunk document (one long PDF) would have spent ~50 s deriving, on a path pnk sync runs from three git hooks. Replaced by grouping chunks by heading path and having each path look up its own ancestors (" > ".join(segments[:d])), which is the same relation and is linear in chunks: 32 000 chunks now 0.59 s. The corpus that would have exposed this is the one that cannot: the RFC realism corpus has an empty heading_path on every chunk, so its 106 806 chunks derived zero hierarchy edges and cost nothing. A performance defect invisible on the only corpus at scale.

HIGH — a mutant that nothing caught, in a filter that looked redundant. authored_pairs filters links on src_kb_id = ? AND dst_kb_id = ? and then joins both ends to doc nodes. Changing the AND to an OR failed no test, because the join already drops a foreign document ULID — a foreign document has no local doc node. That reasoning is wrong in exactly one case, and it is a case that happens: fork a KB — copy the directory, mint a new [kb] id, and every document keeps its permanent ULID. A reverse scan of the fork then writes (fork_kb, D, local_kb, E) where D is also one of our documents, and the OR reads it as "our D cites E" — an edge nobody authored here. The lesson generalises: a filter that a second filter appears to make redundant is only redundant under an assumption, and the assumption is the thing to test. Pinned by test_a_forked_kb_sharing_a_document_ulid_does_not_forge_a_local_authored_edge, which builds two real KBs rather than inserting the row.

MEDIUM — the deriver was cross-checked against the instrument the go decision was measured on, and this was worth more than any single test. tools/reachable_ceiling_probe.py derives the same relations in memory, written independently. Comparing the two censuses on tests/demo-kb (test_the_stored_edge_set_agrees_with_the_probe_the_decision_was_taken_on) caught two mutants no targeted test did, and the RFC corpus reproduced the go decision's drop table exactly — sibling 106 506, shared-tag 643, co-located 262. A second implementation of the same spec is a cheaper oracle than a third round of hand-written assertions, and it answers a question no assertion can: did G3 build the graph G2 measured, or a different plausible one.

MEDIUM — the plan's orientation rule, read literally, makes every hub unreachable. G3's spec says the provider queries "src = ? OR dst = ? for those kinds and src = ? for hub kinds". A hub spoke is stored hub-first, so src = ? answers "who is in me" — and a member asking "what am I in" needs dst = ?. Read literally, no member could ever enter a hub, and co-located/shared-tag are the two kinds the go decision measured carrying all nine liftable questions. The sentence's real content is the symmetric half: a src-only read of a symmetric kind silently drops half of every relation. Built as three explicit functions — peers(), members(), hubs() — so the two halves of a hub kind cannot be confused for one query, and pinned by test_a_hub_is_entered_from_a_member_and_expanded_from_the_hub.

HIGH — a kind selection that could be silently dropped, in the function that preaches against it. peers()/neighbours() took local_kb: str | None = None and skipped authored when it was absent — the same "confident, wrong, smaller answer" its own docstring warns about for a src-only read, from the other side. G5's gate runs with and without authored edges; a caller who forgot the keyword would have got the "without" arm believing it ran the "with" one, and lost the highest-trust edge class with nothing printed. Now a TraversalError naming select_kinds(drop=["authored"]) as the way to mean it. select_kinds() refusing an unknown name was designed against exactly this and did not cover it — the refusal guarded the name and left the ingredient.

MEDIUM — every corpus that certifies this increment has two of the six kinds at zero. parent-child has derived zero edges in every measurement ever taken here — one-level headings in both committed corpora, an empty heading_path throughout the RFC corpus — and no committed sidecar carries a tags: key, so shared-tag is exercised only by synthetic fixtures. The cross-check against the probe therefore passed with _hierarchy_edges deleted, because its non-vacuity guard was a total over all kinds. Fixed by asserting each compared kind is non-zero and naming the two this corpus cannot exercise, so a kind that silently stops deriving fails rather than reads as agreement. A "total > 0" guard on a per-item comparison is not a non-vacuity guard — it is one item's evidence spread over all of them.

MEDIUM — parent-child is materialised pairwise, and its row count is the product of two sections. APPROACH §3 keeps hierarchy direct — "the one relation that stays direct" — so an ancestor heading of a chunks and a descendant of d chunks is a·d rows. Measured at max_tokens=400 on plausible document shapes: 5.8×, 16.3× and 53.5× the chunk count. Extrapolated to 300 documents of the worst shape that is ~10 M rows. Not changed here — hubbing it contradicts APPROACH §3 and restricting it to the immediate parent narrows the relation the go decision's probe measured — but pinned by a test that asserts the arity, and reported to the planner as a spec question rather than absorbed as an implementation choice.

LOW — a duplicated tag inflated a hub's member count where nothing could see it. derive collects edges into a set, so tags: [t, t] produced one spoke — but _tag_buckets appended the document twice, and the < 2 minting rule counts the bucket. A single document repeating one tag therefore minted a hub with one spoke and a divisor of 2. The test named test_a_duplicate_tag_in_one_sidecar_is_one_spoke passed with the deduplication removed, because the set downstream hid the mechanism it named. Deduplicated where the length is decided, and pinned by the single-document case that has no set to hide behind.

LOW — a hub with one member is derived state that connects nothing. A directory holding one document, a tag on one document, a heading with one chunk: expanding it returns only the node that reached it. The spec only says degree-zero hubs are reaped, which full re-derivation gives for free. Degree-one hubs are minted at zero benefit — a node, a spoke, and an entry in G6's hub report — so they are skipped, which also makes the census directly comparable to the probe's (_spoke_count counts buckets of two or more). Reachability is unchanged for the channel as APPROACH §4A defines it today; §4B's all-chunk seeding is itself flagged for re-evaluation on the sqlite-vec tier, so the claim is scoped rather than unconditional. The alternative reading is recorded here because it was a choice, not a deduction.

G5 — the parent-child ceiling, measured before the gate ran (20260804 21:05)

The measurement the arity decision required, and the corpus that could supply it was neither of the two anyone would have reached for. plans/20260804_1844-decision-parent-child-arity.md keeps parent-child transitive and asks for a ceiling "against a corpus whose chunker actually populates heading_path" — because the projection of 5.8×–53.5× the chunk count had never been run against one. Both obvious candidates fail for different reasons, and only the second was known:

corpus	documents	chunks	carry a heading_path	heading depth	parent-child
RFC realism	300	106 806	0	—	0 — structural chunking degraded silently
tests/demo-kb	30	60	60	always 1	0 — every document is flat
this repo's docs/ + plans/	43	2 671	2 671	median 2, max 4	13 232

tests/demo-kb populates heading_path on every chunk and still derives zero hierarchy edges, because a depth-1 path has no ancestor. A corpus can satisfy "the chunker works" and still exercise nothing, which is a second way to get a zero that looks like a measurement — and it is the shape the go decision's own bound warns about, one layer in.

The numbers

Real Markdown, written by hand, with real nesting — 43 of this repository's own documents, indexed into a scratch KB (never committed; docs/ and plans/ are the corpus):

• 4.95 parent-child rows per chunk — 13 232 over 2 671 chunks. Against sibling's 2 628 and in-section's 2 509, hierarchy is 71% of every stored edge, on a corpus where sibling is one row per adjacent chunk. It lands below the 5.8× floor the decision projected, so the projection was pessimistic rather than optimistic.
• Derivation costs nothing. The whole edge set derives in 0.158 s; the hierarchy alone is 0.004 s for those 13 232 rows. G3's ancestor-lookup form is linear in chunks and quadratic only in a document's distinct heading paths (median 11 here, max 76), exactly as it claims. The cost is row count, never wall clock, which is what the decision predicted and is worth stating because it changes which mitigation would ever be needed.
• Index growth is 12.9% — 11.17 MB with the hierarchy against 9.89 MB without, both VACUUMed. On a corpus of this shape the absolute number is 1.2 MB, and it scales with rows, not bytes of text: an index dominated by 384-dimensional embeddings (the RFC corpus is 265 MB for 106 806 chunks) would grow proportionally far less.

The ceiling is not alarming, and the standing risk is unchanged

Extrapolating 4.95 rows/chunk to the RFC corpus's 106 806 chunks gives ~529 000 hierarchy rows against its 107 802 total today — a five-fold graph, derived in well under a second. That is a number, not a problem, and it does not license switching to the immediate-parent variant: the decision is explicit that the variant is the arm to measure if the ceiling proves alarming, never the default to switch to first.

What is still unmeasured is the tail. 4.95 is a mean over documents whose median depth is 2. The decision's standing risk — "a corpus with deep heading nesting and large sections could make parent-child the dominant kind" — is about a shape none of these three corpora has: deep nesting and many chunks per section, where the row count is the product. This corpus's worst document carries 76 distinct heading paths and its arity stays modest because its sections are short. Nothing here refutes that risk; it bounds the ordinary case and leaves the tail where the decision left it.

The tail, measured (20260804 22:39) — and it is alarming

The paragraph above left the standing risk as prose. It is now a number. A purpose-built worst-shape corpus — six documents, heading depth 4, every heading path carrying ~26 chunks, which is the a·d product the risk names — was generated, synced with the same real backend, and measured the same way:

	this repo's docs/ + plans/	worst-shape corpus
chunks	2 671	2 483
heading depth, median / max	2 / 4	3 / 4
chunks per heading path, median	short sections	26
parent-child rows	13 232	132 630
rows per chunk	4.95	53.42
share of every stored edge	71%	94.7%
index growth	+12.9% (1.2 MB)	+113.4% (13.3 MB)
derivation	0.004 s (hierarchy)	0.84 s (140 079 edges, every kind)

53.42 lands at the very top of the decision's projected 5.8×–53.5× band, so the projection was accurate at both ends rather than pessimistic: 4.95 sits below its floor for ordinary prose, 53.4 reaches its ceiling for the shape it warned about. The index more than doubles. Derivation stays cheap — 140 079 edges in 0.84 s — which confirms the decision's own prediction that the cost is row count, never wall clock, and therefore that no mitigation aimed at derivation time would help.

What this corpus is, and is not. Synthetic and deliberately adversarial: generated Markdown with uniform nesting and uniform section length, built to make the product as large as plausible rather than to resemble anyone's notes. It is not evidence that real corpora do this — neither real corpus measured above comes close. It is evidence that the shape is reachable without anything exotic, because depth 4 with long sections is an ordinary specification or manual.

No variant is switched to here, on purpose. plans/20260804_1844-decision-parent-child-arity.md is explicit that if the ceiling proves alarming the immediate-parent form is the arm to measure, never the default to change first — and --drop parent-child is already a reported leg of G5's matrix. This is the input to that decision; the decision is the planner's.

G5 — the expansion channel (20260804 21:35)

HIGH — a ranking rule that would have made depth 2 unreachable, and it read as the careful choice. The channel's output is ordered and then cut at candidates_per_source. The first implementation ranked (distance, -cosine, …), on the reasoning that a two-hop chunk should not outrank a one-hop one on similarity alone — which sounds like the conservative reading of APPROACH §4A's "score expanded chunks by edge weight and link distance". It is not conservative, it is silently disabling half the feature: with distance as the primary key, every one-hop chunk precedes every two-hop one, so on any corpus where one hop already fills the cut, depth 2 contributes nothing to the output at all. The channel would have been depth-1 wearing a depth-2 budget — and the reachability ceiling that unblocked this increment was measured at two logical hops, so the gate would have been measuring something the precondition never covered. Now (-cosine, distance, …): cosine ranks, distance breaks the ties cosine cannot, and test_a_two_hop_chunk_outranks_a_one_hop_one_when_the_query_says_so fails if the order comes back.

HIGH — every tiebreak resolved to a surrogate id, and it looked stable because of another module's ORDER BY. Fan-out, frontier order and the final ranking all broke ties on nodes.id. Those ids are deterministic today — but only because edges.derive enumerates documents ORDER BY path — so the channel's answers depended on an invariant of a different module, unstated in both. That is G1's defect in a new place: _hydrate's unordered WHERE c.id IN (…) was stable in practice too, until a rebuild moved one golden-set question. Every tiebreak here now resolves to (documents.path, chunks.ordinal), the total order G1 gave the rest of the pipeline. The lesson is not "avoid surrogate ids" — it is that "this is deterministic" is a claim about the code that computes it, and if that code is somewhere else, the determinism is an assumption rather than a property.

HIGH — a fixture that was not the shape its own docstring described, in the tests written to catch exactly that. The membership-exclusion fixture built one document of six flat sections, so that ordinal 0 and ordinal 4 would be reachable only through their document's membership edge. Its body was f"## Section {i}\n\nword{i} " * 30 — which repeats the heading thirty times, and produced 180 chunks nested under Section 0 > Section 5. A parent-child hierarchy, inside a fixture whose entire purpose was to have none, in a file whose docstring names "an assertion satisfied by something other than the property it names" as the failure it is written against. It was invisible until an unrelated ordering change made the walk reach one chunk more. Two things followed: dropping the # Title was not enough either — the chunker then reads the first ## as the root and derives Section 0 > Section n, the same defect one heading level down — and a fixture that a test's meaning depends on now asserts its own shape (_assert_flat_sections: six chunks, every heading path depth 1, zero parent-child edges). A fixture is an assertion.

MEDIUM — a mutant that survived eighteen others, in the half of a rule the other half hides. The membership exclusion is two filters: a document never passes through to itself, and a root's own document never contributes member chunks at any depth. Deleting the second left the whole suite green, because at hop 1 the source is the root's document, so the first already covers it — and no fixture ever reached a root document from somewhere else. The shape that does is A —authored→ B —authored→ {A, C} at adjacent_k=1: at hop 2 the source is B, the first filter does not apply, and without the second, A takes the only slot to re-contribute chunks the query already had. Two rules that agree on every case you have built are one rule until you build the case that separates them.

LOW — a throwaway mutation harness produced two runs of contradictory results. Mutating a source file, running pytest and restoring in a loop reported extra failures in tests unrelated to the mutated file, twice, while the machine was also running a two-hour re-index and an eval matrix. Clearing __pycache__ between mutants and running each mutant on a quiet machine gave seven clean, reproducible results. The mechanism was not pinned down — a stale bytecode cache after a same-second restore and a subprocess that failed to spawn under load are both consistent with what was seen. Recorded because the reaction is the reusable part: a mutation result that implicates a file the mutant did not touch is a result about the harness, and re-running it on a quiet machine costs a minute.

G5 — a root spends a fan-out slot it is then discarded from (20260804 22:30)

HIGH — the fourth review round, on a rule the third round had already written down. Round three added a filter to _offer_chunks dropping candidates already found this hop or already emitted by an earlier one, with the reasoning stated in the comment: "a slot spent on it adds nothing, and dropping it before the cut is the same rule the membership exclusion applies one level up". The rule is right. It was applied to two of the three categories it covers.

A root reaching found is discarded twice over. _accept skips it before emitting, and run seeds self._expanded with the roots so it never joins the frontier either — so a root contributes neither a row nor a hop. Yet it had already taken one of the adjacent_k slots on the way, and the neighbours of a fused top-k chunk are very often other fused top-k chunks: sibling connects adjacent ordinals, and adjacent ordinals of a chunk the query matched are exactly what the vector stage also ranked highly.

Measured on graded_neighbour at the shipped default adjacent_k = 8: 4 candidates returned as built, 10 with roots dropped before the cut — and all six of the new ones are chunks fusion had not found, which is the only thing the channel exists to contribute.

Why it survived three rounds. Every existing assertion about roots was set-level — "neither root may be emitted", "expansion still runs" — and a set-level assertion cannot tell a slot spent from a slot saved. Both are satisfied by the defective walk. The test that catches it counts instead: adjacent_k = 1, two candidates, one of them a root, and the single slot must reach the non-root. With the filter removed the walk returns [], which is the failure mode named in the assertion's own message.

What the fix is worth, and where that cannot be seen

tests/demo-kb moved zero questions across all three gate legs — off, expand, expand-no-authored — with aggregates identical to four decimal places. That is not evidence the fix is inert: demo-kb's documents are about two chunks each, so adjacent_k = 8 never saturates and a root never displaces anything. A corpus can be incapable of exercising a change, and reporting "no movement" from one is reporting the corpus, not the code. The RFC realism corpus — 300 documents, 106 806 chunks — is where the fan-out cut actually binds, and it is the corpus the gate runs on.

The backstop that now survives mutation, on purpose

_accept's if node in self._roots: continue is unreachable once _offer_chunks filters. It stays: "a root is never emitted" is an invariant of the emit point, and a later caller adding a second way into found should not be able to break it from a distance. The honest consequence is recorded in the docstring and in the mutation table — deleting that line fails no test, while deleting the _offer_chunks filter fails exactly one. A backstop whose mutant survives is fine; a backstop silently counted as the enforcement is not, which is the same assertion-satisfied-by-something-else failure in its bookkeeping form.

G5 — the gate, run: expand ships off (20260804 22:52)

HIGH — the increment's deliverable is a negative measurement, and it is a clean one. On the RFC realism corpus, at G5's own HEAD, against a schema-3 index rebuilt for this run:

leg	multi-hop	improved	regressed	p
off	7/20	—	—	—
expand, drop authored	4/20	0	3	1.0000
expand, all kinds	4/20	0	3	1.0000

licensing p = 1.0000 (max, the more conservative of the two). expand defaults off, and tools/graph_gate.py exits 1. Clause 1 fails in both runs; clause 2 fails as well — by_kind[multi-hop] 0.350 → 0.200 against a 0.02 tolerance — and clause 4 fails on recall_at_k for the same movement.

Lost: content-disposition-http-takeover (hit at rank 1 → not found at all), http-rate-limit-status-code-provenance (rank 4 → miss), sip-invite-2xx-retransmission-defect (rank 2 → miss). Nothing was lifted.

reachable ≠ retrievable — the finding, and the plan predicted its shape

The reachability probe found 9 of the failing multi-hop questions reachable within two logical hops without authored edges, which is what unblocked the graph release. The retrieval instrument lifts none of them, and the channel's extra candidates displace three answers two-list fusion already had. The plan drew this distinction itself — "a ceiling gauge cannot rank, and an argument cannot measure" — and G5's whole existence is the reason it could be checked rather than assumed. A reachability precondition is necessary and nowhere near sufficient, and the gap between the two is not a small correction: it is 9 questions against 0.

The go decision also anticipated the outcome's meaning: "If expand did not pass, the finding is 'graph structure does not help this corpus', and the response is a corpus or a different channel design, never an escalation to a more expensive one." Nothing here licenses PPR.

Two things that make the number narrower than it looks, both stated rather than worked around

Clause 3 passes vacuously, and clause 4 nearly so. The RFC corpus has [retrieval.confidence] commented out, so every question scores confidence: unknown; neither the confidence-lost nor the newly-found-at-low term can be non-zero there. The decomposition clause 3 exists for is exercised only by the synthetic artifacts in tests/test_graph_channel.py — which is precisely why the plan insisted the gate be driven by synthetic fixtures as well as by the corpus. A gate whose only fixture is the real corpus can only be tested in whichever direction that corpus happens to point, and here two of its four clauses would never have fired.

The --drop parent-child arm is inert on the gating corpus. No chunk in the RFC corpus carries a heading_path, so parent-child and in-section derive zero edges and dropping them changes nothing by construction. The arm the arity decision added cannot say anything here. Its cost was measured separately (see the ceiling fragment); its retrieval value remains unmeasured, and the corpus that could measure it does not exist yet.

The --drop sibling arm answers its question, and the answer is "neither"

The go decision added this arm to ask, with the instrument that measures retrieval quality rather than a reachability ceiling, whether 99.2% of the graph's mass earns its place. On this corpus sibling is 106 506 of the 107 411 non-transit structural edges, and dropping it produces exactly the same 4/20, the same three regressions, and the same p = 1.0000 — one question's rank moves, and it is a miss either way.

So sibling neither helps nor hurts. It is 99.2% of the stored graph and it is inert in both gauges: the reachability probe already found removing it cost nothing, and the retrieval instrument now agrees. Two independent measurements, and the harm the channel does comes from somewhere else entirely — the document-level path, membership transit into co-located (262 edges) and shared-tag (643) hubs, which pull whole documents' chunks into the fusion.

With the caveat that makes it a narrower claim than it looks: every chunk here has an empty heading_path, so a "sibling" in this corpus is an adjacent arbitrary size-slice, not an adjacent section. The arm has measured the value of size-slice adjacency, which is what this corpus's broken structural chunking produced — not the value of sibling as designed. On a corpus whose chunker works the question is still open.

Five legs land on the same number; the sixth does not, and I had already written that they all did

The full matrix, off at 7/20 (recall@k 0.3500, MRR 0.421):

leg	multi-hop	improved	regressed	p	ms/query
off	7/20	—	—	—	2012
expand	4/20	0	3	1.0000	2051
expand-no-authored	4/20	0	3	1.0000	2106
expand-no-sibling	4/20	0	3	1.0000	2067
expand-no-parent-child	4/20	0	3	1.0000	2028
expand-no-link-distance	4/20	0	3	1.0000	2024
expand-in-degree	6/20	1	2	0.8750	2237

MEDIUM — the process finding, and it is the file's own failure class caught in the act. With five of the six legs written I recorded that "every leg lands on the same number, which is what makes the result robust" — a claim about six legs asserted from five, while the sixth was still running. The sixth then came back different: in-degree salience is the only configuration that lifts anything (imap-utf8-two-strategies), and the only one that regresses two rather than three. The claim was wrong within minutes of being written, in a fragment whose subject is an assertion satisfied by something other than the property it names. A generalisation over N runs written while N−1 have finished is not a measurement, it is a prediction wearing a measurement's clothes. The correction is recorded rather than quietly overwritten, because the tempting fix — waiting and writing the true sentence — would have hidden how easy it was to write the false one.

What the sixth leg does and does not license. expand-in-degree at 1 improved / 2 regressed is p = 0.8750: nowhere near the gate, and still a net loss of one question against off. It is reported, never gated — three variables against one threshold is not a decision procedure — and noticing that it is the best-performing leg after seeing the numbers is exactly the exploratory fitting the pre-commitment forbids. It is a direction for a future measurement on a corpus that can carry one, not a result.

Latency, the other exit criterion. off 2012 ms/query against expand 2051 — 1.02×, so the channel costs about 2% on a 106 806-chunk index and the "slow at query time" risk did not materialise. In-degree salience is the expensive leg at 2237 ms (1.11×), which is the one place the matrix's timing separates the configurations at all.

The with/without-authored split is stronger here than on tests/demo-kb

All 391 of the corpus's authored links are intra-KB — every to: names the corpus's own KB ULID — so unlike tests/demo-kb, where only 12 of 16 survive the cross-KB inertness rule, the with-authored leg has every one in play. They still move only two questions' ranks and flip no outcome, and both gated runs regress the identical three questions. The anti-circularity guard had nothing to catch, because there was no win to be circular about.

The pre-commitment held

A result short of the table ships the channel off, with counts and p-value recorded, untuned. Nothing was tuned, no weight moved, no threshold was revisited after seeing the number. The authored weight's measured at G5 marker is discharged by this run as "measured, and it changed no outcome" rather than by a fitted value.

G6 — Edge-hub reporting (20260805 06:20)

MEDIUM — the enumeration order this check's own sort relied on was implicit. The first cut enumerated hub node ids with SELECT DISTINCT src FROM edges WHERE kind = ?, no ORDER BY — the only query touching edges in this codebase without one; every read in graph/edges.py orders explicitly, and that file's own docstring argues for it. It happened to come back in ascending src order today (a covering-index scan on edges_src(src, kind)), which is exactly the "mint order" the first test's fixture needed to differ from degree order to prove the .sort() was doing real work. That property held by accident of a query plan, not by anything this function asserted. Fixed with an explicit ORDER BY src. Found by an adversarial review agent, not by any test — no test could have caught it, since the property under test (the sort) still worked; only reading the query against the file's own convention surfaced it.

MEDIUM — one of the three hub kinds shipped with zero assertions on its printed text. _hub_label has three branches — tag, dir, heading — and the landing commit tested tag (the sort-order fixture) and heading (the human-actionable-label fixture) but never dir. co-located appeared in a docstring only, explaining why the other fixtures were built to avoid triggering it — which reads, on a second look, exactly like the tests were routed around covering it rather than covering it. test_a_directory_hub_is_named_by_its_kb_root_relative_path closes it.

LOW, folded into the fix above — a degree tie had no assertion in either direction. top.sort(key= lambda item: item[1], reverse=True) is a stable sort, so two hubs at equal degree kept whatever order they arrived in — which is to say, the same implicit query-plan order the first finding flags, one layer further in. Given a real tie the printed order was accidental twice over. Fixed by sorting on (-degree, kind, key) explicitly, and test_a_degree_tie_breaks_deterministically_and_ the_rest_are_counted builds four hubs tied at degree 2 whose mint order is the reverse of their correct printed order — reverting the tiebreak makes the mutant print d, c, b instead of a, b, c, which is the sharpest test in this increment: it fails on a reverse=True-only sort that every other test here would still pass.

LOW, self-inflicted — a vacuous assertion, written by the implementer, in this same increment. While extending an existing test's block with a new line, an assertion checking for a string that appears nowhere in src/ — "unchecked until the links release" not in detail — was added twice (once by the edit that introduced the increment's other tests, once copied into a third test written later in the same session). It always passed, proved nothing, and is exactly the failure class this project's own CLAUDE.md names: "an assertion satisfied by something other than the property it names." Caught by cross-referencing the diff against src/ during this same retrospective pass (grep -rn "unchecked until the links release" src/ returns nothing), not by any tool — nothing in check.sh can distinguish a true-but-empty assertion from a load-bearing one. Removed.

Read together: three of these four findings trace to the same root — an implicit order (query plan, sort stability) standing in for an explicit one, discovered because it happened to agree with what a fixture needed. G3's own docstrings warn about exactly this shape for a src-only read of a symmetric edge kind ("a confident, wrong, smaller answer"); this increment reproduced a milder version of it one level up, in the read that reports G3's own structure back to a human.

The [light] backend error — a fixed test that only looked environment-independent (20260805 07:41)

MEDIUM — the existing test_a_missing_extra_names_the_install_command was silently coupled to which extras happen to be installed in the checkout running it, and this dev checkout already has fastembed (a transitive dependency of some other extra) even without [light] explicitly requested. Once BackendMissingError started naming an installed alternative, that test's monkeypatch.setattr(builtins, "__import__", refuse) — which only blocks sentence_transformers — left fastembed genuinely importable, so load_backend picked it up as the alternative and the old assertion ('uv add "pinakes[st]"' in remedy) started failing for the right reason: the new code path executing, not a bug. A version of this test that had merely added the new assertions without also forcing "nothing else is installed" would have been true by luck of this machine's site-packages, not by construction, and would flip on a bare CI leg or a machine with no fastembed at all. Fixed by monkeypatching importlib.util.find_spec directly in both the "no alternative" and "alternative present" tests, so each names its precondition instead of inheriting whatever the environment happens to have — the same discipline docs/RETROSPECTIVES.md already names for tests that read like they exercise a real-clock or real-install branch but route around it.

Confirmed by the mutation pass: forcing _installed_alternative's find_spec check to always report "installed" broke test_a_missing_extra_names_the_install_command specifically (asserting alternative is None), and reverting _import to drop the detected alternative broke both alternative-path tests specifically (asserting alternative == FASTEMBED) — each mutation failed the test that names the property it broke, not an unrelated one.

The heading-coverage check — two findings, one of them about my own test

HIGH — the open correction's diagnosis was approximately right and precisely wrong, and the difference decides the fix. Item 1 read "the heading grammar is Markdown-shaped; RFC section numbering is not, so nothing matches and the strategy quietly becomes size-based". That describes a regex being tried and failing. What actually happens is chunk.py:131:

blocks = _markdown_blocks(text) if kind == "markdown" else _plain_blocks(text)

_markdown_blocks is never called for a .txt file, and _plain_blocks sets heading_path=None unconditionally. Nothing failed to match because nothing was tried. The consequence for the fix is not cosmetic: tightening or extending a regex would have changed nothing, and the real change is adding heading detection to a code path that has none. It also bounds the blast radius in the useful direction — tests/demo-kb is Markdown, so a plain-text grammar cannot move the golden set, and "changing chunking needs eval justification" becomes a thing you can prove rather than argue.

A measurement replaced a threshold. Chunking the committed corpora directly — no index, no embeddings, chunk_document called in a loop — gave demo-kb 60/60 and partner-kb 55/55 at 100% against the RFC corpus's 0%. Bimodal, so the predicate can be "zero for this source type" and the check carries no constant anybody had to calibrate. The alternative, a fitted percentage floor, would have needed a corpus to fit against and would have fired on ordinary documents whose opening paragraph precedes their first heading.

MEDIUM — mutation testing refuted one of my own tests within a minute of writing it. I wrote test_heading_coverage_counts_only_active_documents, asserting the state = 'active' filter. M1 deleted the filter and the test stayed green. The reason is that SoftDelete drops a document's chunks as well as flipping its state, so a chunk-counting query has nothing to over-count either way — unlike _links, which counts documents and genuinely needs the filter it records having shipped without (2 of 1 documents linked (200%)).

The test was kept and renamed to what it proves (test_a_removed_documents_chunks_stop_being_counted), with the refutation written into its docstring so nobody re-derives the wrong claim from the old name. The filter stays as defensive consistency with _links, marked as unreachable by this fixture rather than presented as guarded.

This is the file's own recurring failure class caught in the act: an assertion satisfied by something other than the property it names. Three of the four mutants died as intended (the zero-per-type predicate, the two-cause remedy split, the WARN status); the one that survived was the one whose name made the strongest claim. Green proved the test ran; only the mutant proved what it could detect — and for one of five, the answer was "not the thing in its name".

Item 5 — doctor never prints the operator's home directory (20260805 08:06)

LOW — two residual home-directory leaks exist, both correctly out of this increment's scope. An adversarial review confirmed the sweep of src/pinakes/doctor.py is exhaustive against every PinakesError subclass doctor.py forwards, and found no defect in the fix itself (three independent mutations — a no-op _de_homed, a swapped _local tuple order, and a reverted _sidecars call site — each broke exactly the test that should catch it, and nothing else). Two things remain that print an absolute path containing the operator's home directory, both outside the item's stated boundary ("paths outside the KB stay as they are"):

1. _linked_kbs's except OSError as exc: absent.append(f"{linked.name} ({exc.strerror or exc})") — the or exc fallback stringifies a bare OSError, whose default __str__ includes .filename when set. The path involved is a linked KB's resolved location, not manifest.root — legitimately a different KB elsewhere on disk, not this one — so it falls outside "paths outside the KB stay as they are" by the same reasoning that already keeps hf_cache_dir() untouched. Rare: only fires when why_not_a_kb's OSError.strerror is falsy, an edge case its own docstring already calls out as rare (an unreadable parent directory).

2. budget.prices.PricesMissingError(reason=str(exc)) — ships a package-relative path (prices.toml's location inside the installed wheel or an editable checkout), not manifest.root-derived, so _de_homed correctly leaves it alone. In an editable/source install, that path is often literally under the developer's home directory too (e.g. ~/Code/pinakes/src/pinakes/budget/prices.toml) — the same shape of leak as hf_cache_dir(), for the same reason (a real filesystem location worth showing, not KB-derived).

Worth keeping: if "no home directory in pnk doctor output, ever" becomes the actual goal rather than "no home directory via the KB's own location", both of these are where to look next — they were not fixed here because the item's own text draws the boundary at manifest.root, and extending it is a separate decision.

measure_sync_cpu.py measured the launcher, not the work (20260805 17:37)

HIGH — the tool answered its one question with a number that was precisely, confidently wrong, and every test passed. sample_percent ran ps -o %cpu= -p <pid> against the pid it launched. The invocation the tool was written for — and prints in its own --help and changelog fragment — is -- uv run pnk sync ..., which makes uv the measured process and pnk its child. uv waits and burns nothing.

Measured on this repo before the fix, one identical one-core busy loop:

launched as	reported
the busy loop directly	1.0 cores
the same loop behind uv run	0.0 cores

The failure mode is the expensive one: 0.0 cores for a sync saturating a core does not read as a broken tool. It reads as the finding item 6 went looking for — "the loop is not CPU-bound, so multiprocessing buys nothing" — and it would have been quoted into a design decision.

Why the tests could not catch it. All seven ran sys.executable -c <busy loop>: a direct child that does the work itself. The suite covered the sampler, the units, exit-code propagation, empty and non-positive arguments, and the trailing-interval bug — everything except the one process shape the tool exists to be pointed at. Coverage of the code was complete; coverage of the invocation was zero. This is the recurring class named in docs/RETROSPECTIVES.md — an assertion satisfied by something other than the property it names — reached from a new direction: not a weak assertion, but a fixture that was never the real subject.

Fixed by summing %cpu across the root pid and every descendant from a single ps -A snapshot (one snapshot, so a child starting or exiting mid-walk cannot be double-counted or missed), plus a test whose command is a launcher that burns nothing itself.

The new test's upper bound is load-bearing, and mutation proved it. With the tree walk neutered, the launcher still reported 0.1 cores of its own interpreter startup — so assert peak > 0 would have passed the mutant. Asserting > 0.5 fails it. A threshold above "anything a waiting process can produce" and below one core is what makes the assertion name the property; "non-zero" would not have.

Also corrected: %cpu is a decaying average over up to a minute (man ps), not the instantaneous reading the docstring claimed. Right for the steady-state multi-minute loop this measures, but it means peak is the peak of a smoothed series — a low peak is much weaker evidence of an idle machine than a high peak is of a busy one, and the docstring now says so.

Generalisable: when a tool's purpose is to be run one particular way, one test must run it that way. A synthetic fixture chosen for speed silently replaced the subject here, and no amount of assertion strength on the wrong subject would have helped.

Fixture copies carried the developer's own .pinakes/ into the test workspace (20260805 17:49)

MEDIUM — three tests failed on main immediately after two clean merges, on a machine where nothing was wrong with the code, and the failure impersonated the exact defect this repo watches for. CLAUDE.md says a clean auto-merge is not a correct merge; both branches had been green individually; main then failed. Every signal pointed at a bad merge. The cause was a tests/demo-kb/.pinakes/index.db dated 1 Aug — a leftover from a manual pnk sync predating the graph release's schema_version 3 bump — which five shutil.copytree calls copied into the test workspace along with the documents.

The tests were coupled to whether the developer had ever run the tool by hand. CI clones fresh, so .pinakes/ never exists there and the suite is permanently green; a dev box that has exercised the fixture once fails until the directory is removed. .pinakes/ is gitignored, so nothing in the diff, the merge or the branch could have shown it.

The idiom was already in the codebase and applied to four of nine call sites. ignore=shutil.ignore_patterns(".pinakes") was used in test_eval.py, test_search_reproducibility.py and twice in test_partner_kb.py. The other five simply omitted it. A guard applied at some call sites is not a guard — it is a coin flip weighted by which test the copy happens to be in, and no gate could notice the omission because the states it protects against are gitignored and absent in CI.

Verified in both directions, planting a poisoned fixture rather than reasoning about it. With a deliberately corrupt tests/*/.pinakes/index.db in place, all 105 tests across the four affected files pass. Removing the guard from one call site fails exactly that site's test (test_edges.py::test_the_stored_edge_set_agrees_with_the_probe_the_decision_was_taken_on, StoreError), which is what makes the guard's presence the property under test rather than an incidental line.

Generalisable: when a test copies a directory the tool also writes into, the copy must name what it excludes. And a loud environment-coupled failure is not a cheap one — this one cost real time precisely because it arrived wearing the costume of a merge defect, right after a merge.

An exact assertion between two different roundings — green locally, red on one CI leg (20260805 17:55)

MEDIUM — a coin-flip assertion that passed for the wrong reason, and whose failure was earned by an unrelated correct change. test_reports_cores_the_way_macos_percent_converts_to_them read the two numbers off the peak: line and asserted cores == pytest.approx(percent / 100.0) — at approx's default relative tolerance, i.e. effectively exact. But report() renders percent at 0 dp and cores at 1 dp: they are two roundings of one value, not one value printed twice. Exact agreement is a coincidence of the input, never a property of the code.

It held only while a single-process sample sat at exactly 100.0. The tree-sum fix (measured the launcher, not the work) made a one-core loop read 101.4 — parent plus child — and "101"/100 != 1.0 turned CI red.

The tell is which legs failed. check (light pdf) failed; check (light) and check (light pdf claude) passed on the same commit, same code, same test. Three legs disagreeing about one assertion is not a flaky environment — it is an assertion whose truth depends on a measured value nobody controls.

And it failed on merged main, not on the branch. The branch's own ./check.sh was green, twice, because this machine's readings rounded agreeably. A local gate cannot rule out an assertion that is only usually true.

Fixed with abs=0.06 and the arithmetic written down: 0.5 of display error in the percent is 0.005 of a core, plus 0.05 from the cores field's own rounding, so 0.055 is the largest honest disagreement between the two fields.

The loosened bound still bites — verified, not assumed. Dropping the /100 from CpuTrace.cores fails exactly this test (1.0 against 101). A tolerance that admits formatting disagreement while still rejecting arithmetic disagreement is the assertion the test always meant to make.

Generalisable: comparing two rendered numbers is comparing two roundings. Either compare the values before formatting, or state a tolerance derived from the display precisions — never an exact comparison "because they should be equal". This is the repo's recurring class once more: the assertion named "the conversion is right" was actually testing "the two roundings happen to agree".

Mutation testing found a guard that could not fire (20260805 19:07)

LOW as a defect, worth keeping as a method result. The numbered-heading predicate's document-level check was written straight from its own spec, which says the numbers must form a valid outline walk and that no number repeats. Both were implemented: a seen set alongside the step-validity rule.

Mutating the clauses one at a time, eight of nine mutants were killed by the test named for that clause. The ninth — deleting the no-repeats check entirely — broke nothing.

The first instinct is "write the missing test". The right answer was that no such test can exist: every step the walk permits raises the number tuple lexicographically — a sibling raises its last component, a first child appends to it, an ancestor's next sibling raises a shallower one — so an accepted sequence is strictly increasing and a repeat is unreachable. The check was dead code wearing a guard's clothes.

It was removed rather than kept as defence in depth, and the reasoning put in the docstring. A guard that cannot fire still reads as one, and the next person to touch the step rule would weaken it believing this had their back. The spec keeps the no-repeats sentence — as a statement of intent it is correct, and the implementation note now says why it needs no code.

Generalisable: a surviving mutant asks a question before it asks for a test — is this reachable at all? Adding a test for unreachable code is how dead code acquires the appearance of coverage. This is the inverse of the failure this project keeps meeting: usually an assertion is satisfied by something other than the property it names; here a guard was satisfied by something other than itself.

Running it found what reading it could not (20260805 19:15)

MEDIUM, and the reason it is recorded beside the note above: the two findings came from opposite methods on the same increment. The dead guard was found by mutating code. This one was found only by building a KB and using the feature the way a user would.

[chunking] headings = "numbered" added to an already-synced KB, then a plain pnk sync:

	result
plain pnk sync	1 unchanged · every heading_path still empty
pnk sync --rebuild	1 indexed · the three heading paths, and the first parent-child edge

An incremental sync re-chunks a document only when the document changed. A manifest-only edit changes no content hash, so the feature silently does nothing — and pnk doctor then reports exactly the condition the user just tried to fix.

Every test passed throughout, and no test could have caught it. The unit tests call chunk_document directly with the parameter set; the mechanism that drops it lives in sync.py's change detection, one layer up. The defect is not in either component — it is in the seam, and a seam is only visible from outside both.

It is also pre-existing: max_tokens and overlap have always behaved this way. Three releases did not surface it because no [chunking] key had ever been worth flipping on a KB already indexed. Adding the first one that is, is what made an old defect newly reachable — and "my change did not cause this" is not the same as "my change did not make it matter".

Generalisable: for any change that adds a knob, turn the knob on a real KB before landing. Unit tests verify a component honours a parameter; only using it verifies the parameter arrives.

A warning that cleared itself without the fix being applied (20260805 20:20)

HIGH — the first draft turned a silent defect into a lying one, and every test passed. The chunking-drift warning was correct. What was wrong sat 300 lines away: sync.py wrote the current chunking identity into meta at the end of every sync, including the incremental one that had just refused to re-chunk anything.

So the sequence was:

step	what the user saw	what was true
edit manifest, pnk sync	1 unchanged + the new warning	index still built the old way
pnk sync again	1 unchanged, no warning	index still built the old way
pnk doctor	OK chunking coherence	index still built the old way

A warning that clears itself without the fix being applied is worse than no warning: it converts "the tool said nothing" into "the tool said it was fine". The index actively claimed a coherence it did not have.

Found by running it a second time, not by testing it. The unit tests asserted the warning appears — it did. Nothing asserted it persists, because persistence only fails on the second invocation, and a test that runs an operation once cannot see a defect that needs it twice. The fix's own test is now ..._persists_until_the_rebuild_actually_happens, which syncs three times.

The correct rule turned out to be narrow: record the identity only when every chunk in the index was produced by this run — a rebuild, or a first build into an empty index. An incremental sync re-chunks only what changed, so after one the index is a mixture, and there is no single honest value to record. Leaving the old value is right: it keeps warning, which is exactly what a mixed index deserves.

Generalisable, and it is the second time today: a state-writing side effect belongs with the work it describes, not with the command that happened to run. set_meta is called once per sync and was treated as "the place identity goes" — but identity is a claim about the chunks, and only one of those code paths actually produced them all.

The corpus rejected two of my fixes, and that is the result (20260805 21:00)

The measurement §5.3 demanded was run, on real RFCs, in doubling rounds: 66 → 131 → 259 → … The rule was state the predicate first, measure second, and treat a poor match as a finding rather than a licence to loosen a clause. It held, twice, in the direction that costs something.

Round 1 (66 documents) found a false positive. RFC 769 lists facsimile command codes as 56 - SET-UP, 57 - DATA, 58 - END. Consecutive integers, short labels, column 0, blank lines around — every clause passed, and the predicate produced three headings that are not headings.

The first fix was wrong, and the corpus said so. "A heading's title must not begin with punctuation" kills it. It also killed three genuine documents: 5.1. /get, 2.7.3. "iprev", and RFC 2010's entire outline, which numbers real sections 1 - Rationale and Scope — the identical shape as the false positive. Form cannot separate them. What separates them is where they start: an outline begins at section 1, a list of opcodes begins at 56. That rule changes exactly one verdict across the corpus, and it is the wrong one. It shipped as clause 9.

Round 2 (131 documents) found a second false positive, and rejected a second fix. RFC 778 numbers a procedure — 1. Connect to COMSAT-GAT host…, 2. Send the command… — starting at 1, so clause 9 does not catch it. The obvious discriminator is that a heading stands alone: require a blank line after the candidate. Measured, it removed the false positive and four genuine documents with it, because real headings wrap:

7.4.  The Network Information Center and
      Requests for Comments Distribution Contact

Rejected, and RFC 778 is recorded as an accepted bound instead. Labelling the steps of a numbered procedure as sections is defensible; 56 - SET-UP was not. Not every false positive is worth a rule, and a rule that costs real structure to buy a marginal one is a bad trade even when its net count looks fine.

What the corpus did buy: clause 10. A recurring convention numbers top-level sections 1.0, 2.0, mixing the two freely — RFC 2006 runs 6 then 7.0, RFC 2024 runs 1.1 then 2.0. Read literally those are depth changes no walk can accept, and the document is rejected whole. Normalising a trailing zero fixes it, and is safe precisely because a real subsection never carries .0.

Clause 10's own test then caught a bug the walk could not see. The walk normalised 2.0 to 2 and accepted the document — while the heading stack still used the raw depth, nesting 2.0 under 1.0. The document passed and the hierarchy was wrong. Two places consume a number's depth and only one had been taught the convention.

Generalisable, and the reason the doubling protocol matters: clause 9 was derived from 66 documents and looked complete; 131 documents produced a false positive it could not catch. A fix validated at one corpus size has been validated at one corpus size. Every round both re-checks the previous fixes and gets a vote on the next.

The sync-CPU number, and the instrument proving itself in the field (20260805 21:56)

The measurement item 3 demanded since 20260804 was finally run, and it reverses the item's own framing. 55 modern-era RFCs — 16 557 chunks — rebuilt under fastembed:

wall-clock	1 497.7 s (~25 min), 1 451 samples
peak	500% — 5.0 of 10 cores
mean	480% — 4.8 of 10 cores

The loop is serial and the backend under it is not. sync.py embeds one document at a time, so the loop is single-threaded — but ONNX Runtime is already using half the machine beneath it. Item 3's own fork therefore resolves against the change it was written to consider:

The backend already saturates the machine → the loop is fine, and the win is a bigger batch (embedding several documents' chunks in one embed() call), not processes.

It also lands exactly on the trap that item named: "do not stack a process pool on top of a threaded backend". At 4.8 cores already consumed, two workers would take ~9.6 of 10 and anything beyond oversubscribes. The intuitive fix — a pool sized os.cpu_count() - 1 — would have been nine workers on a machine with room for two.

The instrument proved itself in the field, and it is the reason to trust the number. Sampled live from the same process tree:

process	%cpu
measure_sync_cpu.py	0.7
uv run	0.0
the actual pnk sync python	491.9

The pre-fix tool watched the launched pid — uv run — and would have reported 0.0 cores for a workload using five. That is not a number anyone would have questioned: it is the finding item 3 went looking for, and it would have licensed exactly the process pool this measurement rules out. A tool whose failure mode is "confirms your hypothesis" is the most expensive kind.

Bounded, and the bound is stated rather than buried: fastembed only. sentence-transformers needs the 2 GB [st] extra and is unmeasured, so nothing here licenses a claim about torch.

A second number fell out of the same run, unasked: 15 559 of 16 557 chunks carried a heading_path — 94%. The corpus that opened this whole line of work indexed 106 806 chunks with zero. The grammar works on real documents, and that is the first evidence of it outside a test.

Removing a blanket guard exposed what it had been incidentally protecting (20260805 22:11)

MEDIUM — and the finding came from an existing test, not from reading the diff. pnk init's emptiness check was removed so a directory with content could be adopted. test_ci.py then failed: it asserted that an existing .github/workflows/pinakes.yml is refused, and its evidence was the string "not empty".

The obvious read is "update the assertion". The real finding is that the emptiness check had been holding a second job nobody had written down. write_workflow does refuse to overwrite a hand-edited workflow — but it runs after pinakes.toml has been written. With the emptiness check gone, that refusal would leave a half-made KB: a manifest the user never asked for, and a re-run that now fails with "already a KB", whose only way forward is deleting a file init created itself.

Moved to _check_target, before any write, with the failure mode in the comment. The test now asserts "already exists" — more precise than "not empty", and it names the only thing actually in the way.

A second decision was refined in the same pass, and it is recorded rather than quietly taken. The decision as written said "add a refusal naming any file init would write that already exists". Implemented literally, that refuses on README.md and .gitignore — which a real repository always has — so adoption would still have been impossible in exactly the case the item exists for. The intent was "do not destroy the user's files"; the implementation honours it by never overwriting and reporting, which is strictly safer than refusing and actually achieves the goal. --ci is the one exception, because it is an explicit request rather than a side effect.

Generalisable: a coarse guard removed is not one behaviour removed. Before deleting one, ask what else has been quietly standing behind it — and take the failing test that follows as evidence about the system, not as a chore.

Slimming CLAUDE.md — a relocation's real cost is its pointers (20260806 00:33)

HIGH — the reference sweep has to run on the source file's name, not on the text that moved. Extracting two sections out of CLAUDE.md left seventeen citations across the tree pointing at content the file no longer carries. Not one of them quotes the moved wording; they name the file — `CLAUDE.md` calls "the free path stays free" non-negotiable, "the invariant CLAUDE.md gains says so explicitly", "for the reason CLAUDE.md states". A grep for the moved sentences finds zero of them, which is exactly what "keep docs in sync — grep for what changed" instructs you to run.

The two sweeps measured the difference:

Sweep	Scope	Found
Neighbourhood audit, on the moved terms	docs/, tools/	4
Second pass, on the string CLAUDE.md	whole tree	13 more, all in src/ and tests/

src/ and tests/ were never opened by the first pass, because a docs-only change does not look like it touches them. They held the majority: embed.py, sidecar.py, cli.py, ids.py, extract/claude.py, extract/pageyield.py, budget/ledger.py, and five test docstrings, each citing an invariant that had just moved to docs/INVARIANTS.md. The rule now: after relocating anything out of a file, grep -rn '<that filename>' over the whole tree and re-judge every hit — a pointer names its target, so it is invisible to a content-based search.

MEDIUM — docs/INVARIANTS.md is an index because the facts already had owners. Before extracting, each of the nine invariants was checked against the docs that would hold it: eight were already stated in DESIGN.md (§1 allowlist, §2.2 sidecar, §3 storage/schema, §5 ledger and Decimal), MANIFEST.md (the id rows, the round-trip bounds table), VERIFICATION.md (§ The sidecar round-trip) or CLI.md (--clear-cache=paid). A verbatim move would have created a second copy of eight facts inside the file set whose stated rule is one fact, one home — and a second copy drifts silently, because nothing compares them. So the page links each owner and writes out only the five implementation rules nothing else states.

MEDIUM — the no-loss check has to be mechanical. Normalising every sentence of the old file and matching it against the new homes flagged 29 candidates: 25 were false positives and 4 were real losses — a measured number (980 RFCs), the "first time since it opened" qualifier, a plan's self-description, and the why behind read the clock, never compose a timestamp, whose only in-repo home had been the sentence being deleted. Re-reading the diff would not have separated those four from the twenty-five; only a per-sentence check did.

Not fixed, out of this change's scope: tests/test_extract_claude.py and tools/record_claude_fixtures.py both still require a local YYYYMMDD HH:MM for a recording's --at, against the repo's UTC rule adopted 20260804 11:32. One for plans/20260731_1202-open-corrections.md.

A plan's blocking premise was inferred, not measured — and it was wrong for two days (20260806 04:05)

HIGH — the reason a plan gives for a blocker is reused long after the blocker is gone. plans/20260805_1721-metadata-as-retrieval-context.md said tests/demo-kb could not measure the injection experiment because "no section spans multiple chunks, so there are no continuation chunks to rescue — the mechanism has nothing to act on". Measured with the real chunker and the real tokenizer over all 30 documents: 30 of 30 sections span more than one chunk, and 29 of the 30 continuation chunks do not contain their own heading text. The corpus carries the exact mechanism the plan said it lacked.

The error was an inference from the wrong mechanism. The plan's fact was right — the documents are ~7 lines — but it reasoned that a short document fits in one chunk. chunk.py splits on paragraph blocks first (Block is "one paragraph under one heading path") and only then applies token limits, so a 7-line document with two paragraphs yields two chunks of 27 and 31 tokens under a 120-token budget. Nothing about the token budget was ever reached.

What makes this worth recording is that the plan's conclusion was right. Use the RFC corpus — correct. So the wrong premise cost nothing at the time and would have cost a great deal later: it is the sentence a future agent quotes when deciding whether some other experiment can run on the demo KB. A conclusion is checked when it is acted on; a reason is copied forward unexamined.

MEDIUM — the honest reason demo-kb cannot license this is arithmetic nobody had done. 66 answerable questions, 4 misses, and 56 of 62 hits already at rank 1. The whole improvable pool on recall@k is 4, and the project's own sign_test(4, 0) returns p = 0.0625 — so a perfect result fails the p < 0.05 bar the graph channel was held to. That is a power limit, not a mechanism limit, and the two have different remedies: a mechanism limit is fixed by a different corpus, a power limit by a different metric or more questions. Writing the wrong one down sends the next person to the wrong fix.

MEDIUM — reasoning from a committed artifact nearly produced a confident wrong number. The miss count above was first read out of tests/demo-kb/eval/outcomes.json, whose header predates G5 — no graph_channel, no edge_kinds, no retrieval.adjacent_k, and graph_gate.read_leg reports its channel as (absent). Two independent reasons the number could not be trusted from that file alone: the artifact is only rewritten under --write-baseline, so CI never refreshes it; and compare() tolerates a ±0.02 drift, which spans 4 to 6 misses — and sign_test(5, 0) = 0.0312 passes. The claim "a perfect result cannot license" was therefore one question wide. Re-running the eval settled it: aggregates identical, and all 74 rows match the committed file exactly. The rule is not "distrust artifacts" but an artifact whose header does not match the current binary is evidence about the past — the identity check graph_gate.check_identity performs on legs is the same check a reader owes any committed number.

MEDIUM — four silent failures sat in front of an experiment costed at "~2 h rebuild + eval". The RFC corpus stamps no max_tokens, so the default 510 applies against a measured window of 512 with 2 special tokens: zero headroom, and prepending anything pushes every full chunk past the window. Embedding an over-length string raises no warning and no error (measured, empty warnings list), and assert_chunkable cannot catch it because it validates max_tokens, never max_tokens + prefix. The direction is what makes it dangerous: truncation removes text from exactly the long chunks the hypothesis is about, biasing the result toward no movement — a false negative that reads as a clean result. Separately, the lexical channel cannot be injected at all without a new chunks column, rewritten FTS5 triggers and a schema_version bump, because chunks_fts is an external-content table filled by triggers copying new.text.

The estimate was not wrong about the run; it costed the run alone. Step 2 is three increments and a measurement, and none of the three was visible from the plan as written.

LOW — a plan that forbids retrying below "the threshold" has to name it. The document's anti-circularity clause read "a result short of the threshold is reported rather than retried" while nothing in it ever said what the threshold was. An unfalsifiable gate is not a gate; it was fixed by decision rather than discovered, but it survived a full adversarial review of the plan because the sentence sounds like a commitment.

2a — Real titles and a measured chunking reserve for the RFC corpus (20260806 06:12)

The plan's worked reserve was wrong by more than twofold, and only measuring found it. The injection experiment's plan offered 480 as the corpus max_tokens, from "a measured max prefix of 30" — RFC 9110's largest title > heading_path with section numbers stripped. Measured properly, over every heading path of RFCs 8600-8799 (195 documents, 5 of the 200 numbers unpublished), tokenised with the corpus's own BAAI/bge-small-en-v1.5: the largest prefix is 68 tokens, the per-document largest has median 31, p95 51, p99 61. The median document already exceeds 30. RFC 9110 was an unrepresentative sample for one reason nobody spotted: its title is two tokens long, and titles run to 32.

Reserving 30 would have truncated roughly half the corpus's longest chunks — silently, since an over-length embedding input raises no warning and no error. That truncation removes text from exactly the continuation chunks the hypothesis is about, so it biases toward no movement: a false negative that reads as a clean result. The reserve shipped is 96, deliberately 41% above the measured maximum, because 200 numbers is under a third of the modern band.

The plan said "do not re-derive these" about a table whose own line numbers had already drifted. Same failure class, one level up: a number recorded as measured invites reuse, and reuse is exactly what makes a sampling flaw permanent. The four "read this before writing a line" findings in §2 were all correct and all load-bearing; the one worked number beside them was not.

A defect I nearly introduced, caught in my own adversarial pass. Preserving the KB id across a re-run — needed once the output directory holds permanent document ULIDs — makes a re-run look identity-preserving while write_kb still rewrote pinakes.toml. That would silently discard the [retrieval.confidence] thresholds 2c will fit onto this corpus, with every command reporting success: the same KB by its id, no longer the same KB by its calibration. The fix removed the reason rather than the symptom — an existing manifest is not rewritten at all, so the id is stable by construction and the existing_kb_id reader that motivated the concern was deleted. Its cost is that a re-run does not adopt a changed [chunking], so the run now names both values.

What nothing else in the repository exercises: a title with YAML punctuation. RFC 8713 is "IAB, IESG, IETF Trust, and IETF LLC Selection, Confirmation, and Recall Process: Operation of…". Every committed corpus is hand-titled in plain words, so the first colon in a title: arrives from the RFC Editor — into the one file holding a document's permanent ULID. ruamel quotes it and it round-trips; that is now asserted through sidecar.read rather than assumed.

Verified by execution, which the plan had none of. Two RFCs built and synced with the real fastembed backend: both documents indexed under their published titles, 801 chunks, largest token_count exactly 414 — so the stamped cap binds on real text, as finding 1 said the default 510 does against a 512-token window.

A bound left in place, not fixed. corpus.json describes the run, not the directory: a re-run over a smaller --rfcs set records fewer RFCs than docs/ still holds. Pre-existing, and now slightly more visible because titles.kept_from_earlier_run hints at accumulation. Worth fixing before the run at 2f, where the corpus's composition has to be attributable.

2b — Refusing a prefix that would not fit (20260806 08:24)

A refusal with no caller is a refusal nobody has watched fire, so this one was written to be runnable on day one. The manifest option that turns injection on belongs to 2d, so nothing on the indexing path calls assert_prefix_fits yet. The temptation was to ship the check alone and let 2d supply the thing that exercises it. What shipped instead is the whole prefix construction — metadata_prefix, embedding_text, and the unnumbered_heading_path they read — so the refusal measures exactly the string that will later be embedded, and every part of it is reachable from a test today. 2d then adds one option and one call, not a mechanism.

The additive estimate was measured rather than argued. The check counts each distinct prefix once, with its separator attached, and adds chunk.token_count rather than re-tokenising every injected string — a document has orders of magnitude fewer heading paths than chunks. That is only safe if the sum never under-counts the concatenation. The reasoning (a tokenizer that splits on whitespace before merging cannot produce more tokens from prefix + sep + text than from its parts) is sound but is the kind of reasoning that is wrong once per project, so it was run: against BAAI/bge-small-en-v1.5, over 43 503 chunk/prefix pairs from 195 RFCs, the estimate was exactly equal to the concatenation's real token count every time — not merely bounding. The same run reproduced 2a's corpus figures from an independent code path (largest prefix 68, per-document largest median 31 / p95 51 / p99 61, longest title 32) and confirmed the refusal fires for 195 of 195 documents at the default max_tokens = 510 and for none at the corpus's 414.

The reserve is checked, not the worst chunk in hand, and that was a decision. Per-chunk pairing is more permissive and more exact: it refuses only what would actually truncate today. It was rejected because the two legs of an A/B comparison must chunk under the same max_tokens or they are different corpora — so what has to be safe is the setting, not this morning's text. A document that passes because none of its chunks happens to reach the cap would start truncating on the next edit, mid-experiment, silently. Refusing the setting is stable across documents and across edits to them.

A field added to a frozen dataclass is a field somewhere else forgets to copy. _with_pages rebuilt every PDF chunk field by field to attach page numbers, so unnumbered_heading_path would have arrived as None for PDFs only — green suite, silent gap. It now uses dataclasses.replace, which cannot omit a field, and a test pins the property. The same reasoning kept the field out of as_row: the stored form is the citation form, and a second column is a second thing to keep in step with it.

Markdown keeps whatever number its author typed, and that is the same rule rather than an exception to it. ## 1. Introduction yields 1. Introduction in both paths, because nothing parsed a number there — # is syntax and is already gone, and the text after it is the author's. Only the grammar that parsed a number is entitled to remove one. A regex over the joined string would have been shorter, would have drifted from the grammar's own rule, and would have eaten the 404 from # 404 Not Found.

Two findings from the increment's own adversarial pass, both about a value with more than one home. The first: the prefix's join was introduced as a new constant, which would have made four literal " > "s in this module plus graph/edges.HEADING_SEPARATOR reading them back — for a format that is persisted in chunks.heading_path, where a disagreement empties three edge kinds and reports nothing. It is now one HEADING_JOIN used at every site that builds a heading path, with the consuming copy named in its docstring. The second: title is the user's field in a hand-edited sidecar, so it can be "" or " ", and the first draft treated whitespace as content — injecting a separator with nothing in front of it into every chunk of that document. Neither would have failed a test or a run; both are the shape this project keeps meeting, a value that is wrong in a way nothing reports.

2c — A golden set authored by agents that did not know what it was for (20260806 11:42)

The anti-circularity rule was implemented rather than promised. The plan requires the questions frozen before any injection code exists, "so that no number can influence them" — and the deeper risk it names is fitting the question set to the mechanism, which "is undetectable afterwards". An author who knows the hypothesis cannot prove they ignored it, and no reviewer can check it from outside. So the set was authored by six agents over disjoint document slices, each told only that it was writing an evaluation set for a retrieval system and forbidden from reading this repository, where the plan would have told it. Blind authorship is the only version of that guarantee that survives review.

The exit criterion failed the first time, and how it failed was the useful part. 70 questions produced an improvable pool of 9 against a criterion of 10. The shape mattered more than the number: 51 of 60 answerable questions were already at rank 1, lexical and simple-lookup both scored 1.00, and 8 of the 9 pool members were paraphrase. On a corpus of distinctive technical vocabulary — protocol acronyms, registered code points, field names — BM25 with a reranker essentially solves the classes that share words with their document.

The obvious fix was the wrong one, and it would have passed. Authoring more paraphrase questions reaches a pool of 10 quickly. It also enriches the set with exactly the class a change to the embedded text is most likely to move, which is fitting the instrument to the hypothesis one step removed: the criterion would pass and the result would mean nothing, with nothing in the artifact to show it. The two new slices carried the same proportional mix as the first four, over 135 documents no question had touched. The pool went to 15, and it changed shape — 11 paraphrase, 2 lexical, 2 simple-lookup, where before it was one class. A pool spread across classes is what makes compare()'s per-class guard able to catch a change buying one class out of another.

The expensive error in a question set is self-concealing, so the set carries its own evidence. A question pointing at the wrong document looks exactly like a retrieval miss. It would have inflated the pool and made the power criterion pass for the wrong reason — the measurement then resting on questions nobody could answer either. Every answerable question therefore records the sentence from its document that answers it, verbatim, and tools/verify_rfc_golden_set.py refuses the set if a sentence is not there. All 96 verified. Whitespace is normalised because RFC bodies are hard-wrapped at ~72 columns, which also means the recorded evidence is often the fragment on one line rather than a whole sentence.

A slice can only check its own slice. Each author confirmed its no-answer topics absent from its own 50 documents. Checked across all 195, three had mentions elsewhere — "camel case" in rfc8618 against a question about the CAMEL protocol, one mention of blockchain databases in rfc8673, Bluetooth as a transport example in rfc8628 and rfc8793 — and later, SCADA as a category label and NFC as both a URI transport and Unicode Normalization Form C. None answers its question, and all were kept deliberately: a calibration set wants plausible near-misses, and a lexical collision that answers nothing is exactly that. Two topics were duplicated across slices and dropped.

filter and multi-hop are absent by decision. filter needs metadata worth filtering on and these sidecars carry a title and nothing else. multi-hop is the graph channel's class and graph_channel is off here, so such a question would score as an ordinary lookup while being reported under a name claiming otherwise.

The corpus is not committed, so everything that reads it had to be. The questions, the fitted confidence thresholds and the before leg all live in the repository; the 195 documents do not. That is the same lesson build_rfc_corpus.py was written for — a 300-RFC corpus once produced this project's most useful finding and died with the machine that held it. The thresholds are stamped into the builder's manifest template rather than pasted into a generated pinakes.toml, which also makes both legs of the comparison fitted identically by construction: refitting after a change would measure the refit.

A defect found by asking what would protect the artifact this increment produces. Capturing a before leg raises the question of what stops it being compared against an after produced differently — and graph_gate.check_identity turned out to guard only the graph channel's own settings. Reading eval.header for what it does record showed the gap was one level lower: the function's docstring promises "every setting that can move a row", and [chunking] was not in it. Every other outcome-deciding setting was. So the leg committed here could not have been shown to have been chunked at 414 rather than the default 510 — the precise confusion 2a's reserve and 2b's refusal exist to prevent, surviving into the artifact that records their result. Three fields, no schema bump, because no row changed.

2d — The screen said no, and the controls are what make that worth believing (20260807 09:07)

The result: 6 improved, 6 regressed, 84 unchanged, over 96 answerable questions. The pre-registered criterion was strictly more improvements than regressions, so the screen is a no-go and the schema_version bump at 2e is not taken. Per the pre-registration these numbers are the whole report — they are not evidence for or against the hypothesis in either direction, and they appear here and in the increment's commit message and nowhere else. The screen's own artifacts were deliberately not committed.

What was measured, stated exactly, because a null is only as good as its controls. Injecting title > heading path into the text that is embedded — the vector channel alone — moved nothing net on this corpus at rerank = "none". The both-channel form was not tested: the lexical channel needs a new chunks column and a schema bump, which is the cost this screen existed to decide. The dilution objection that disqualified vector-only as a gate applies in full to this null: RRF fuses an injected vector channel against an unchanged BM25, so a real effect is attenuated before it reaches a rank.

A null result is a claim about the world only if the instrument was pointed at it, so four controls were run before the comparison — three of which the plan did not ask for.

Control	Result
The uninjected index still reproduces 2c's baseline (rerank = "local")	110 of 110 rows identical, twice — once on main's binary, once on this branch with the option off
Both legs are the same corpus	195 documents, 43 353 chunks, and one sha256 over every chunk text, equal
The injection actually reached the vectors	mean cosine 0.8398 between the before and after vectors of 2 000 sampled chunks; zero unchanged
The prefix was the intended string	195 of 195 published titles, zero filename stems — finding 5's confound absent — and 93.2% of chunks carrying a heading path

The third is the one that decides how to read the null. Chunk texts are byte-identical between the legs by construction, so if the injection had silently not happened, every artifact in the experiment would look exactly as it does now — same corpus, same questions, a clean flat result — and the conclusion would be drawn from a no-op. Measuring the vectors is the only thing separating "no effect" from "no injection", and it cost one script.

The option-off path being a verified no-op is not a formality either. The same binary that carries the injection reproduced the frozen baseline row for row, which is what licenses comparing this branch's after leg against a before leg captured on it.

The shape of the movement is more interesting than the count, and it is not being reported as a result. Of the 6 improvements, 5 were paraphrase — the class the hypothesis targets, and the only class with power on this corpus. Of the 6 regressions, 2 were simple-lookup questions that had been at rank 1. That is what a dilution cost looks like: a prefix adds tokens to a vector whose question needed none of them. Twelve of 96 rows moved, so the mechanism is doing something; it is simply not doing more good than harm through one channel. This is an observation about a measurement that was pre-registered not to be interpreted, and it must not become the premise of a retry — the anti-circularity rule says a result short of the threshold is reported rather than retried with a different injection format.

Two things surfaced by building it that the plan had not anticipated.

• The refusal is a per-document failure, not an aborted run. assert_prefix_fits raises a ChunkingError from inside _index_document, and every PinakesError there is already caught by sync's per-document handler: the transaction rolls back, the document is named in the report and recorded in the index for pnk doctor. That is the right shape — one pathological heading path should not cost a 195-document corpus its other 194 — and it still removes the silent truncation, because a refused document is not indexed at all. But "refuses the corpus" was the plan's phrasing, and the test written for it originally asserted a raise that never comes.
• With injection on, every document is prefixed, because skeleton() falls back to the filename stem. A document can reach the embedder with no heading_path; it cannot reach it with no title. So on an uncurated corpus the injected string is a filename — finding 5's condition, now located at the sync boundary rather than in the abstract. It is why the RFC corpus mints published titles before its first sync, and it is the strongest argument for the option defaulting off.

A test that passed vacuously, caught during development and worth naming. The assertion that the injected prefix uses the sidecar's title was written over a sidecar-only edit — which is a RefreshMetadata action, so it updates the row and re-embeds nothing. all(...) and not any(...) over an empty list are both true, and the test was green while proving nothing. The fix was --rebuild plus an explicit assert backend.embedded precondition. Any assertion of the form "everything embedded looks like X" needs a companion assertion that something was embedded.

tools/two_leg_gate.py exists because the instrument had the same gap one level up. 5993521 made an eval artifact record the chunking it was produced under; nothing compared it — graph_gate.check_identity takes three legs shaped to the graph channel and inspects k, embedding, rerank, ranking and retrieval, but not chunking. Two legs chunked at different max_tokens therefore compared clean, which on one RFC is 63 of 1 858 chunk texts differing: a rechunk reported as the effect under test. The new tool refuses on any header difference outside one named key, and it excepts that key by path, not by block — excepting the whole chunking table would hide exactly the rechunk it is there to catch.

2d review — the grep that was necessary and not sufficient (20260807 09:54)

HIGH — a rebuild could leave a half-injected index, and the check that should have caught it was looking for the wrong thing. Before landing the injection, this increment verified that sync.py has exactly one .embed( call on the indexing path — grep -rn '\.embed(' src/ returns that one, plus two query-side calls. The conclusion drawn was "every vector on the indexing path goes through the switch". It does not follow, and the counter-example was already in the file: _copy_forward_protected_document writes rows into embeddings with an INSERT … SELECT from the index being replaced. It produces vectors without embedding anything, so no grep for .embed( could ever have found it.

The consequence was the failure class this project exists to prevent. A KB with a paid-extracted PDF, [chunking] metadata flipped to "prefix", and pnk sync --rebuild: the protected document keeps its uninjected vectors, set_meta stamps chunking_metadata = "prefix" over the whole index, and the next pnk sync and pnk doctor both report no drift. Every command succeeds, and half the index is injected.

The fix separates the two costs that had been treated as one. The docstring said the document is "never re-extracted, never re-embedded", as though those were the same protection. Extraction is what spends money; embedding is local and free, and the chunk texts are carried forward anyway. So the chunks are still copied and the extraction still never re-run, and the vectors are recomputed under this run's settings — which fixes both directions, since turning injection off again had the mirror-image defect.

One guard is deliberately louder than it needs to be today. unnumbered_heading_path is not persisted (chunk.Chunk says why: the stored form is the citation form, and a second column is a second thing to keep in step). A carried-forward chunk therefore cannot say what its path looks like with the section numbers removed, so injecting the stored form would quietly prepend the citation form this experiment measured at 44% numbers and rejected. Unreachable now — only PDFs are ever protected and the PDF path records no heading path — but step 5 of this plan is PDF layout heuristics, which is exactly what would make it reachable. It refuses with a named remedy instead.

What the review round found beyond that, all of it in code this increment wrote:

--sign-test printed FAIL at 0.05 and exited 0	the flag names the 2f gate, which licenses the irreversible schema bump; a driver branching on $? would have taken it
A miss was written to the artifact as Infinity	invalid JSON: JSON.parse rejects it, jq silently coerces it to 1.8e308 — the one outcome the rank ordering exists to make visible became a very good rank
A truncated artifact exited 1, same as a genuine no-go	read_outcomes only refuses a file that parses; a JSONDecodeError is a ValueError, which the first version of the handler did not catch
Nothing recorded which leg was which	transposing --before/--after inverts the verdict, and the identity check cannot catch it — it is never told which value is the baseline, only that the two must differ
pnk doctor's half of the drift promise was untested	a mutant reading a constant there passed the entire suite
Two more vacuous assertions	backend.embedded == [rows] is [] == [] when a run indexes nothing; all(…) over an empty list again

Three of those six are the same defect in three costumes, and it is worth naming once. An assertion of the form "everything we produced looks like X" is silently satisfied by producing nothing. It has now been found three times in this increment alone — once during development, twice in review — so the rule is: any assertion over a collection the code under test produced needs a companion assertion that the collection is non-empty, and preferably one that names the expected count.

The review that found this was itself a measurement, and it half failed. Five adversarial lenses were run as independent agents; a usage limit killed 15 of the 17 agents mid-flight, including every verifier. The workflow returned {"confirmed": [], "refuted": []} — which reads exactly like a clean review and was nothing of the sort. The findings were recovered from the agents' own transcripts and verified by hand. An empty result from a harness that partially failed is not evidence of absence, and a report that does not distinguish the two is worse than no report: confirmed: [] was one careless sentence away from becoming "the review found nothing".

2d review, round two — the argument for the key's placement was false when written (20260807 10:34)

HIGH — the option was put in [chunking] rather than [retrieval] because [chunking] is recorded in the index and therefore cannot flip silently. That reasoning was correct about the mechanism and wrong about every KB in existence. chunking_drift treats a key absent from meta as unknown, never drifted — the rule that stops an upgrade demanding a rebuild of every index — and chunking_metadata is absent from every index built before this release. Only a --rebuild ever stamps the chunking identity, so the absence is self-perpetuating. Measured on a KB built by this branch with the key then deleted, which is exactly the shape a 0.15.1 index has:

drift ()            embedded 0          backend calls 0
pnk doctor:  OK  chunking coherence: index matches the configured chunking

An affirmative OK over uninjected vectors, forever. For a pre-existing index the two manifest sections behaved identically, which is the thing the placement argument was written to rule out.

The fix turns on a distinction the original rule did not need. max_tokens and overlap have been settable since v0.1, so an index that fails to record them could genuinely have been built under any value: absence there is ignorance. chunking_metadata is different — no release that could have written any existing index was able to inject anything — so absence proves off. store.ABSENT_MEANS says so, and because it resolves to the default, it fires only for a user who explicitly opted in: the compatibility guarantee it looks like it threatens is untouched. Both directions have tests, and the second one matters as much as the first — an unclearable warning on every upgraded KB is the failure mode the heading-coverage check already had to answer for.

MEDIUM — the fix for round one's finding shipped with the same class of hole inside it. Round one found that --rebuild carried a protected document's vectors forward, and the fix re-embedded them. But the re-embed called embedding_text without calling assert_prefix_fits, so the one path that re-embeds without re-chunking became the only path with no truncation guard — and it is the path that needs one most, since its chunks were sized by whatever max_tokens built the previous index and are never re-chunked. A fix for a silent-truncation defect that reintroduced silent truncation one function away.

MEDIUM — and it also introduced a way to publish a half-written document. DETACH requires the transaction closed, so that function commits in a finally; with the writes inside that block, a document whose embedding failed was committed active with chunks and zero vectors, which _apply's rollback() could no longer undo and --rebuild's unconditional index swap then published. No later sync repairs it: the file's content hash is unchanged, so every future run says Skip. Now the old rows are read under the attach and every write happens after it, in one transaction the caller can still roll back.

MEDIUM — the eval artifact labels a leg from the manifest, never from the index. Every [chunking] value in a header is read from pinakes.toml at eval time. Because flipping metadata changes no chunk's text, hash or span, an eval run against an index that was never rebuilt produces a byte-for-byte plausible artifact stamped metadata: "prefix" over uninjected vectors — and tools/two_leg_gate.py would accept it as the injected leg, since it compares headers to headers. The instrument that licenses an irreversible schema bump could be handed a leg that never existed. eval.run now compares the index's recorded chunking against the manifest and refuses before scoring a single question.

The four rules this round leaves behind.

1. A grep for the operation is not a proof about the outcome. Round one's defect survived a check for every .embed( call site because the offending path produces vectors with INSERT … SELECT. Round two's survived a reading of chunking_drift because the defect is in what the function does with a key that is not there. Both times the search was over the wrong set — the question is never "where is this called" but "what else can reach this state".
2. A fix is a change, and gets the same review as one. Two of this round's findings are in code written to fix round one, hours earlier and with the defect fresh in mind.
3. State an argument in terms of the population it covers. "[chunking] is recorded, so the flip is reported" was true of indexes this release builds and false of every index that existed. The sentence never said which it meant.
4. A partially-failed harness must not report like a completed one. Round one lost 15 of 17 agents to a usage limit and returned {"confirmed": [], "refuted": []} — indistinguishable from a clean review. The findings were recovered from the agents' transcripts by hand; one was the HIGH above. Round two ran complete: 14 findings, 14 judged, 9 confirmed, 5 refuted.

T1 — The version archive, and what mutation testing found in it (20260807 19:36)

HIGH — the plan's own baseline undercounted the sites it had just corrected, and the second count was as wrong as the first. The template-release plan was re-verified against main at 71911e2 on 20260807, and that re-run specifically corrected the notes@1.0 site list: it had said "both, in test_init.py", and the correction raised it to "six sites in five files", with a box explaining that the original defect was a grep scoped to one file. Running the corrected command at that same commit returns nine sites in eight files. The three it still missed were tests/test_sync_links.py:66 and the two committed KB manifests, tests/demo-kb/pinakes.toml and tests/partner-kb/pinakes.toml. The lesson is not "grep wider" — the plan already said that. It is that a count in a document is a measurement, and re-running the command is the only thing that distinguishes a corrected count from a confidently wrong one. The correction was written and believed; nobody re-ran it.

HIGH — a formatter can edit a file the project has promised is frozen. The archive's whole value is byte-identity: _versions.toml records a SHA-256, and pnk upgrade will diff against those exact bytes. check.sh runs ruff format --check . over the whole repository, and ruff reformats Python inside Markdown fences. A template README.md that ever gains a python fence would be rewritten in its archived copies too — the project's own formatter editing a version that already shipped — and it would surface as a ledger mismatch one leg away from its cause. Latent today (no template README has a fence) and closed with a [tool.ruff] extend-exclude. Generalisable: whenever a repository declares some bytes immutable, list every tool with write access to them. The gates were audited; the formatter was not, because it is not a gate.

MEDIUM — mutation testing found a branch no input could distinguish, which is the same defect class as a gate that cannot fire. git_history_reason probed --is-inside-work-tree and then --is-shallow-repository. Neutering the first changed no test result: outside a checkout the second exits 128 anyway, so the first was dead weight wearing the clothes of defensive care. It was removed. This increment exists because pnk doctor carried a check that could never fire for eleven releases; shipping a redundant branch inside its fix would have been the same mistake at one-tenth the scale. A branch that no test can tell apart from its absence is a branch nobody knows works.

MEDIUM — the first mutation run was invalid, and its invalidity was informative. Three mutants changed content_hash and turned the whole suite red rather than one test, because the committed ledger pins the hash function: change it and leg (iii) fires everywhere. That coupling is a feature — the hash definition cannot drift silently — but it destroys the per-assertion signal, so the harness was corrected to regenerate the ledger under each hash mutant, which is the tree such an implementation would really have shipped with. Separately, two tests both tampered with README.md, so a mutant exempting the README failed all three. A mutant that travels to another test means the two tests share a vector, not that either is wrong — but the sharing is worth removing, because it makes every future mutation result harder to read.

MEDIUM — leg ordering decided which remedy the user is given, and the wrong order gives the opposite one. Editing a frozen _versions/<live>/ file also makes the live files differ from it. With the live-vs-archive comparison first, the gate reported "the live files differ from archived 1.1" and advised bumping the version — when what happened was that a published version was edited and needs restoring. The ledger check now runs first, so each fault reports its own leg. Found by running the failure paths, not by reading the code: both orderings look correct on the page and only one is.

LOW — "exactly one commit" was unsatisfiable in the increment that introduces it. Leg (vii) as specified fails on its own landing commit, because ./check.sh runs before the archive is committed and git log returns zero. It is at most one: zero means not yet committed, one means added and never touched, two or more is the violation.

HIGH — the gate reproduced, inside itself, the exact defect it was built to catch. Given --templates as a relative path, leg (vii) built its git log pathspec against the process working directory while git resolved it against the templates directory. It matched nothing, git log returned empty, zero commits read as "not committed yet", and the gate printed history leg (vii) ran … none edited and all legs green over a tree carrying the coordinated three-file edit. That is the strong mode claimed while nothing was checked — the thing this increment exists to stop pnk doctor doing. Every leg-(vii) test passed --repo explicitly, so nothing covered it. A flag that changes where a tool looks needs a test that runs it from somewhere else; resolving paths once at the argument boundary is the fix, and the general rule is that a relative path and a cwd argument must never be chosen independently.

HIGH — leg (vii)'s first design blocked the project's own procedure, and no single mechanism fixes it. Counting every commit that touched an archived directory fails a branch that adds one and then corrects it during review — which is exactly what docs/BUILDING.md requires (green ./check.sh before review, review fixes in their own commit). The failure text said the archive "still says what the version said when it shipped" about a version that had never shipped, and the only escape — amend or rebase — is the operation that also defeats the leg. Two candidate fixes each looked sufficient and each was blind: counting only commits already on origin/main stops catching the coordinated edit before it merges; comparing content against origin/main stops catching an edit that has already merged. The leg needs both halves, and the way to see that was a table of three scenarios against two candidate rules, not a closer reading of either candidate.

MEDIUM — a test asserting on a configuration was satisfied by the prose describing it. The test pinning fetch-depth: 0 onto the gate's CI job grepped the workflow file for the string. The job's own comment explains why the setting is there and contains it, so deleting the setting left the test green. Caught only by mutating it. Grep a config file and you assert about a document; parse it and you assert about the configuration — the test now loads the YAML and reads jobs.template-drift.steps[checkout].with.fetch-depth.

MEDIUM — nothing pinned that the gate was wired in at all. Deleting the check.sh line and the entire CI job left all forty-odd tests in tests/test_template_drift.py green, because every one of them drives the tool directly. A test suite for a gate proves the gate works; it says nothing about whether anything runs it, and those are different claims. Two assertions now cover the second one.

MEDIUM — the hash covered bytes that do not ship, and could bake them into the ledger. It walked the working tree, so a gitignored .DS_Store in the template directory turned ./check.sh red on a clean checkout with the remedy "bump the version and archive the new files". The worse direction: present while --print-hash generated a ledger row, it was folded into the committed sha — leaving the author green and failing only on a clean CI checkout, pointing at an archive nobody had touched. The fix is ignored, not untracked, and the distinction is load-bearing: measured against a real hatchling build, a gitignored file does not reach the wheel while an untracked-but-un-ignored .orig does. Hashing git's tracked set would have hashed away a stray file that really publishes, and given a brand-new archive the digest of the empty string.

Process note — two independent skeptics disagreed, and the disagreement was the useful output. One confirmed the stray-file finding; another refuted it, having implemented and measured a tracked-set fix and shown it strictly worse. Both were right about what they tested, and neither had tested the rule that was actually adopted. A refutation kills a proposed fix, not necessarily the finding — the two have to be judged separately, and the second reviewer's evidence is what made the third option obviously correct.

T2 — Reporting template drift as a line count (20260807 22:28)

HIGH — A test that never calls the function under test cannot kill a mutant inside it. test_the_kb_identity_block_never_produces_a_hunk was written for the mutant the plan called the one that matters: rendering the old side with the KB's recorded reference and the new side with the installed one, which is what a reader of init.py:75 would naturally write, and which puts a [kb] hunk in every report on every KB. The test asserted exactly that property — by rendering both versions itself. So the mutant went into doctor._template, the test rendered its own two sides correctly, and it passed. The property was right, the altitude was wrong. It now also asserts the count doctor reports, against a synthetic pair that differs on one line: correct is two, a leaking identity block is four.

HIGH — A line-count assertion is blind to an edit that substitutes rather than adds. The invariance test edits the user's manifest and asserts the reported count does not move, which is what catches a report built from pinakes.toml instead of from two archived templates. With that defect injected, replacing fake-model with fastembed-model and final_k = 5 with final_k = 4 left the count identical — one line replaced by another is still one line on each side of a diff. The test was invariant under the implementation it existed to reject. Appending a comment line fixes it, because no substitution can absorb an added line.

Both were found by running the mutation pass, not by reading the tests. Two of five mutants survived a suite that was green, and neither survivor was a bug in the increment.

MEDIUM — A comparison that reads one of four files must not report 0. A template version denotes four consumed files; this check diffs pinakes.toml.j2. A bump touching only eval/questions.yaml renders two identical manifests, and the first implementation said 0 lines differ — true of the manifest, read as nothing changed. It is not a hypothetical: of the ten commits between the notes template's first version and its second, five touched the golden set and none touched the manifest. It now says same manifest and names what a version covers beyond it.

MEDIUM — The most-read string promised something no release can deliver. The cannot compare remedy ended "from the next template version onward the comparison is automatic". Under D-2b notes@1.0 is deliberately unarchived, so a KB recording it stays uncomparable however many versions ship after — the sentence was false for precisely the readers who see it most, which is every KB in existence. What a later version changes is the next KB, and the remedy now says that and says the missing content is gone rather than pending.

A second copy of "the variables this build supplies" was one commit from existing. tools/template_drift_gate.py leg (vi) asserts every archived version renders, under a context written out in the gate. The product now renders both sides of a comparison under template.render_context. Two literals, and the failure mode is the gate staying green while pnk doctor raises on the KB in front of it — the gate would have been asserting that the archive renders under a context nothing uses. The gate now builds its keys from template.CONTEXT_KEYS, and test_render_context_supplies_exactly_the_declared_union pins the remaining seam.

T3 — pnk upgrade, print only (20260808 00:02)

HIGH — a same-length mutant is invisible to Python's bytecode cache, so a mutation run can report the exact false negative it exists to prevent. Two of T3's eight mutants — == 1 → >= 1 on each branch of the placement predicate — came back SURVIVED. Applying one of them by hand and running the same test killed it immediately. The harness was mutating src/, and CPython invalidates a .pyc on the source's (mtime truncated to the second, size): == 1 → >= 1 changes neither, so a mutation written and reverted inside one second was executed from stale bytecode. Every mutant that did die changed the file's length.

Two things follow, and neither is about this increment.

• A mutation harness must invalidate bytecode explicitly — delete the module's __pycache__ entry after writing and after restoring. Without it, "mutation-verified" is a claim the harness cannot support for precisely the smallest, most surgical mutants, which are the ones worth running.
• The failure was silent and reported success. SURVIVED and "this code path has no test" are the same string, and only one of them was true. What caught it was disbelief — the surviving mutant looked like it should have been caught by a test that named uniqueness in its own title — not the tooling.

And the disbelief was half right, which is the second finding. Once the cache was cleared, one mutant still survived: relaxing already applied from "at exactly one position" to "somewhere" killed nothing, while the same relaxation on the clean branch was caught at once. ::test_a_hunk_whose_context_matches_twice_is_a_conflict covered one branch of a two-branch rule and read, from its name, as though it covered the rule. ::test_an_already_applied_hunk_matching_twice_is_a_conflict_too is the missing half — a user who adopted a change and kept a second copy of the block — and it was written because a mutant demanded it, not because anyone re-read the predicate.

MEDIUM — the invariance test demanded the wrong property, and passing would have been worse than failing. "A user's edit never appears in the report" was written as "the two outputs are byte-identical", and it failed: a user's provider = "fastembed" does appear, as unchanged context, because the context is what their own manifest renders to. The property that matters is narrower — their edit never appears as a +/- line — and the first version would have forced the implementation to hide context lines to satisfy a test, which is the tail wagging the dog. The distinction is now in the test's docstring: the context lines are the user's, the changed lines are the template's.

MEDIUM — textwrap broke a command across two lines, and the wrap was added for readability. The first wrapped remedy printed run `pnk at the end of one line and init` on a throwaway directory at the start of the next — a remedy naming a command nobody can copy, produced by the change meant to make remedies easier to read. _fill now glues the spaces inside a code span before wrapping. Worth remembering wherever prose meant for a terminal contains something meant to be typed.

LOW — the plan predicted a mutant would kill two tests and it killed one, because the implementation is stronger than the plan assumed. Dropping uniqueness was expected to fail both the twice-matching test and the reordered-manifest one. It fails only the first: reordering is caught by matching the hunk's lines contiguously and in order, which is a different clause. A separate mutant — every line merely has to be present somewhere — kills the reordering test. The plan's prediction assumed a looser predicate than the one built; the two tests assert two properties, which is what it was really asking for.

HIGH — the "writes nothing" test could not see a write, and its docstring stated the reason backwards. T3's central claim is that the command writes nothing, and the plan names the mutation that proves it: open the manifest for writing and confirm ::test_nothing_under_the_kb_is_written fails. It did not fail. The snapshot compared the bytes of the files only, so a rewrite of identical content was invisible, and directories were filtered out, so mkdir(".pinakes") was too. Both mutants survived and the suite was green.

Two things made it hard to see, and the second is the lesson.

• The mutation run had reported this mutant as KILLED, because the mutant I wrote also dropped a stray file — a strictly stronger mutation than the plan's. A mutant that does two things cannot tell you which one the test caught.
• The helper's docstring asserted the opposite of the truth: "Bytes rather than mtimes: an mtime comparison passes for a rewrite of identical content." Mtime is precisely what catches that; bytes are what miss it. It read as a considered trade-off, which is why nobody re-derived it. A wrong reason stated confidently is worse than no reason, because it ends the inquiry.

The snapshot now compares the path set, the bytes and st_mtime_ns, over files and directories alike, with a table naming what each of the three is blind to on its own.

HIGH — the already applied predicate asked a whole-file question about a per-hunk fact. It tested whether the hunk's removed lines occur anywhere in the manifest. A manifest is comment-dense and repeats blank lines and bare # everywhere, so any hunk deleting one could never be already applied: the user who adopted that change by hand was told conflicts — and under T4's all-or-nothing rule, that refuses their whole --apply run. It now asks whether the hunk's before image is still there, which is the same question scoped to the hunk's own region, with the pure-addition case carried by an explicit not removed guard where a reader can see it.

MEDIUM — a control test passed under the very mutant it was written to kill. The first version of "a deletion not yet applied is still clean" deleted a line in the middle of the file. With trailing context, removing a line breaks the after image's contiguity on its own, so the first half of the predicate answered and the half under test was never consulted. Only a deletion at the end of the file — where there is no trailing context — reaches it. The shape of a fixture is part of what it tests, and "it deletes a line" was not specific enough to be a test of anything.

MEDIUM — _range and _section were written by hand and asserted by nothing. Six mutants across them survived: emitting a two-number range where a one-line range is bare, dropping the empty-range back-off, an off-by-one on the start, _section returning None for everything, reading the wrong side, and dropping the column from the listing. The diff is the part of this command a user may paste into patch, and its numbers had no test. What looked like coverage was assert "[budget]" in out — satisfied by the diff body's own context line, whatever the listing said. The range check is now a property test against difflib.unified_diff rather than literals, so it cannot drift with the fixture.

MEDIUM — one message, two copies, nothing watching. cannot compare was byte-identical in doctor.py and upgrade.py, with a docstring in the second saying "deliberately the same message pnk doctor prints" — a convention with no gate, which is the shape F1 already is in this same plan. It now lives in template.py beside the archive it describes, with a test that calls both surfaces and compares.

METHOD — a scratch copy of this repo that includes .venv runs the original worktree's code. Reported by the reviewer against its own first attempt: rsyncing the tree and running pytest in the copy reported all 29 mutants surviving, including "always return CLEAN". uv run python3 -c resolved the copy's source correctly while pytest did not, so a spot-check of one mutant would not have exposed it. rm -rf .venv && uv sync --frozen --all-extras in the copy is what makes the harness honest. Together with the __pycache__ finding above, the rule is one rule: a mutation harness must prove it can kill something before its silence means anything — run a known mutant first, and treat a run with no kills as a broken harness rather than a clean bill.

MEDIUM — three of the first review's own fixes shipped with no test, in a commit whose message said each had one. The second pass reverted them one at a time and the suite stayed green: the summary line's nouns, the cannot compare: prefix on the no-template path — which docs/CLI.md publishes as a scriptable contract, "every one of them opens with cannot compare:" — and the conflict trailer that had just been rewritten to stop asserting a cause it cannot know. A fix applied under review inherits the confidence of the review and none of its scrutiny. The rule that follows: a review's fixes get the same treatment as the code they fix, mutation included.

The same pass found the wrapper's glue restore untested — dropping .replace(_GLUE, " ") printed a private-use codepoint into the middle of run `pnk init` on the one message every KB in existence receives, with 31 tests green.

MEDIUM — a test written to hold a fix could not fail. _TABLE was tightened to stop a multi-line array's continuation line being reported as a table, and the test written with it wrapped include = [ over three lines. A wrapped array of strings has continuation lines opening with ", so both the loose and the tight pattern answered [sources] and the test passed under either. It is now a unit test whose fixture is an array of arrays, which is the only shape that discriminates — and the tightening turned out to have lost [[links.kb]] and [budget] # caps, two legitimate headers, which nothing had noticed either.

MEDIUM — a new test loaded real model weights, and CI could never have told us. The test comparing pnk doctor and pnk upgrade called the whole diagnose() on a fixture naming the real sentence-transformers provider. On a checkout with pinakes[st] it downloads weights, takes three seconds and dies on a FutureWarning that filterwarnings = ["error"] turns into a failure. CI's matrix is [light], [light,pdf] and [light,pdf,claude] — it never installs [st], so this would have been green on every run and red on the machine of anyone who had the default extra the README recommends. Naming a provider nothing registers keeps the report offline and instant.

And the plan is a document a later increment builds from, so its errors are not cosmetic. The second pass found T3's own specification still carrying the whole-file predicate this increment had just proved wrong, and the exit-criteria block still instructing "compare bytes and the path set, not mtimes" — the inverted sentence whose implementation let this increment's named mutation survive. T4 reads that page next. Both now carry a dated correction rather than a silent edit, because the wrong version had already been built once from the words as written.

MEDIUM — the remedy named the newest archived version where the sentence means the oldest, and one archived version is why nobody saw it. "A KB stamped from X or later is compared automatically" printed archived[-1]. With a single archived version [-1] and [0] are the same string, so the message read correctly on every KB that exists — and would have started telling covered users they are not covered at the next template bump, on both surfaces, in the message this project calls the path 100% of KBs take. A one-element collection makes two different intentions indistinguishable, which is the same reason T1's gate leg (ii) is vacuous today and says so out loud. The test that pins it uses the fixture that archives two.

MEDIUM — the correction broke the thing it was correcting. Pass 2's ⚠ blockquote about the placement predicate was inserted between rows 1 and 2 of the predicate table. In Markdown a table block ends at the first non-row line, so rows clean and conflict rendered as literal text inside the blockquote: the rule T4 must build appeared on the published site as a one-row table. It was caught by rendering the file, not by reading the diff — and the diff looked right, because the words were all correct and in a sensible order. A correction to a document is a change to that document and needs the same reading; the fact that it is a correction is why it gets less.

LOW — three assertions compared a payload against the enum that produced it. payload["outcome"] == Outcome.NO_BASELINE.value is green under any rename of both sides, which is exactly the rename that breaks a consumer's parser. Eight JSON strings lived in that gap. The replacement writes the keys and values out as literals, once, so a wire change is a visible diff in a test rather than a silent break in somebody's script. A test that derives its expectation from the code under test is a tautology wearing an assertion's clothes — the same shape as the "[budget]" in out assertion two passes earlier, which the diff body satisfied.

HIGH — three edit batches aborted partway and I reported all three as landed. The fixes were applied by scripts that walked a list of (file, old, new) and wrote each file as they went. When an anchor failed to match — because an earlier edit in the same run had already changed the text — the script raised, leaving the files before the failure written and the files after it untouched. Six edits were lost that way: the assertion pinning the oldest-archived-version fix, the up to date branch assertions, three of the four cannot compare: prefix assertions, a docs/KB-UPDATES.md contradiction and a docs/RETROSPECTIVES.md heading. Two of them were the headline fixes of the commit that claimed them, and the commit message, the retro fragment and docs/VERIFICATION.md all said they were pinned by a named test. The next pass found the test asserted nothing of the kind.

Three rules came out of it, and the third is the one that would have caught this on its own.

• An edit batch is all-or-nothing. Resolve every anchor first, write nothing until all of them match, and name the file and the anchor when one does not.
• Never let one batch depend on text an earlier edit in the same batch produced.
• A claim that a fix is pinned is a claim about a failing test. Revert the fix and watch the suite go red, or do not write the word "pinned". Four separate fixes across three passes were described as pinned by tests that were green without them — and each was found by the next reviewer doing the revert I had not.

MEDIUM — an assertion can be true because the current wording is kind. "pnk init appears unbroken in the output" passed with the code-span protection removed, because at this width the remedy's spans happen not to straddle the wrap column. It was a real assertion of a real property that simply could not fail on this input. The property belongs to the wrapper, so it is now tested against the wrapper, with a fixture built so that plain textwrap provably splits the span — and a second assertion that checks the fixture still has that property, because otherwise the test degrades into the one it replaced the moment someone edits the string.

Where the review loop stopped, and what each pass cost. Five adversarial passes, finding 30 → 22 → 13 → 6 → 1. The fifth found no defect in shipped behaviour — one documentation overstatement and seven coverage gaps in code that was correct — which is the signal the loop was waiting for. What the shape of that curve says is worth more than any single finding: the first pass over an increment is not the expensive one. Passes 2 and 3 each found that a previous pass's fixes were wrong or untested, and pass 4 found the tooling that had lost them. A review that stops at one pass stops before it has reviewed anything it changed.

The classes, in the order they cost the most:

Class	Instances	Why it survives a careful reading
A fix claimed as "pinned" whose test is green without it	4	The claim is about a failing test, and nobody runs the failure
A test that cannot fail on its own fixture	3	It asserts a real property of a real surface; only the input is wrong
A correction that broke what it corrected	2	The words are right, so the diff reads right
A whole-file question standing in for a per-hunk one	1	Both are true on the fixture that motivated the feature

T4 — pnk upgrade --apply (20260808 05:14)

MEDIUM — a plan can specify a test that cannot discriminate, and the name is what hides it. T4's plan named ::test_a_key_carried_only_by_a_conflicting_hunk_is_not_recommended as the load-bearing operand test: the one that separates parse(base + applied) from parse(ours). It cannot. Under the same plan's all-or-nothing rule a conflicting run refuses, writes nothing and prints no recommendation at all — so both operand choices produce an identical observable, and the test would have passed under the wrong implementation while reading as the proof that it was right. The case that actually discriminates is an already applied hunk: skipped, its key already in the file, and therefore not something this run introduced. Written as ::test_a_key_carried_only_by_a_skipped_hunk_is_not_recommended. The plan's own ground rules say every test name in it is a prediction and not binding; what this adds is that the scenario can be a prediction too, and a scenario nobody re-derives is a test that measures nothing under a name that says otherwise.

MEDIUM — the consent predicate had a fourth near-miss the plan did not enumerate. D-10 requirement 2 lists three cases where a spending-cap heading must not appear — an already-applied budget hunk, a conflicting run, and a bump touching no [budget] line — and specifies the predicate positionally: a clean hunk falls inside [budget]. The shipped template's only real [budget] drift (M3) rewrote three comment lines as well as two caps, so a hunk that moves only comments is inside [budget], applies cleanly, and moves no money. Positionally it would print a spending-cap heading with nothing under it — the fourth near-miss, and the one most likely to recur, because comment churn is what template drift mostly is. The predicate shipped as a clean hunk inside [budget] that changes at least one key, still structural and still with no key-name list, plus its own negative control. The plan's requirement 2 argues for exactly this and its three-case list is what fell short of its own reasoning.

LOW — an open correction that names a fork resolves to both, and saying which matters. The CRLF item required T4 to "either preserve each line's ending or refuse a manifest whose endings are not uniform, and say which". The answer is both, and the split is not a hedge: a uniform CRLF file is preserved, because that is an ordinary Windows manifest and rewriting it would be a change nobody asked for; a mixed file is refused, because it is already the product of two tools disagreeing and picking a winner silently rewrites lines the user never touched. The report path is unaffected either way, since reporting reads.

LOW — a refusal ordering decision that only shows up on the second run. Content refusals (a conflict) are checked before environmental ones (a stray .orig, a held lock). The other order is defensible until you follow it through: it tells a user to clear a backup file so that the retry can tell them their manifest conflicts anyway. Two round-trips to deliver one permanent fact.

The mutation record, because "pinned by test X" is a claim about a failing test. Eleven mutants run 20260808 05:14; each is the code broken on purpose and the tests that noticed, and two of them are here because the first attempt measured nothing:

Mutant	What went red	The control that had to stay green
remove the conflict refusal	…refuses_entirely_when_any_hunk_conflicts (+2)	—
write the .orig before the conflict check	the same test, on assert not _backup(root).exists() — checked by reading the assertion that fired, not the test name	it must not fail on byte-identity instead, or the ordering is not what is pinned
remove the re-parse	…unloadable_manifest_is_rolled_back, alone	—
drop the rebuild-naming line	…names_the_rebuild…, alone	…does_not_run_a_sync stayed green — that pair is the whole point of finding it
a key-level writer instead of the hunk applier	…the_comment_the_template_added_is_present_after_apply (+2)	—
suppress the spending-cap heading	both-values ×2, ordering	…budget_hunk_is_applied_like_any_other_hunk stayed green; if it goes red the heading tests are entangled with the write
print the heading unconditionally	all four negative controls (+4)	—
print the new value only	both-values ×2, on the old value's absence	—
write before any printing	ordering	both-values stayed green — which is exactly why ordering needs its own test
write requires_pinakes	…is_never_written (+2)	—
recommendation over every hunk, not the applied ones	…carried_only_by_a_skipped_hunk…, alone	the other three requires_pinakes tests pass under the wrong operands, which is what makes that one load-bearing

Two of the eleven were measured wrong on the first attempt, and both failures looked like success. The syntactically-invalid mutant made pytest print ERROR, not FAILED, and a harness counting FAILED lines read that as nothing noticed — the mutant survives: a false negative that would have sent someone to strengthen a test that was already correct. And the .orig ordering mutant turns twenty-two tests red, so the run "failing the right test" says nothing until you read which assertion fired. A mutation harness needs its own controls for the same reason the tests do.

MEDIUM — the first adversarial pass found five defects, and four of them were invisible from the tests because every one of them is a property of the write, not of the diff. T4's whole test surface was built from a plan concerned with placement, consent and refusal; none of those notice what happens to the file's inode.

Found	Why it matters
str.splitlines() breaks on \u2028, \u2029 and \x85; split("\n") does not	the report and the writer would disagree about which lines the file has, so a hunk the report called unique could match somewhere else, or nowhere. All three are non-ascii under TOML's own comment grammar, so a manifest can carry one and still load. Refused, because rejoining on \n would silently turn it into a newline in a file the user owns
os.replace onto a symlinked pinakes.toml destroys the link	the real manifest is left elsewhere holding the old text, and the KB now has a regular file. sidecar.write had already learned this and resolves the same way — the lesson was in the repository and the new writer did not inherit it, which is the argument for the write helper being shared rather than re-written
mkstemp creates 0600, so the rename narrowed the manifest's permissions	a user who had made it group-readable loses that, silently, on an unrelated command
a dotted key parsed to its first segment	budget.monthly_eur = 30.00 would be announced as a key called budget — in the consent line, which is the one output that must name what it is asking about
the rollback wrote non-atomically	it restores the file after a failed write; doing that non-atomically reintroduces the failure mode it exists to recover from

All four fixes were then re-mutated, and each turns exactly one test red. The fifth has no test and is not claimed to have one: a crash mid-restore cannot be staged in-process, and a row in docs/VERIFICATION.md says none rather than pointing at a test that would pass either way.

HIGH — the second pass found the one defect that would have printed a false spending-cap heading, and it was a correct function used one scope too wide. Hunk.section is read out of base by scanning backwards from the hunk's first changed line, so it is the table the hunk starts in. That is right for the placement listing, which labels a region. It is wrong for attributing a key, because a hunk that adds a whole new table carries keys belonging to a table base does not contain — and every one of them was credited to the preceding table. A template adding a table directly after [budget] would have had each of its keys announced under a spending cap changes, and one added after [chunking] would have invented an index rebuild.

Two things are worth keeping about how it was missed. It is not a bug in _section; it is a value being reused at a scope where its definition no longer holds, which no test of _section could ever catch. And every consent test passed, because each fixture's hunk stays inside one table — the negative controls guard against a heading printed when nothing moved, and this prints a heading naming the wrong thing, which is a different failure they were never shaped to see.

LOW — two refusals in splices were unreachable from every fixture, so they shipped unseen. One is unreachable by construction (_placement established uniqueness over the same text moments earlier) and the other needs a manifest repeating a region in a shape difflib still calls two hunks. The function is public now, for the reason fill and restamp are, and both guards have a test that reverting them turns red — including a control on the overlap guard, since a refusal that fires on everything passes the same assertion as one that fires correctly.

MEDIUM — and committed in this increment, in the file that documents the mistake. Pass 2 was committed with ./check.sh 2>&1 | tail -2 && git commit. In a pipeline the shell reports the last command's status, so tail succeeding made a failing ruff invisible and the commit landed over a red gate. check.sh's own header comment describes this exact shape as the reason the script exists — "uv run pyright | tail -1 && git commit reports the tail exit status, so a failing checker looks green" — and it was reproduced by a reader of that comment, one line below it, while running that script. A gate is only a gate when its status is the one the next command reads: write ./check.sh > log 2>&1; echo $?, never ./check.sh | tail.

LOW — pass 3, and both findings are the first two fixes not being finished. Writing through a symlink (pass 1) puts the backup beside the file it backs up, in another directory — and the output still printed its bare filename, which is true of every ordinary KB and misleading in the one case where finding the file takes work. A fix that changes where something lands has to be followed into whatever names it. And docs/CLI.md's refusal row still listed five causes when the code had eight, because passes 1 and 2 each added a refusal and neither went back to the table that enumerates them.

The same-manifest gap is stated rather than closed, deliberately. A version bump that leaves the manifest byte-identical produces no hunks, so --apply writes nothing — the [kb] template restamp included — and that KB keeps reporting drift with no way to record the new reference. T4 specifies --apply in terms of hunks and there are none, so writing anyway would be behaviour the plan does not describe. It is a test asserting today's behaviour, a row in docs/VERIFICATION.md, and an item proposed for plans/20260731_1202-open-corrections.md — not a decision taken quietly inside an increment.

MEDIUM — pass 4 found nothing in the code and three falsehoods in documents the increment never opened. docs/README.md still said "what is still a proposal is --apply"; docs/KB-UPDATES.md §1 said adoption "is still deferred" and its drift-axis table said "Adopting it is still absent: nothing writes the change into a user's manifest". All three were true when written and false the moment T4 compiled, and none is in any file T4 touched — which is exactly why the audit-the-neighbourhood convention exists and why a diff-scoped review cannot satisfy it. The search that found them was two greps over docs/ for the claim being falsified ("writes nothing", "--apply" near planned/deferred/absent), not a re-read of the change.

And the counterpart worth stating: docs/ROADMAP.md's ## 0.19.0 section still says "It writes nothing" and was left alone. That section is a record of what 0.19.0 shipped, and it was true of 0.19.0. A release-notes entry rewritten to match a later release stops being a record. The line that does need the sweep is the page's own "Where things stand right now", which is current-state and is the release's job, not the increment's.

Every documented exit code was then run rather than read — report 0, --apply 0, a conflicting report 0, a conflicting --apply 1, cannot compare under --apply 3 against the shipped notes, and an up-to-date --apply 0. Six rows, six commands, matching the table in docs/CLI.md.

Pass 5 found no new defect and is recorded anyway, because what it did is the part usually skipped. It checked the two factual claims the increment writes into three documents and one runtime message — that upgrade.apply is the only thing rewriting a pinakes.toml after pnk init (init.py:88 is the other write; nothing else in src/ writes it), and that init's .gitignore covers .pinakes/ and nothing else. Both hold. Then it mutated the seven T4 tests no earlier mutant had killed. Every T4 test is now killed by at least one mutant — which is a weaker claim than "every test asserts its property", and it is the strongest one a suite can actually make about itself.

T5 — The plan asked for a decision the file next to the code had already taken (20260808 06:28)

HIGH — D-4 was open in the plan and settled in manifest.py, three lines below the tuple it was about. The plan spent a four-option table, a recommendation and a paragraph of D-12 cross-reference on "what happens to vector_tier = "sqlite-vec" before the tier exists", and framed the choice as turning on a judgement about docs/MANIFEST.md: is its row a promise or a disclosure? Meanwhile GRAPH_CHANNELS' docstring, at manifest.py:52 against VECTOR_TIERS at :51, already stated the answer as a rule — "a manifest that can ask for a mode the code does not implement is a manifest whose setting silently does nothing, and table.choice refusing the name is how a user finds that out at load time" — and applied it to "ppr", a value in the very next row of the same documentation table. The plan cites neither.

What that changes about reading a plan. A plan's decision table is a list of questions its author could not answer from the plan, which is not the same as questions the repository has not answered. Two of the plan's four open recommendations here were about consistency with existing behaviour, and the cheapest evidence for both was adjacent to the line being changed. The habit worth keeping: before weighing a plan's options, look at what the sibling key does — the file is often more decided than the document about it.

MEDIUM — the plan's own two halves disagreed about what T5 could deliver. It asked for resolve_tier to be called by both sync and search, "so meta's claim and the code path cannot disagree", and then admitted two paragraphs later, correctly, that "with exactly one real tier there is nothing else to discriminate". Both cannot hold: if there is nothing to discriminate, search has no dispatch to make, and a tier parameter threaded into _vector that can hold one value behind an unreachable branch buys a shape that looks like a shared decision while being decoration. Built the resolver with one caller and said so in its docstring. This is the eighth of the template-release plan's own measurements or specs to be wrong, and the second found by building rather than by reading it.

A smaller one, on honesty in a one-tier world. The first draft of resolve_tier was return "numpy" — correct, and a function that ignores its only argument. It became return "numpy" if tier == "auto" else tier, which reads the manifest and honours an explicit tier. Today both arms return the same string, so no test can tell them apart; what the second form buys is that the increment restoring "sqlite-vec" gets it honoured by that line and owes only auto's side of the choice. Worth the branch; not worth pretending a test covers it.

T7 — pnk templates, and a template declares its files (20260808 09:54)

HIGH — a test with two cases in one body only holds the case that runs first. test_a_template_file_entry_that_escapes_the_target_is_refused covered both write-side escapes: ../../evil.md and a symlinked directory in the target. It was green, and the mutation pass showed it was green for the wrong reason. Removing the destination containment check turned it red on its first assertion — ../../evil.md escapes the template as well as the target, so the other layer caught it with a different message — and pytest stopped there. The symlink case, the one only the removed check can catch, never executed under the mutation it existed to detect. Split into two tests; both now go red, and the source-side test stays green. The general shape: when two guards can both catch one input, a test that exercises that input proves nothing about either of them, and a shared test body hides which one fired. This is the same family as T4's …by_a_conflicting_hunk_… — a test that cannot discriminate — but arrived from the opposite direction: not a case that looks identical under both implementations, but a case caught twice.

HIGH — a plan's rule sentence and its test list can specify different things, and the test list was right. T7 states that each files entry "is validated to land inside the target KB", then lists a test refusing "a symlinked directory inside the template tree". Those are two layers: a symlink in the template resolves outside the template while its destination stays inside the target, so the target rule cannot catch it and the named test could never have passed. Both are built. The write side stops an entry writing into a directory the user never pointed Pinakes at; the read side stops it copying something the template does not own into a KB that is then committed and published — which for a public repo is the more expensive direction. Read a plan's test list as part of its specification, not as an illustration of it. Where the two disagree, the test list has been forced to be concrete and the sentence has not.

MEDIUM — a coverage row that cannot fail is worse than no row. T7's free-path deliverable asked for main(["templates"]) and a module added to test_paid_path.py's surface list. But pnk init already runs in the same fixture, so pinakes.template is in the import graph regardless, and run_templates lives in pinakes.cli — no row would have failed if the new call were deleted. The call was added and the row was not, with the reason written at the call site so the omission is not "fixed" later. Eight of this plan's measurements were already wrong; this is the ninth and the tenth, and both were found by trying to satisfy its own test list rather than by reading its prose.

MEDIUM — a new listing command inherits every failure mode of the thing it lists, multiplied. template.describe raising a bare OSError on a damaged install is an open correction from T3, and it reaches init, doctor and upgrade. Before pnk templates, that cost you the run that named the damaged template. A listing iterates all of them, so one bad directory produced a traceback and reported nothing about the templates that read perfectly. The general fix stays open — it is not this increment's to settle — but the blast radius this command introduced is contained here: the failure is caught per template, shown as a row, and the exit code is non-zero. Adding a "list everything" surface converts a per-item defect into a whole-command defect, and that conversion is the new increment's to own even when the defect is not.

LOW — the runtime agreed with a bug the linter caught. A missing Callable import passed the whole test run and failed ruff. The worktree's uv environment is Python 3.14, where PEP 649 defers annotation evaluation, so an annotation naming an undefined type never evaluates. On the 3.13 this project requires, it is a NameError at import. A green test run on a newer interpreter is not evidence about the one in requires-python — which is what the linter is for, and why ./check.sh runs it before pytest rather than after.

HIGH — a new key can defeat an existing invariant by living in the file that invariant excludes. tools/template_drift_gate.py hashes every file in a template except template.toml, and that exclusion is load-bearing: hashing the file that declares the version would make every bump change the hash by construction, so "a version bumped with no content change" could never be detected. The gate's own limit (b) recorded the cost as the description could be edited unnoticed — true, and cheap, while a description was the only editable thing in there. T7 added files = [...] to that same file, and files decides which files a KB is stamped with. So the key that determines a template's stamped content landed in the one file excluded from the check that exists to stop stamped content changing without a version bump. Closed by folding only the list into the digest: name, version and description stay out, so leg (ii) can still fail. An absent key contributes nothing, which is what makes the change backward-compatible — the shipped notes hash is byte-identical before and after, so no published _versions.toml row needed migrating. When an increment adds a key, ask which existing invariants were scoped by the assumption that the file holding it did not matter. Nothing in T7's plan connected the two; the link is only visible from the gate's side.

MEDIUM — git checkout -- <file> restores to the last commit, not to the state before the mutation. The mutation pass ran fine against template.py, whose changes were already committed, then silently destroyed the uncommitted drift-gate work when the same restore step ran against it. The mutation's result was valid — both target tests went red, the negative control stayed green — but the code came back as main's, and the next full run showed two failures that looked like the fix was wrong when it was simply gone. Commit before mutating, always: the restore step is only as precise as the thing it restores to, and a mutation harness that quietly reverts real work reports a false failure rather than a false pass, which is the better direction but still costs a debugging cycle.

Open corrections 1 and 3 — a guard can route a failure into a wrong message (20260810 01:29)

HIGH — turning a traceback into a PinakesError moved the failure into a handler that already existed and said the opposite thing. Item 3's whole content is stop these five reads escaping as tracebacks. Doing it made describe raise TemplateError — and doctor.py and upgrade.py both already wrapped their describe call in except PinakesError, answering it with "is not installed here" and a remedy about installing the template. That arm was correct while the only thing reaching it was a template genuinely absent; a damaged install had been going straight past it as a traceback. So the fix silently recruited a handler written for the opposite case, and a user whose template.toml was unreadable would have been told to install a template sitting right there.

A traceback is loud and a wrong sentence is quiet, so this is a downgrade that reads as an upgrade. Both surfaces reported WARN, both exited 3, every test stayed green — the increment's own tests included, because they assert the new messages and the pre-existing ones assert the absent-template case that still works. Nothing was red. It was found by asking who else catches what this function now raises, which is a different question from does my change work.

The correction is TemplateNotInstalledError, a subclass so that every existing except TemplateError keeps working, with _unknown as its only raiser and the two callers splitting the arms. Both surfaces get their own test, because the wording is a fact with one home but the routing is a decision each caller takes for itself — one test would leave the other free to merge the cases back.

Generalises past exceptions: widening a type is an interface change on every except upstream. The grep that finds it is not for the function being changed but for the type it starts raising, and the question is whether each catcher's answer is still true for the new cause. PinakesError is caught in 30-odd places here precisely because it is the type that means print this and stop, so anything newly raised as one inherits whatever those handlers already say.

Second, smaller, and mechanical: git checkout -- <file> during a mutation pass deletes the fix being verified. The pass ran before the commit, so restoring after each mutation restored to main — three mutations in, both source files were back to their unfixed state and the tests were "failing correctly" against no fix at all. The evidence was still true and the work had to be typed again. Commit, then mutate: BUILDING.md orders the steps that way, and this is the reason rather than a convention.

A release sweep is table-shaped, and a narrative is not a row — 20260811 12:18 UTC

What happened. With every plan built out and the open-corrections list empty, the first check of the next session was what does the repo say about itself. docs/ROADMAP.md said 0.21.0. Its release table carried a 0.22.0 row, its per-release section carried the full 0.22.0 write-up, and its open-corrections section said none live, all four shipped in 0.22.0 — but the two prose blocks that state the project's position, ## Where things stand right now (stamped 20260808 06:41) and ## The template release, were three releases behind. One of them still said "T4 and T7 are still to come" about increments that had shipped on 20260808 and 20260808.

Why it survived five sweeps. The landing checklist asks which file to edit, and the file was edited — every time. What a sweep naturally finds is the row it is adding: a table has one line per release, so the release being cut points straight at its own row. A paragraph that summarises all releases has no row to add, so nothing in the act of cutting a release makes it obvious. The per-release sections and the tables were correct at every commit; only the summaries were wrong — which is the worst arrangement, because a reader checking one against the other finds agreement in five places out of six.

The second instance is sharper. docs/README.md's plan-routing table — the table whose whole job is to tell a session which plan is live — had no row at all for plans/20260811_0720-decisions-gates-and-corrections.md, the plan CLAUDE.md names as the live build order and the authority for eight decisions. The plan was written, its six increments were built and landed, and the index of plans never learned it existed. A missing row is invisible to every check that reads rows; only asking "is everything that exists listed here?" finds it, and that question is not part of landing an increment.

What generalises. "Update the doc" and "update the doc's summary of itself" are different actions, and only the first is prompted by the work. A document that both enumerates and summarises will drift at the summary, in the direction of the last release that bothered to rewrite prose. Two checks are worth adding to a landing:

• Grep the docs for the previous version number after a release, not just for the fields the checklist names. grep -rn "0\.21\.0" docs/ would have found both ROADMAP blocks in seconds.
• Ask what is missing, not only what is wrong. The routing table's defect had no wrong text to find. Reading ls plans/ against the table is a ten-second check that no diff review performs, because nothing in a diff is absent.

And the check found more than it was looking for. Auditing the neighbourhood surfaced that the 20260807 audit's 40 documentation corrections have never been worked — the file has one commit, the one that created it — and that the same audit deferred a full review of docs/ROADMAP.md until after T2, which shipped in 0.18.0. Neither is visible from any release's own sweep either.

A row can be complete, correct, and in the wrong place (20260811 13:27)

HIGH — five release rows were out of order in three sequences, and nothing could see it. docs/ROADMAP.md's release table and its per-release sections both read 0.20.0, 0.22.0, 0.22.1, 0.21.1, 0.21.0, 0.20.1; docs/STATUS.md's roadmap table put 0.15.1 after 0.16.0 and 0.20.1 after 0.22.1. Every misplacement is wrong on both readings — SemVer and release time — so no ordering convention made them right. CHANGELOG.md was checked and is clean, headings and link definitions both.

Ordering is a property of the sequence, not of any row in it. That is why nothing caught it: the tables are complete, every anchor link resolves, mkdocs build --strict is green, and a reader checking any single row finds it correct. Every check this project owns reads rows.

How it happened, recovered from the six release commits. 0.20.1 was appended correctly (2da0e07). 0.21.0 (96b3b35) then inserted its section one position too early — after 0.20.0's rather than after 0.20.1's — and the next three sweeps (c83e877, df832fe, 93c20ab) each used that same slot.

Commit	Release	Section tail after it
d429a2c	0.20.0	… 0.19.0 0.20.0
2da0e07	0.20.1	… 0.20.0 0.20.1 ✅
96b3b35	0.21.0	… 0.20.0 0.21.0 0.20.1 ← the first error
c83e877	0.21.1	… 0.20.0 0.21.1 0.21.0 0.20.1
df832fe	0.22.0	… 0.20.0 0.22.0 0.21.1 0.21.0 0.20.1
93c20ab	0.22.1	… 0.20.0 0.22.1 0.22.0 0.21.1 0.21.0 0.20.1

The tail was locally self-consistent at every step, which is the whole lesson. After the first error it read strictly newest-first, so each following sweep saw a coherent pattern around its own edit and matched it — reproducing the error is what reading the neighbourhood carefully produced here. Only the join between the ascending head and the descending tail was wrong, and that join is a single line no sweep's diff ever touched. Read the sequence, not the neighbourhood.

It had already been found and left. The 20260807 documentation audit's finding docs/STATUS.md:303 names the 0.15.1 row exactly, with PyPI upload times as evidence. It sat unworked for four days, and in that window three more sweeps added three more misplaced rows. A verified finding nobody schedules is worth what an unverified one is.

The fix is a gate, not a checklist item — tools/release_order_gate.py, in check.sh and CI, on the threshold this project already applies (status_header_gate.py: a checklist missed it four times). Two design points are the reason it will still work in a year:

• Direction is declared per sequence, never inferred. Inferring it from whichever way most adjacent pairs agree would let a badly scrambled file elect its own answer and pass.
• A sequence shorter than a floor is a failure, not a pass. An empty sequence is sorted by definition, so a pattern that silently stops matching — a reformatted table, a changed heading style — is exactly how this class of gate dies quietly. A test asserts the real documents clear that floor, so a reformat goes red in the commit that does it.

Watched failing before it was trusted: run against the pre-fix tree it names all nine misorderings and exits 1; CI scrambles a copy and asserts both the exit and the stated reason.

E1 — pnk ask, and the output nobody could check (20260811 15:17)

MEDIUM — a doc that pastes command output goes stale silently, and nothing in this repo can see it. docs/GUIDE.md § Searching showed a pnk search result whose last two lines were the escalation notice — Paid synthesis (pnk ask --deep) is planned for the deep release. E1 changed that sentence in cli.py, and the pasted block kept printing the old one. It would have shipped: ./check.sh is green on it, mkdocs build --strict is green on it, and every link in the file resolves. The conventions already say verify a doc by running the commands it shows (docs/README.md) — this is the first recorded instance of the rule catching something, and it caught it because the increment happened to edit the very sentence, not because anything checked.

What makes this class different from a stale sentence. Prose describing behaviour is written by someone who read the code; pasted output is a measurement, and a measurement has no author to notice it aged. There are three such blocks in GUIDE.md and two in CLI.md. Nothing pairs any of them with the code that produces it.

Not turned into a gate here, and the reason is worth recording. A gate would have to run each documented command against a fixture KB and diff the output — which is a golden-output test on formatting, the most brittle kind, and it would fail on every deliberate wording change with no way to tell those from regressions. What is cheap and was done instead: both new blocks in this increment were pasted from a real run (tests/demo-kb for the calibrated one, tests/partner-kb for the uncalibrated one), so at least the version now on main was true once, with the KB that produced it named beside it.

LOW — a test that can only fail in a later increment must say so in its own docstring. test_a_confident_kb_gets_cited_evidence_and_the_price_of_one_call asserts no ledger.jsonl exists after pnk ask. Nothing in this build can spend, so the assertion cannot fail today; it is a tripwire for E4, which adds a paid loop to this same command through the shared _retrieve. Left unexplained it reads as dead weight and the next reader deletes it — one increment before the one it was laid for.

LOW — the vocabulary was spelled twice in one file. run_search compared confidence against the string literals ("low", "unknown") while _escalation, forty lines below, compared against search.py's LOW/HIGH/MEDIUM constants. Both correct today. The renderer also asked if escalation.branch != "none" to decide whether to name the deep release, putting the branch vocabulary in a third place — and the decision that actually mattered (never offer to answer a question nothing matched) lived in none of them. Both fixed: the constants throughout, and the notice carried on the value rather than re-derived at the print site.

E2 — The deep release's round estimator (20260811 16:15)

HIGH — the worst case left out the one input the user controls: the question itself. The plan's formula prices carried memory, final_k passages and a prompt constant, and the first draft implemented exactly that, with the question folded into PROMPT_TOKENS. But a question arrives as an argv string — pnk ask "<question>" — with no length limit anywhere in the CLI, and it is carried into every call of a run, so a 50,000-token question would have been reserved for at 1,500 tokens. Fixed by pricing it separately (QUESTION_TOKENS) against a stated ceiling (QUESTION_CHAR_CEILING) that E4 must enforce, since this module refuses nothing but a stale price table and an oversized request. Lesson: when a worked example pins an estimate to the cent, it also silently fixes the size of every input the example did not vary. The question was the one term with no bound anywhere in the system, and the arithmetic looked right because the example never made it big. Ask what is in a call's input that the formula does not name — not whether the formula was implemented correctly.

HIGH — the second review pass found a ceiling below its own measurement, in a module written to refuse exactly that. PASSAGE_ENVELOPE_TOKENS carried the comment "the longest path — heading_path pair is under 120 characters", asserted without running anything, and 100 tokens was sized from it. Running it — by extending tools/measure_passage_tokens.py to report the envelope as well as the chunk — returned 220 characters, about 110 vendor tokens at the pessimistic conversion the module's other constants use. The constant was under-reserving against a number nobody had taken; it is now 250, with the measurement and its date beside it. Lesson: the review question that found it was "which of these numbers did I measure, and which did I merely write?" — asked of a file whose every other constant carries a command. A measured neighbour makes an asserted one look measured too, and prose like "measured from the corpora above" is how the two become indistinguishable.

MEDIUM — a total divided by its call count and multiplied back was not the total. Decimal division is exact to 28 significant digits and no further, so per_call_eur * calls landed one digit above total_eur at the shipped defaults — meaning, in the other direction, a per-call reservation could sum to less than the operation it belongs to. Caught by a test asserting the two were equal. Fixed by deriving the total from the per-call price rather than the per-call price from the total, which removes the class rather than the instance: no set of constants can reintroduce it. Lesson: Decimal fixes base-10 representation, not associativity — money code should multiply a unit price up, never divide a total down, and the direction of the last digit is chosen by nothing.

MEDIUM — a "measured" docstring falsified by the fix in its own increment. The docstring above cited the two 28-digit values it was measured at; the fix changed the arithmetic that produced them, so re-running the same comparison now round-trips exactly and the cited numbers describe code that no longer exists. Rewritten to say what was observed and when it stopped being reproducible, rather than left reading as a claim about the shipped module. Lesson: the measurement most likely to go stale in a docstring is the one that motivated the change in the same diff — grep your own increment's fixed defects for the numbers that justified them.

LOW — a purity test written as a denylist, in a repository that had already learned not to. The first version asserted deep/estimate.py imports no anthropic, sqlite3, httpx, requests — a list that is never finished, and whose next omission is always the one that matters. check.sh's NUL scan records the same lesson about file suffixes ("a denylist of binary formats is never finished"), one file away. Rewritten as an allowlist of what the module may import, so a new import has to be argued for in the test. Lesson: for "this module stays pure", enumerate what is allowed. The repository's existing gates are worth reading for the shape of the assertion, not only for the rule.

A finding handed to E4 rather than solved here. At the shipped defaults (final_k = 8, [chunking] max_tokens = 510, [budget] per_operation_eur = 0.30) the cheap branch prices at EUR 0.2627 and fits inside the cap; a five-round loop prices at EUR 2.81 and is 9.4x it. So on a stock KB, pnk ask --deep would answer a confident question and refuse an uncalibrated one at round 0 — which is precisely the combination D-22 option E was chosen to avoid, arrived at through the caps instead of through the signal. E2 declines to fix it by lowering a ceiling: that is the trade PAGE_TOKEN_CEILING refused, and the numbers are conservative by design until E6 measures them. It is pinned as tests/test_deep_estimate.py::test_the_shipped_defaults_leave_the_loop_outside_the_default_operation_cap, reading both sides out of the manifest defaults so it tracks them rather than restating them.

E3 — The deep client, and four rules that were about to be copied (20260811 17:12)

HIGH — a second paid entry point is where a "one home" rule quietly becomes two. E3's section of the plan says the key is PINAKES_ANTHROPIC_API_KEY and never an ANTHROPIC_API_KEY fallback, "enforced the same way here" — a sentence that reads equally as reuse it and as write it again. Writing it again would have been the smaller diff, and it would have produced two copies of four rules that each fail silently when they drift: the key's name, max_retries=0, whether a failed call billed, and how a reconciliation is computed. CLAUDE.md calls an ANTHROPIC_API_KEY fallback "the same defect, one layer apart"; a second copy of the rule forbidding it is the same defect one file apart. So src/pinakes/paid.py exists, holding all four with no client in it, and extract/claude.py now imports what it used to declare. What made the move safe rather than brave was that its 1,606-line suite is unchanged: the only test that failed was the allowlist count, which is the gate refusing the widening until it was declared.

The module is deliberately not allowlisted, and that is the interesting part: classify is handed the caller's already-imported SDK module rather than importing one, so gate 2 scans it like any other file and would refuse an import anthropic added to it. The allowlist stays two entries — extract/claude.py and deep/client.py — which is the complete list DESIGN §1 describes.

HIGH — the shared classifier had no direct test, and it decides void versus unknown outcome. Moving it surfaced that every branch of _classify had only ever been reached through a fixture that raised an already classified error: the replayer constructs TransportError itself, so the mapping from an SDK exception to a billability had never been executed by a test in either suite. The relationship it turns on is the one stubs/anthropic.pyi warns is easy to get wrong from memory — APITimeoutError is a subclass of APIConnectionError, so checking the parent first classifies every timeout as not-billed and voids a reservation for a call that may have been charged. It now has five tests against a fake SDK whose hierarchy is held against the stub's, and against the real package where it happens to be installed.

MEDIUM — the renderer was spending 90% of a ceiling on an identifier the model never emits. The first draft of render_passages mirrored cli.py's printed block exactly, citation line included — which carries the path and the heading a second time. E2 measured PASSAGE_ENVELOPE_TOKENS against one copy (220 characters, the widest real passage in the corpora) and set it at 250 with 2.3x headroom; two copies came to 452 characters, 226 of the 250 reserved tokens, leaving nothing for a KB whose headings run deeper than the two corpora measured. Found by writing the test that pins the arithmetic, not by reading the code. The fix was to drop the citation line rather than raise the constant — E2's own rule, and PAGE_TOKEN_CEILING's: a ceiling that close to its own measurement is not a ceiling. The test now asserts the envelope stays under half the constant, because "inside it" was true of the draft too.

MEDIUM — three smaller findings from the same pass, each a disagreement between two halves of one bound. The subproblem cap was computed twice, once for the request's schema and once for the parser that re-checks the response — the shape where the second check silently becomes about a different limit than the first, at which point one of them is decoration (now subproblem_cap, one function, driven through both users by one test). answer_schema wrote max(passages, 1), so a call over zero passages would have declared citation [1] legal to the API and had every index refused by the parser — prose with nothing behind it, paid for; both halves now refuse (NoEvidenceError), which is E2's reason for not pricing the none branch, arriving one layer down. And billed_call voided on any exception the transport did not classify — but a defect is not proof the call never billed, and proof is what INVARIANTS requires before a void, so an unclassified failure is now left unresolved instead.

LOW — a test that asserted its own base class. The first version of "every error carries a remedy" checked issubclass(..., PinakesError), which is true by construction and would have passed for a subclass that omitted the keyword entirely. It reads the super().__init__ call out of the parsed source now. Worth recording because it is the third time in this repo a safety check has been written in the shape that cannot fail; the tell each time was that it passed the moment it was written.

What E4 inherits from this increment, beyond what the plan already names: the three bounds E2 prices against are enforced in the client — the question ceiling, the carried-memory ceiling (CARRIED_MEMORY_CHAR_CEILING, added beside the token constant it derives from), and final_k passages per answering call — and each refuses rather than trimming. E4 should refuse earlier and more kindly, with a sentence about the command; what it must not do is assume the client will quietly cope. Citations arrive as passage numbers, so E4 owns the mapping back to documents, and E7's rule that a suggestion's endpoints must be documents this run retrieved is already a property of the wire format rather than a check to add.

Second review pass — the knob nobody would have passed, and a version nothing checked. Two more, both found by reading the module as a stranger would rather than as its author:

• max_tokens was a parameter on both request builders, defaulting to the estimator's constant. Harmless as written and a hole by construction: output bills at five times the input rate and is two thirds of a round's whole price, so a caller passing a larger ceiling is billed for output no reservation covered — the same hole the question, memory and passage bounds had just been closed for, sitting one line above them. It is gone; _request always sends MAX_TOKENS. Unlike the extractor, this client never re-asks a truncation at a raised ceiling, so nothing wanted it.
• PROMPT_VERSION and SCHEMA_VERSION were decoration. They existed for E5's transcript and E6's measurement, and nothing read them — which is exactly T1's failure with a template version: pnk doctor compared a version against the installed one for ten releases while the files that version denoted changed underneath it, and every KB recorded a reference that matched and meant something different. So the prompts and both schemas are now hashed and pinned to their version, and a reworded prompt fails the suite until the digest and the version move with it. E6's numbers are about a specific wording; without this, the record of which wording would be a guess.

The pass also promoted DeepCallFailedError's three kinds to constants, for the reason E1 gives for carrying its escalation value rather than re-deriving it: E4 has to tell a refusal from a truncation in order to label a run, and a caller matching a string literal puts the vocabulary in two places.

And a note on the mutation testing itself. The first attempt at mutating max_tokens back into _request reported "54 passed" and I nearly recorded it as a test gap — the edit had silently matched nothing, because ruff format had collapsed the signature onto one line after the change. A str.replace that matches nothing returns the string unchanged and tells no one, which is the defect conftest._rewrite exists to prevent, reappearing in a throwaway script. Every mutation in this increment now asserts its target matched exactly once before running the suite; the retried mutation failed the right test immediately.

Third review pass — two findings, both about what a reader would look for and not find. The shared-rule tests had landed inside tests/test_deep_client.py, so src/pinakes/paid.py — a module both paid entry points depend on — had no test file bearing its name, and the nearest match, tests/test_paid_path.py, is about something else entirely (which modules may import a client, not what they must then do). Split into tests/test_paid.py, with each file's docstring saying which question it answers, and VERIFICATION gains a section of its own for the rules rather than filing them under the client that happens to use them.

The second: AnthropicTransport.max_retries was a property nothing asserted. build_client_kwargs() == {"max_retries": 0} is tested, and says nothing about this transport passing it — the same gap in a smaller frame as the classifier's. Constructing the client is offline, so the test is not paid-marked; it skips with a reason where [claude] is absent.

Fourth review pass — the sentence this increment falsified was in the file it edited. extract/claude.py's first line read "the only module in src/ permitted to import anthropic", and AnthropicTransport's docstring said "the only place anthropic is constructed". Both were true when written, both became false in the same commit that added the second entry to the allowlist, and neither is reachable by grepping for what changed — the words deep, client and allowlist do not appear in either sentence. Found by grepping for the claim shape ("only module", "the only place") rather than for the diff, which is the same move that turned up docs/ROADMAP.md:139 at E2's handover: a stale pointer has no wrong text to search for, only a claim that is no longer true.

A Ctrl-C voided a call that may have billed — in both paid clients, for as long as either existed

Found by working an exit criterion nobody had tested. E4's plan asks that "interrupting mid-loop leaves a reservation pnk budget reports as unknown outcome, never a lost record". Nothing asserted it, so it was probed with a transport that raises KeyboardInterrupt, and the ledger came back voided — EUR 0 recorded for a request that had already been sent.

Why it was invisible. Every deliberate branch was right. billed_call classifies a timeout as billable-unknown, voids a 429 that never billed, and — in the deep client — catches Exception for anything unclassified and leaves it unresolved. A KeyboardInterrupt is not an Exception, so it fell past all of it into ledger.paid_call's finally, whose job is to close an unfinished call and whose default is to void. Every layer behaved as written. The extractor was worse and had been since I7b: no catch-all at all, so an ordinary defect voided too.

Three things worth keeping:

• The likely interrupt is the one nobody models. A paid run is slow, visible and cancellable; Ctrl-C during one is the normal way it ends when a user changes their mind. It was the only exit path with no test.
• A safe default one layer down is not a safe default. close_unfinished voids because most unclosed calls never billed. That is correct there and wrong here, and the caller is the only place that knows which. except BaseException is what says so.
• The sibling had it too, and fixing one would have been the defect surviving. One invariant, two call sites, two identical clauses — so both moved in the same change, and both tests raise a BaseException rather than a RuntimeError, because a narrower one passes against broken code.

The same GUIDE block quoted a retired sentence twice — so the second time it became a gate

E1 rewrote pnk search's escalation notice and left docs/GUIDE.md displaying the sentence it had just replaced. Its retrospective recorded that nothing in the repo could have caught it. E4 rewrote the same sentence and left the same block stale again, caught only because someone grepped for the old wording before shipping.

Why every existing check is blind to it. The prose is well-formed. Every link resolves. mkdocs build --strict is green. tests/test_verification.py checks that named tests exist, not that quoted output is current. A fenced block showing a previous build's output is correct Markdown describing a program that no longer exists, and nothing in this repo reads it as anything else.

The checkable half is the negative one. Diffing a fenced block against real output would need the command, its models and a corpus. "Every printed constant must appear in the docs" is simply false — most should not. But a sentence this build can no longer print must appear nowhere, and that is a grep. tests/test_docs_quote_the_shipped_sentences.py holds the retirement list, one row per retired sentence with what replaced it; retiring a sentence is a deliberate act, so adding the row is part of it.

Two things that make the gate honest rather than decorative. It searches src/ as well as docs/, because a retired sentence surviving in a docstring is the same defect one layer in — and this project's docstrings are where its reasoning lives. And it was run against the pre-E4 tree, where all four rows fail: a gate never observed failing is a gate nobody has tested.

E5 — The run transcript (20260812 05:11)

MEDIUM — a mutation restored a file and silently took a real edit with it. The mutation pass works by editing a source file, running the suite, then git checkout <file>. Between the first commit and mutation 5 there was exactly one source edit — a comment in cli.py correcting a claim about clear_cache_paid — and git checkout src/pinakes/cli.py reverted it along with the mutation. Nothing failed: the tests pass either way, ./check.sh is green, and the reverted text was a comment, so no gate could see it. It was found by reading the increment's own diff in the adversarial pass and noticing the comment said what the earlier draft said.

The lesson is narrow and mechanical: git checkout restores to the last commit, not to the state before the mutation. Either commit before mutating, or restore with git stash/a copy. This increment did the former for the code and got caught by the one edit that landed in between.

And it happened a second time, two passes later — git checkout tests/free_path_run.py, run to undo a deliberate deletion that was proving a gate row discriminates, reverted the uncommitted addition of that same call. Knowing the trap was not enough to avoid it; the only thing that caught it the second time was checking for it immediately afterwards, because it was now expected. Write the check, not the resolution: after any git checkout <file> in a mutation loop, grep for the thing that was supposed to survive.

MEDIUM — a test that asserted the wrong half of its own claim. "The temp file is .tmp, not .json, so a killed write leaves nothing the readers count" was tested by planting a .tmp-abcdef.tmp file and checking the glob ignored it. That proves the glob ignores .tmp files. It says nothing about what the writer names its temporaries, and it kept passing when the suffix was mutated to .json. It now spies on os.replace, whose source argument is the file a kill one instruction earlier would have left behind, and asserts on that name.

Generalisable: when a test plants the input it then checks, ask which half of the claim it actually reaches. The planted value came from the test's understanding of the code rather than from the code, so the two could disagree without the test noticing — which is the whole failure mode the mutation pass exists to find, and it found it.

LOW — the confirm-then-re-call path had never been tested, on either store. sys.stdin.isatty() is False under pytest, so every --clear-cache test since I4 took the --yes route straight past the prompt, the y/n branch and the second sync() call. Three tests now walk it. Worth recording because the gap was invisible in the ordinary way: the flag had coverage, the interactive flag did not, and no coverage number distinguishes them.

A decision worth writing down: --clear-cache=transcripts names a store, and the two values before it name authorisations. --clear-cache and --clear-cache=paid both clear the whole extraction cache and differ only in what they permit — a documented distinction, with a comment in cli.py explaining why the bare form is not called =free. Layering transcripts onto that axis would have meant --clear-cache=transcripts also emptying the extraction cache, which destroys more than the flag names. Mixing the two axes in one flag is a real cost, and it was paid deliberately rather than by accident: D-26 asked for a --clear-cache target, and a target is what it is.

What E6 inherits. The transcript is the record the measurement run reports out of: it carries call_ids, the estimate and the reconciled spend per run, so the over-reservation factor E6 must publish can be computed from the files a measured run leaves behind rather than from a spreadsheet kept beside it. transcript.call_ids() plus sync.ledger_spend() is the join, already written for the confirmation prompt.

E6 — A seam the tests never crossed (20260812 06:41)

CRITICAL — pnk ask --deep has never worked against the live API, and 0.22.0 through 0.25.0 all ship it. The first real --deep call E6 made returned a 400 before it billed anything:

output_config.format.schema: For 'integer' type, properties maximum, minimum are not supported

deep/client.py: answer_schema emits {"type": "integer", "minimum": 1, "maximum": passages} on every answer call, and structured outputs does not accept numerical constraints. A second probe found subproblems_schema's maxItems is refused the same way, so the decompose call was broken too — fixing the answer call alone would not have produced a working loop.

The lesson is about the seam, not the schema. E3 introduced Transport so the whole loop could be driven from recorded fixtures with anthropic absent, and E4 and E5 were both tested that way, green throughout. That is a good seam and it bought real things. What it also did was guarantee that no test in the suite has ever sent a schema to the API — the one field the API validates and the fixtures cannot. A fixture asserts what we believe a response looks like; it cannot assert that the request was acceptable. Every layer above the seam was correct, which is exactly why nothing went red: the defect lived in the only inch of the path the seam removes.

The generalisation: a seam introduced for testability defines a region the tests cannot reach, and that region needs its own gate. For this one the gate is cheap and fixture-free — assert the two schema builders emit no keyword on structured outputs' documented unsupported list. It would have failed at E4, on a branch that was green.

Three things behaved exactly as designed and are worth recording, because a bad failure is where you find out. The accountant reserved, refused and voided: the failed call billed €0.0000, so a run of 400s cannot consume a budget. The error surfaced the API's own sentence rather than a paraphrase, which is what made the cause obvious in one read. And the blast radius was genuinely bounded to the one path — pnk search, pnk ask without --deep, and the paid extractor were all untouched, because none of them shares this client.

Running the instrument found two bugs in the instrument. tools/deep_reservation.py count filtered its token-count payload to model/system/messages, dropping output_config — so it measured PROMPT_TOKENS with the schema excluded, a fraction of the real figure. And QUESTION_TOKENS differenced an arbitrary 200-word probe, so it returned ~200 by construction: it measured the probe, not the question ceiling. Both would have published a wrong constant with no symptom. A measurement probe is code and earns the same adversarial pass as anything else — two of five constants were measured wrong on the first run, and reading the script is what found neither; running it is.

E6 — The measurement run (20260821 07:53)

HIGH — the instrument that publishes the numbers had no tests, and its --json had never run. vars() on a slots=True dataclass raises TypeError, so tools/deep_reservation.py count --json and report --json both failed on their first row, in every version that ever existed. It survived because every measurement session read the printed table instead. The general shape is worth keeping: a tool written to be run by hand has exactly one exercised path — the one the author happened to type — and the flag nobody typed is not a lesser-tested path, it is an untested one. The fix is cheap; noticing was the whole cost, and what noticed was sitting down to write the tests rather than any run of the tool.

HIGH — the published factor was not reproducible, and nothing said so. The handover recorded 19.0× on synthesis and 16.5× on the loop. Those came from measurement KBs under /tmp, which the operating system reaped after nine days, taking every transcript and every ledger row with them. The money was spent and the numbers were real, but by the time anyone re-ran report the evidence for them was gone — and report cheerfully printed a different factor over the surviving records without any indication that it was answering a smaller question. A measurement whose substrate lives in /tmp has a shelf life, and the number outlived it. What made this recoverable rather than merely lost is that the runbook's rebuild step is free: the KBs come from tests/demo-kb and tests/partner-kb, so the run could simply be done again.

HIGH — the runbook's own step (c) measures the wrong branch. It names three no-answer questions to exercise the loop. On the calibrated KB one of them — "Which software does the catalogue run on?" — scores medium, which takes the cheap branch: running the runbook as written buys a synthesis call and records it as a loop measurement. The free pre-flight the same document prescribes is what caught it, one paragraph after the document tells you to run it. Step (b) already warns that "a 2 means the branch was mis-selected and the run measured the wrong thing"; the inverse case had no such check, and the questions were never re-verified after the thresholds were fitted.

HIGH — and the reason step (c) chose those questions is itself false. It argues that no-answer questions are the right instrument because "nothing in the corpus answers them, so the sufficiency gate cannot stop the run early and it goes to the round cap — which is the worst case the reservation was sized for, and therefore the only case that measures it." Measured: both decomposition runs stopped at sufficiency, after 2 rounds and after 1 round of 3. A sufficiency gate reading a calibrated signal is perfectly willing to conclude that enough has been established about a question the corpus cannot answer. So on a calibrated KB the loop's worst case is not reachable by choosing a hard question, and the branch that actually reaches the round cap is unknown, on a KB with no thresholds at all — the branch the runbook does not mention.

MEDIUM — the more calibrated the KB, the more over-reserved its loop is. The three branches came out 29.75× (synthesis), 50.92× (calibrated loop) and 22.35× (uncalibrated loop). The ordering is not noise and it is not a defect: a reservation must cover max_rounds, and calibration is exactly what lets a run stop before it gets there. The uncalibrated branch is the least over-reserved because it has no early stop and spends the rounds it reserved. Reporting a single blended figure would have hidden this entirely, which is the argument D-28 made before any of it was measured.

MEDIUM — MAX_TOKENS carries most of the over-reservation, and is the constant least safe to lower. 8,000 reserved against a widest-observed 660 across 22 reconciled calls (mean 241) — 12.12×, against 1.50× to 8.93× for the five input constants. It dominates because output bills at five times input and is two thirds of a round's price under the shipped defaults — and four fifths at the narrower geometry these runs actually used. It is also the only one of the six that is a truncation rather than a bill: an input ceiling set too low over-reserves, while an output ceiling set too low cuts an answer off mid-sentence. The temptation to lower the one constant that would visibly improve the headline figure is therefore precisely inverted from where it is safe to do so, which is worth stating plainly next to a 12× ratio.

LOW — the [budget] block the runbook tells you to append is now two-thirds the shipped default. It prescribes per_operation_eur = 2.00 and daily_eur = 6.00; D-30 raised the defaults to exactly those values in 0.24.0. Both measurement KBs ran the whole plan with no [budget] section at all. Harmless, but it is a hand-editing step the document still asks for and no longer needs — and the two keys that do still differ (confirm_above_eur, monthly_eur) are the two it does not explain.

LOW — the refusal path had never been run, and works exactly as specified. Step (d), untouched through two measurement sessions. With per_operation_eur = 1.00 against the loop branch's €1.38 estimate: refused before the first call, exit 1, no ledger row (22 lines before and after) and no transcript (7 files before and after), with a message naming the cap, the headroom, the branch, the call and round count, the complete manifest edit that would admit the run, and the two cheaper routes before raising a cap. D-23 and E5's "a run that never returned writes none" both hold.

A prose tool pointed at a comment run containing load-bearing whitespace (20260821 22:48)

Found by the E6-close adversarial pass, after ./check.sh was green — twice. While annotating estimate.py's constants, textwrap was used to reflow the affected comment runs to fix line-length errors, and it flattened an indented \-continued shell command into ... \ tests/demo-kb/docs on one line — a literal backslash argument for anyone who pastes it. Caught before the tag, fixed in 4d5debf.

The class, and why no gate can see it: the result is legal, correctly-lengthed comment text, so ruff accepts it; it is a comment, so pyright never reads it; and a diff review reads the new prose rather than the old command. No assertion anywhere could have failed — which makes it a clean example of what the mutation battery cannot reach, now noted in docs/BUILDING.md § 4. The rule: a prose tool's output over text containing load-bearing whitespace is re-read as the thing it is — a command, a table, an indent — never as prose.

The sibling case, from the same day's release sweeps: a script that navigates Markdown by a heading prefix steps over every shallower heading. 0.25.3's sweep inserted its ROADMAP section with a startswith("## ") scan for the next section — and # Part 5 · What is not built starts with "# ", so the scan stepped straight over the divider and the Part heading and placed a finished release section inside the unbuilt-work part. release_order_gate could not see it: it checks order among release sections, never which part they sit in. Diagnosed by the sweep's own author from the script, not guessed; the other three insertions from that sweep were each anchored immediately after a located line and are correctly placed — the one that navigated forward to find its slot is the one that crossed a boundary it could not represent. Moved in 0.25.4. The rule both halves of this entry share: a prose-shaped tool applied to a file whose structure carries meaning it does not model needs its output re-read against that structure — heading level, load-bearing whitespace, a divider — never against the text alone.

The same pass also found a shorthand ("output is two thirds of a round's price") that is correct beside the shipped defaults and wrong at the measurement KB's geometry (four fifths there); it was qualified at its five new uses only, deliberately — the pre-existing uses sit beside default-geometry arithmetic, where the shorthand is exact, and a blanket qualifier would cost every reader to protect none.

E7 — Printed suggestions, and the mutant that proved the battery blind (20260822 01:24)

The increment that closes the deep release. A --deep run ends by printing the links[] entries its own citations propose. Small — one module, one CLI seam — and it turned up four defects the tests could not see, three of them found by mutating rather than by reading.

HIGH — the control mutant survived, and it was the first one run. docs/BUILDING.md § 4 says to kill a known-catchable mutant first, because a run with no kills is a broken harness rather than a clean bill. This run had seven kills and lost the control: REL = "co-cited" → "related" left all 71 tests green. Every assertion about the relation imported REL and compared it with itself, so the constant and the expectation moved together. The shipped value of the thing a user pastes was pinned by nothing.

The generalisation is worth more than the fix: a constant imported into both sides of an assertion is not tested, it is restated. test_cli_ask.py's FINGERPRINT comment says exactly this one file away — "a test that derives the expected value from the same object it checks would still pass if fingerprint() started returning the empty string" — and this suite was written without noticing it applied. One literal test now names ("co-cited", "deep"). Half of it was already caught by accident: rel = "" went red, because sidecar._links refuses an entry without one. Only the plausible wrong value survived, which is the direction that matters.

HIGH — a test that certified containment was satisfied by absence. The escaping-path test cited ../outside.md, a path with nothing at the end of it, so the read failed because the file was missing and the assertion passed with no containment check at all. The mutation pass is what separated them: the bypass mutant killed the neighbouring deleted-document test and left this one green, on the same code. It now cites a real document in a second KB next door, with a sidecar carrying the id the citation claims — so only containment can refuse it, and what it prevents is nameable: a fragment carrying pnk://<this KB>/<that KB's document>.

MEDIUM — three direction tests could not observe direction. ULIDs are monotonic, and the fixture built its documents alpha, beta, gamma, so the ids ascended in the same order as the paths. Every assertion about which sidecar an entry lands in was green whether the code ordered by path or by id. Minting the fixture backwards made the two orders disagree — and immediately paid: the entry order inside a sidecar was by URI, which is mint order, which is arbitrary to a reader. A fixture whose two orderings agree cannot tell them apart, and the agreement is usually accidental.

MEDIUM — a newline in a filename would have broken the fragment, silently. The module builds YAML as text, which is defensible: every value in it is a ULID URI or one of two constants, and a test pins that against sidecar.needs_quoting. But the document paths go into YAML comments, and a POSIX filename may contain a newline — one ends the comment and turns the rest of the path into a node. needs_quoting cannot see it, because it answers about a scalar and a comment is not one. The lesson is the sibling of 0.25.3's reflowed shell command: a value that is safe as a scalar is not thereby safe as a comment, and the check that certifies the first says nothing about the second.

Two more, from reading the diff as a stranger. propose is public and re-checks every endpoint it is handed, but would have proposed a document linked to itself — which pnk link refuses outright. And resolution was quadratic in disk reads: a block citing n documents makes n(n-1)/2 pairs, each resolving both endpoints, so every sidecar was read and YAML-parsed n-1 times, with [retrieval] final_k having no ceiling and the money already spent.

The design decision worth keeping: observing and proposing are two functions, and the split exists so a test can exist. § 5's rule is that a suggestion's endpoints must be documents this run retrieved. Enforced inside one function, the rule is unfalsifiable — every candidate comes from the same expression that validates it, so a working guard and a missing one produce identical output on every input a test could construct. co_citations observes; propose re-checks what it is handed; the refusal test calls propose with a pair no run would produce. A guard whose input is built by its own validator is not a guard, and its test is a tautology in test clothing.

What the battery still cannot reach. The prompt-injection test — a retrieved passage instructing the model to add a link, obeyed in prose, producing nothing — asserts the absence of a behaviour. No mutation makes it fail, because the code that would fail it was never written: nothing in the module reads AnswerBlock.text. It is a real test of a real property and it is not mutation-backed, which is the same class docs/BUILDING.md § 4 names as a defect with no assertion anywhere, seen from the other side.

And an old row that had been false for four releases. docs/DESIGN.md §9 still bounded --deep with "no orchestration the free path doesn't have" — written before the loop existed, and contradicted by the loop the moment it shipped. docs/graph/PINAKES_APPROACH.md § 6 had asked for that exact row to be amended in the increment that shipped the design, and named the replacement bound. E4 shipped and the row did not move. Found here by auditing the neighbourhood rather than the diff — the increment that closes a release is the last cheap chance to fix what the release made false, because after it nobody is reading those rows for a while.

Ten of ten mutants killed after the fixes, control included, with the containment mutant now caught by the containment test rather than by its neighbour.

The mutation harness — turning the last unguarded step into a tool (20260822 06:14)

plans/20260821_0745-mutation-harness.md, in one increment. The precedent is tools/land.py: when prose has failed repeatedly against a class of mistake that fails silently, the rule stops being a rule and becomes a tool.

HIGH — a skipped test and a passing test are the same exit code, and this repository skips constantly. Not in the plan; found by measuring pytest before writing any code. A selector whose tests all skip exits 0, which is byte for byte the SURVIVED signal, and Pinakes skips on a missing extra by design (pdf, paid, model, the three-leg CI matrix). A battery aimed at one of those on a [light] checkout would have reported every mutant in it unpinned — a confident, wrong claim about exactly the assertions a mutation pass exists to check. The mirror case is worse and equally invisible: a selector that is already red reports KILLED for every mutant aimed at it. One pre-flight run per selector closes both, before any file is written.

HIGH — SIGTERM skips finally, so a try/finally restore is not a restore. Measured, 1 of 1. SIGINT already raises KeyboardInterrupt, which is why the hazard is invisible to anyone who only tests with Ctrl-C — and why the first version handled only SIGTERM until the harness's own battery showed that dropping SIGHUP and SIGQUIT was a one-word edit no test could see. SIGKILL cannot be handled at all, which is the real reason the plan's refuse if the target differs from HEAD step is not a formality: after a hard kill the only recovery is git checkout <file>, and the refusal is what makes that recovery correct rather than the seventh instance of the trap.

MEDIUM — the T3 trap reproduces 6 times out of 6, and its test must not race the clock. A same-length mutant (min(value, MAX) → max(value, MAX)) written in the same wall-clock second as the previous compile passes every test, because CPython validates a .pyc on (mtime-to-the-second, size). Reproducing it first is what made the test honest: the obvious test — assert the same-length mutant is KILLED end to end — goes green on a slow machine with the invalidation deleted, because the second boundary is crossed anyway. Two tests replace it. One asserts no bytecode cache exists during a mutation, which has no clock in it at all; the other forges the stale condition with os.utime and watches the mutant vanish, so the clearing has a control rather than being ceremony.

MEDIUM — pytest's <error> tag covers two opposite events, and conflating them threw away real kills. A collection error is the invalid mutant: nothing ran, no assertion was tested. A setup or teardown error is a real node the mutant broke on the way in or out — and in this repository fixtures build indexes, manifests and KBs out of src/, so fixture-mediated detection is the common shape rather than a corner. Treating both as ERRORED, and testing that before failures, reported a mutant that tripped a fixture and failed a plain assertion as "the mutant did not run", tallied 0 killed. The two are now told apart structurally — a collection failure carries no line attribute — rather than by matching pytest's message text.

The correction that mattered more than the finding. The reviewer who filed that one also proposed the fix: route setup and teardown errors into KILLED. An independent skeptic, asked only to refute the finding, confirmed the defect and rejected the remedy — routing them into KILLED would manufacture the false green the tool exists to prevent, because no assertion fired and nobody may write "pinned by test X" for a fixture noticing something. The shipped fix keeps the conservative direction and repairs only the sentence. An adversarial pass is worth more when the verifier is allowed to disagree with the finder's remedy as well as with the finding.

The shape that recurred three times: a guard the CLI cannot reach. Pointing the tool at itself found, in three separate rounds, four clauses that no battery-driven test could kill — the second anchor check inside applied(), classify's timed_out and setup_errors branches, and the report's only-a-KILLED-row-may-name-a-killer guard. Each is redundant against the code as it stands today and each guards a state one line of drift away. Every one surfaced as a SURVIVED row, which is the row the tool exists to print; none would have been visible to a reviewer reading the diff. They are kept, and each gained a direct test on the constructed state rather than being deleted as dead code — redundancy nothing tests is indistinguishable from redundancy that has quietly stopped working.

Twice, the battery's own selectors were the thing that was stale. Both times a guard read SURVIVED because the battery still named the test that existed before the fix, not the one written for it. The lesson is narrow and mechanical, and it is about the battery rather than the code: a battery is source that goes stale like any other, and a SURVIVED row is a claim about a pair — the mutant and the selector — either half of which can be wrong. The third time, the anchor pre-flight caught it before a single mutation ran, which is the whole argument for hoisting that check.

What the tool still cannot see. A survivor is a claim that wants checking by hand — the record holds two increments where a mutant that did not reproduce the real prior logic was briefly taken for a result, and the E7 session's own control mutant survived because every assertion imported the constant it compared against. A mutation result implicating a file the mutant never touched is a result about the harness, not the code, and nothing here detects that.

The exit criterion, met the only way it could be. The tool was run against itself: 25 mutants, each disarming one of its own guards, 25 killed, each by the test named beside it. Before that, two mutants of src/pinakes/graph/traverse.py's caps, both killed, tree clean afterwards — because a tool that has only ever been run on its own fixtures has not been run.

A gate cited by a procedure it cannot read (20260822 06:35)

docs/RELEASING.md's sweep table names five places a release stales. One is STATUS's Published on PyPI prose. Its "where the new entry goes" row answers: python3 tools/release_order_gate.py decides it. No pattern in that gate matched the list. So the procedure delegated a placement decision to a check that could not read the document, and the green line 5 sequences in release order was read as covering a list it had never opened.

It drifted, as delegated-to-nobody things do: 0.25.1 → 0.25.3 → 0.25.2 → 0.25.4, wrong on SemVer and on verification time, surviving every green run from 20260821 to 20260822.

The generalisable shape is not "a missing pattern". It is a citation nobody checked. A document naming a tool as the authority for something is a claim about that tool's coverage, and it is exactly the kind of claim that is written once and never re-read. Grep the other direction occasionally: for each gate, what do the documents say it covers, and does it?

Three things that fell out of building it, each worth more than the fix:

• A gate reads an order; it cannot see a count. Landed the same day, from the other side: a concurrent session appended "thirty-seven" to a sentence that still said "thirty-six", one line apart, through a green check.sh, a green make docs and a green release_order_gate. Caught by grepping the neighbourhood, not by reading the diff — the wrong half was context, so it never appeared in the diff at all. Counts stay a documentation rule for that reason.
• "Tolerate it" needs a direction, or it is a hole. This list is legitimately short between a release landing and its verification, because a claim about the index is held back until it is verified from the index. Exempting it from agreement is right; exempting it from direction is not. It may lag every other sequence and may never lead one — a paragraph about a release the CHANGELOG has never heard of is a claim nothing else records. The first draft had the exemption and no direction, and nothing could have told the two apart.
• Say what the exemption costs, in the gate. Because a missing newest entry is legal, an entry written in a shape the pattern does not match is indistinguishable from one not yet written — silently unchecked. The floor catches wholesale rot; it does not catch one mis-shaped newest entry. Undocumented limits get trusted past.

And one environmental trap, found before it could produce a false green: this repo has no .python-version, so uv gave a fresh worktree CPython 3.14.7 while the primary checkout and CI run 3.13. A green ./check.sh in a new worktree is therefore not evidence about the interpreter anything else uses. Pinned by hand here; the root fix is a proposal, not this increment's.

Verify the remedy, not only the finding (20260822 07:25)

docs/BUILDING.md § 5 already says a fix applied under review inherits the review's confidence and none of its scrutiny. This is its sibling, and it is about the review pass rather than the commit after it: a correct finding arrives with a proposed remedy attached, and the remedy is the half nobody re-checks. The finding survives scrutiny; the remedy rides in behind it.

Two instances the same night, in two sessions working concurrently, each caught only because the second pass argued with the fix rather than stopping at the defect:

• A remedy that would have manufactured a false green. A classify bug was real and confirmed. Its proposed fix would have made the check report success on input it had never examined — a repair that removed the symptom by removing the observation.
• A remedy that was right and unbounded. STATUS's Published on PyPI list is legitimately short between a release landing and its verification, because a claim about the index is held back until it is verified from the index. Exempting that sequence from the release-order gate's agreement check was correct. Exempting it without a direction was a hole with a docstring: no input could distinguish a working exemption from a missing one. It became may lag, never lead — the list may fall behind every other sequence and may never name a release the CHANGELOG has not heard of.

A review culture that re-checks only findings ships both of these at full confidence. Both were defensible, both were proposed by the agent that had just been right about something, and that is exactly what makes a remedy hard to doubt.

The cheap test: state what input would look different if the remedy were absent. If none exists, the remedy is unfalsifiable and the test written for it will be a tautology — the same failure E7 recorded, where a guard whose input is built by its own validator cannot be shown to guard anything.

The cheap structural version, if only one thing survives from this: when an adversarial reviewer is asked to refute a finding, ask it to refute the proposed fix as a separate verdict. Both instances above were caught that way and neither was caught any other way — one by accident, when a skeptic exceeded its brief, and one by design, when the second pass asked tolerant in which direction?. It costs one more question per finding. The mechanism and the first instance came from the concurrent session; this fragment shipped without them and had to be amended, which is the argument in miniature.

And it worked because reviewing was somebody's assigned job. That amendment was found by a session whose remaining task was to look at work other people had finished — development split to one agent, planning and review to another. A review that happens as a courtesy after the work is done is the one that gets skipped when the work runs long; the catch here depended on it being nobody's optional extra.

A coda from the concurrent session, worth keeping beside this. The night's most serious defect — pnk serve raising ModuleNotFoundError on every fresh install of every published version, from a lower-bound-only dependency pin that 37 --frozen CI invocations never resolve — was not found by a test, a gate or a review. It was found by someone asking what should we build next and installing the product to answer it. docs/BUILDING.md warns that a test seam defines a region no test reaches; here the region no test reached was the product itself. Reproduced independently in both sessions before it was written down.

Capping mcp, and the leg that had to be run against the defect (20260822 08:20)

HIGH — a CI leg written to catch a defect is a claim until it is run against the defect, and the mutation battery cannot reach this one. No mutant of any tracked file makes uv resolve mcp 2.0.0: the thing under test is a dependency resolve, and tools/mutate.py mutates bytes in the tree. So the leg was verified the only way left — git archive HEAD | tar -x into /tmp, the cap deleted from the copy, uv build, and both new steps pointed at the resulting wheel:

wheel-import: 1 module(s) did not import against the resolved dependency set:
  pinakes.serve: ModuleNotFoundError: No module named 'mcp.server.fastmcp'

and pnk serve exited 1 with no "serverInfo" anywhere in its output. (That block is the run's real stdout, pasted. The first draft of this fragment paraphrased it into two module lines — in the fragment whose whole thesis is that the artifact has to be run. Caught by review; a quoted output that was retyped is a claim wearing evidence's clothes.)

Without that, "the leg would have caught it" is a statement about a check that has only ever been run against a fixed tree — the shape docs/BUILDING.md already refuses for a test ("pinned by test X is a claim about a failing test"). The generalisation: when the battery cannot reach a guard, build the broken artifact by hand and point the guard at it. It cost about four minutes.

HIGH — the review's own remedies were where the remaining defects were, exactly as verify the remedy, not only the finding predicted the night it landed. Two of the three worst findings in this increment were not in the first draft's logic but in the mechanisms written to guard it:

• An allowance keyed on the library forgave the module the gate exists for. --allow-missing pypdfium2 excused any module failing on pypdfium2 — including pinakes.serve, had anything on its import chain ever reached it. Demonstrated against a synthetic package: serve did not import and the gate printed a clean pass. An allowance now names the module and the library, and a --required module may never appear in one.
• The "the gate can still fail" step was satisfied by an environment where the gate never ran. It grepped the tool's generic failure headline — which the tool also printed when the package itself was not importable, i.e. when --with installed nothing (a mistyped extras syntax does exactly that; measured). The two branches now print different headlines and the step names the one failure the bare wheel genuinely has. The defect was inside the step written to prevent that class, and a test pinned the collision rather than catching it.

The fragment's cheap test applied to this increment's own thesis — what input would look different if the remedy were absent? — is the uncapped wheel above. It exists, and it is the answer to the question this increment turns on: without it, the cap is the only real change and the CI leg is decoration.

HIGH — and then the remedies for those had defects, in the same shape, found by a second review that was asked only to refute them. Three rounds, each finding less than the last, and the third still finding four things that mattered:

• The guard against a missing wheel failed open. set -- dist/*.whl; test -f "$1" && test $# -eq 1 — under set -e, a failing first command of an && list does not abort, so an empty dist/ fell through to wheel="$PWD/dist/*.whl" and carried on. The two-wheel case aborted, which is why it looked right. It is an if now. Two sessions found this independently within minutes of each other; it came in as a review suggestion and was implemented verbatim.
• make smoke exited 0 while printing a traceback. pnk serve … | grep -q '"serverInfo"' — grep matches, closes the pipe, the server dies on BrokenPipeError, and the pipeline's status is grep's. The target printed "answers an MCP handshake" over a crashed server, and a changelog entry claimed it ran the same checks as CI. Output to a file, then grep it twice.
• continue-on-error: true defeats an assertion written against if:. The test said a gate that can be switched off is not a gate and then checked one of the two ways to switch one off. On the step in front of uv publish, where being wrong is a version number PyPI will not take back.
• The source-tree refusal was scoped to the checkout the script lives in. This project mandates a worktree per change, so another checkout's editable install is a path the gate has never heard of — demonstrated with one checkout's interpreter and another's gate: a clean green pass over src/pinakes. A negative check has to enumerate every wrong answer; the positive one — the package must be inside a site-packages or dist-packages directory — has one right answer, and that is the difference between the two shapes.

And a declaration test only pins the file the last edit touched. --min-modules 50 sat at four call sites across three files; deleting it from any one of them survived every test, and the tool then falls back to a floor of 20 against a package of 57 modules. The assertion now iterates the invocations rather than grepping the tree, which is the same lesson as read the sequence, not the neighbourhood in docs/RELEASING.md: a property of a set is not a property of any member of it.

MEDIUM — a gate that reads a range of lines reads the prose inside the range. check.sh's extras-not-core gate is awk '/^dependencies = \[/,/^\]/' pyproject.toml | grep -qiE 'pypdfium2|anthropic'. The comment added above mcp — explaining that anthropic was measured and deliberately not capped — turned it red and reported anthropic as a core dependency. The gate had been correct for a year only because nobody had written a comment in that block. Same class as 0.25.3's textwrap reflow of a \-continued shell command: a line-shaped tool applied to structure it does not model. The parsed-side test never had the problem, because it reads the requirement list rather than the lines.

MEDIUM — nothing resolves a test path written in a comment. check.sh and tests/test_paid_path.py both pointed readers at test_packaging.py::test_paid_and_pdf_clients_stay_out_of_core. That test has never existed under either file's history; the real name is test_extractors_stay_extras. docs/VERIFICATION.md is gated for precisely this reason — a table of test paths is prose until something executes it — and a comment is prose that nothing gates at all. Both fixed; the class is open everywhere else.

MEDIUM — measure the sibling before capping it, and measure the surface you actually call. Capping all three lower-bound-only requirements was the obvious response and would have been wrong. Both siblings were measured: anthropic 1.0.0 and sentence-transformers 6.0.0 keep every constructor parameter, response-model field (Message, Usage, MessageTokensCount) and method signature (encode, predict, get_sentence_embedding_dimension, max_seq_length) that src/ touches. The first pass measured only the constructors, and review caught that. The gap mattered: extract/claude.py consumes response.model_dump() as a dict, so a renamed Usage field would compute a cost of 0 — the spending guard silently disabled, with no exception anywhere. A constructor signature says nothing about that. The remedy for the class is testing the resolve, not capping on reflex; a project that caps everywhere buys the same silence with a different cause.

The battery is what turned four rounds of prose into four rounds of evidence, and the survivor count went the wrong way twice before it went the right way.

Round	Mutants	Survived	Where
1 — the increment	11	0	—
2 — after review 1's fixes	21	3	tests written from the fix's own description
3 — after review 2's fixes	26 (reviewer's) → 33 (mine)	12 → then 2	the remedies' own surface
4 — after review 3's fixes	25 (reviewer's) → 40 (mine)	19 → then 0	the remedies' remedies

Round one said 11/11 and meant it about eleven assertions, which is all a battery ever means. Every later round found survivors in the surface the previous round's fixes had added — exit 1 changed to exit 0 in a step that then printed ::error:: and reported success; a continue-on-error on the job under tests that guarded the steps; --min-modules 1 under a test that asserted the flag was present; a leading - in a Makefile recipe, which is make's own way of ignoring an exit status. A remedy is new code, and new code nothing has tried to break is a claim. The count that matters is not 40/40, it is that three consecutive reviews each found something and the fourth round's mutants were written by someone who had not written the fixes.

What the new legs still cannot see, named so nobody reads them as more. Import-time breaks only, on the install states CI runs. A dependency that keeps its module layout and changes a signature passes every step added here and fails on a user's machine. [st] — the default backend — is resolved fresh by nothing, because it is ~2GB of torch. And ci.yml runs on push and pull_request, so a major published on a quiet day is caught at the next push or at the tag, never before: release.yml is where that matters, and it now carries both checks in front of uv publish.

A gate must not derive a constant from the document it polices (20260822 17:52)

Membership's first draft derived each sequence's start version from the sequence itself — "the oldest release it contains" — and justified it in the commit message: the starts are observable and monotonic; they never move backwards, because releases are never deleted. Every clause of that is true and the conclusion is still wrong. The gate does not observe releases. It observes the document. Delete STATUS's 0.2.0 row and the derived start becomes 0.2.1: the sequence is still sorted, still contiguous, still internally consistent, and the gate reports green on precisely the deletion it was built to catch.

It is the same failure the gate's own docstring already refuses one paragraph earlier — the direction is declared per sequence, never inferred, because a badly scrambled file would otherwise elect its own answer. I read that sentence while editing the file and then wrote the inferred version of a different constant. A rule stated about one field does not defend the next field somebody adds. Four declared constants cost four lines and cannot be argued into being wrong.

The general shape, worth checking on any gate here: for each constant the gate uses, ask where it came from. If the answer is "the thing being checked", it is not a constant, it is an echo.

A status read is not a status gated on

CLAUDE.md names this trap in its piped costume: check | tail && git commit reports tail's status, so a failing checker looks green. I hit the unpiped one and it is worth naming separately, because the written rule did not cover it and I had read that rule the same day:

./check.sh > log 2>&1; echo "exit=$?"      # status printed, and read, and correct
git add -A && git commit ...              # runs regardless, on the next line

The status was captured, printed, and true. Nothing consumed it. A commit landed over a red tree — a lint error, caught later — and the log shows a green-looking sequence of steps. The pipe was never the mechanism; the mechanism is that the exit status must be what the next command reads, and a human reading it off the screen is not the next command. The fix is one wrapper:

./check.sh > log 2>&1; RC=$?
if [ $RC -ne 0 ]; then …report…; else …commit…; fi

And a third instance of verify the remedy, not only the finding

The exception mechanism for 0.11.0 was agreed with the planner as a loose match — find the version anywhere in a ## heading. Implementing it meant first finding where that heading actually is: docs/ROADMAP.md:1721, inside # Part 5, with the release table linking to it on purpose. 0.11.0 has no Part 4 section by design. The loose match would have taken that heading as its release section and failed twice on a correct document — placement, because Part 5 declares no range, and ordering, because it follows 0.27.2.

The finding was right: membership needs an exception mechanism. The remedy attached to it was wrong, and the risk that killed it had been named in the same message that proposed it — "matching a version anywhere widens what counts as a release section" — and read by both of us as hypothetical. It was not hypothetical; the widened set already had a legitimate member. A risk stated beside a remedy is not a risk that has been checked.

The mcp 2.x port — the port was four lines, the gate was not (20260822 18:23)

HIGH — A gate whose passing depends on a dependency's teardown timing is not a gate. The handshake in both workflows wrote three JSON-RPC lines into pnk serve and closed stdin. Measured on the same three lines, same KB, ten runs each:

	initialize answered	tools/list answered
mcp 1.28.1	10/10	10/10
mcp 2.0.0	10/10	2/10
a real ClientSession, 2.0.0	8/8	8/8

1.x drained the queue before shutting down on EOF; 2.x does not. Nothing in the step named that dependency, and nothing could have: it is a property of a library's shutdown path, not of its API. The generalisation is worth more than the instance — if a check's passing depends on a race nobody in this repository wrote, it is not a check — and the remedy is not a longer sleep but removing the race's cause, which here meant driving the session with a client that holds the connection open until it has its answers.

The timing is the uncomfortable part. That handshake was written the same morning, in 0.27.2, to catch the mcp outage — and it was written against the behaviour of the version it was about to lose. A gate built for a dependency major was itself a hostage to that major. git log -S '"method":"initialize"' returns exactly two commits: the one that added it and the one that removed it, four hours apart.

HIGH — The third copy is the one nobody looks at. Two workflows were fixed; make smoke — the pre-tag check a maintainer actually runs — kept the dead shape and was red on every run against a healthy wheel. Review found it, and found the second half too: test_make_smoke_exercises_the_wheel_rather_than_only_its_version asserted '"method":"initialize"' in body and passed, certifying "drives a handshake" against a recipe that no longer produced one. When a shape is replaced because it is wrong, grep the whole tree for that shape before believing it is gone — and check what the tests pinning the old copies now assert, because a test written against an implementation outlives the implementation.

HIGH — There is no independent source of a project's own version inside its own checkout. The version test compared what the server advertised against importlib.metadata.version("pinakes"), reaching for independence from pinakes.__version__. [tool.hatch.version] reads that same file, so the metadata is the same constant copied into .dist-info at install time — and uv run does not refresh it when the constant changes. Bumping __version__ to cut the release diverged them at once and reddened two tests, with a message blaming the serverInfo defect the increment had just fixed. A landmine the release commit would have stepped on twenty minutes later.

The lesson is not "use the other source". It is that a longer route to the same origin is not a second opinion, and saying so in the docstring is worth more than the appearance of rigour. The genuine cross-check exists, but only in CI: the expected value comes from the built wheel's filename and the advertised one from a separately installed copy of it.

MEDIUM — An error message is part of the assertion surface. assert EXPECTED in result.stderr looked like it checked that a failure names what was expected. It was satisfied by the gate's own sentence, "Until 0.27.2 this field carried the mcp library's version" — because EXPECTED was 0.27.2. The assertion held with the fix deleted. A version literal in a diagnostic is a value a test may accidentally be reading; the message now names no version.

MEDIUM — The negative form of a rule is not the rule. Two exit-status pins asserted the absence of one literal string, out=$(timeout — the shape the step used to have. | tee, || true, if …; then and a backticked capture all stayed green, and | tee is the likely edit, since the shape it replaced existed precisely to print the gate's output. The positive form — this invocation is a simple command under timeout — is one assertion instead of an ever-growing list of forbidden ones. Generally: when a rule says "the status must reach set -e", assert the shape that makes it true, never the shapes that made it false last time.

MEDIUM — step["run"] is a YAML literal block, comments included. release.yml's pins were bare substring searches over it, so commenting out the commands left every assertion satisfied by the prose above them. Its ci.yml twin had required command lines since 0.27.2 for exactly this reason and the release side never got it — a fix applied to one of two twins is half a fix, and the release path is the half where being wrong is irreversible.

MEDIUM — The only untested branch was the one the gate exists for. A server dead on import — the 0.27.2 outage itself — reached the gate's generic session-failure path, and no test drove it. Also untested: the timeout ceiling and serverInfo.name. All three now have one, and --timeout became a real flag so the timeout test costs a second rather than thirty. Ask which branch the tool was built for, and check that one has a test before counting the others.

MEDIUM — A test can pin a message and leave the behaviour free. The timeout test asserted "no complete session in 1s" and nothing about the clock. The mutation battery wired timeout= to a constant while the message still read the flag, and reported ERRORED rather than a kill — the harness refusing to call a run that never finished either a kill or a survivor, which is exactly what tools/mutate.py was built to do. The test now asserts elapsed time as well, with a loose bound: what must be true is seconds rather than the job.

LOW, and a pattern rather than an item — four false statements, all in prose written to explain why the code was careful. SESSION_TIMEOUT_SECONDS was commented "Ten" beside a value of 30; a docstring said comparing __version__ with itself "would hold with the version= argument deleted", which is simply untrue; tools/wheel_import_gate.py still said uv.lock pins 1.28.1; and a changelog fragment called the flaky handshake "a coin flip that landed the same way for a year" when git log -S puts it at four hours old in a repository 28 days old. Explanatory prose is where false claims accumulate, because it is written once, at the moment of most confidence, and never re-read against the code it explains.

LOW — Two run-the-real-thing checks that no unit test could replace. The three CI legs were run verbatim against a built wheel resolved fresh onto mcp 2.0.0, which is the only way to learn that uv run --isolated --no-project --with <wheel> puts pnk on PATH for a child process the gate spawns. And the gate's timeout was driven with a shim that spawns and stays silent: it exits 1 after 32s and leaves no orphan behind it.

Process — the primary checkout's tree can change under a running gate. A peer session ran tools/land.py while ./check.sh was running in that same directory, and three tests failed with "matched 0 # Part heading(s)" — loud, plausible, and pointed at the file the peer had just touched. Nothing was wrong; a re-run at the new sha was green. Run gates in your own worktree. The primary checkout is where landings happen and its tree can change under a running pytest at any moment; a red run there may be about a tree that no longer exists.

Review economics, recorded because the numbers are the argument. Round one: four lenses over the diff, each finding verified by an adversarial refuter. 31 findings raised, 19 confirmed and 12 refuted — and the refutations were substantive, one of them disproving a claimed process-group escape by reading multiprocessing.util.spawnv_passfds on the machine. Two of the confirmed findings were defects I had already found myself; the other seventeen I had not. The mutation battery grew from 15 to 24 mutants over the same increment, all killed. The refutation stage earned its cost twice over: it is what makes 19 findings worth reading rather than 31 worth arguing with.

When a constant cannot be declared, constrain it (20260823)

The echo class — a gate reading a constant out of the document it polices — turned up four times in one gate, and the interesting part is that the same remedy did not work every time.

Echo	Exploit	Remedy
sequence start = its own oldest entry	delete the oldest row; the start moves and hides it	declare it
lagging ceiling = its own newest entry	delete the newest entry; the ceiling drops with it	bound how far it may lag
Part range = its own heading	append a range to # Part 5 and a misfiled section is legal	constrain it
Part count floor = one below the truth	demote # Part 5; the floor passes exactly	raise it to the real count

The third could not be declared, and that is the lesson. Reading Part ranges out of the headings is why the mapping cannot drift from the document — replacing them with a table in the tool would reintroduce the drift the gate exists to catch. So the echo had to stay and be made unexploitable instead: two Parts may not claim the same version, and the Parts must ascend. # Part 4 declaring `0.8.0` onward is then precisely what stops # Part 5 declaring it.

Declare, bound, constrain — in that order of preference. Declaring is strongest and cheapest when the value is genuinely external. Bounding admits the echo and limits its travel. Constraining leaves the echo and removes the freedom that made it exploitable. Reaching for the first when only the third is available produces a table nobody updates.

A floor one below the truth is a floor with a bypass. PARTS_MINIMUM was 4 against 5 real Parts, so demoting the last heading passed it exactly while handing every section beneath to the open-ended Part above. Floors here are written as "this only ever holds, because things are never removed" — which is an argument for setting them at the count, not below it. A floor with slack is a floor someone can stand in.

And a test that asserts a sentence asserts only that something went wrong. Three instances in one increment, each found by mutation and none by reading:

• a range-form test asserting a failure — satisfied by breaking the form entirely
• placement fixtures guarded by assert "reads ascending" not in stderr — satisfied by a reworded message, and by a second failure appearing beside the one under test
• an ordering test asserting "must ascend with the document" — satisfied with the comparison reversed, because a different, correctly-ordered pair then fires and prints the same words

The positive form in each case: assert exactly one failure and which one — the pair, the Parts, the versions. failures_of() exists for that.

On the audit itself. These were found by four independent lenses over the landed gate, each finding then handed to a separate agent told to refute it — not by re-reading the diff. Two of the four lenses found nothing. The two exploits came from the lens that was asked one question only: for every constant this gate uses, where does its value come from?

A correction is a claim like any other (20260823 02:05)

Two sessions spent 20260822–23 finding one defect shape from four directions, and the fourth instance landed on the correction rather than on the code.

A handoff file said the release-order gate's placement and membership halves shipped in 0.27.1 at ba4d7ae. That was wrong. The reviewing session corrected it to "placement and membership landed after 0.28.0 and are unreleased" — which was also wrong. git describe --contains settled it in one command:

09f5449 (placement)   -> v0.28.0~2
0a8bd38 (membership)  -> v0.28.0~1
ba4d7ae (lag + Parts) -> in no tag

The reviewer's own diagnosis is the thing worth keeping: one true fact was established — ba4d7ae is tonight's merge, not 0.27.1 — and extended to a second that did not follow. The true half made the false half feel checked. The evidence to settle it was one command away and was run only after pushback.

This is the same shape as a fix applied under review inherits the review's confidence and none of its scrutiny (BUILDING.md § 5) and its sibling verify the remedy, not only the finding — one level further out. The correction is the least-reviewed statement in any review, because it arrives attached to a finding that has just been demonstrated, and demonstrating the finding feels like demonstrating the fix.

And a second rule fell out of the same exchange, stronger than the one it replaced. The file's ownership table had a held last by column naming sessions. The reviewer said "put the right name in". The right name could not be kept: the writing session was renamed mid-file, and the two sessions' ListAgents views disagreed about which of them existed. So the column was deleted, not corrected — ownership is by role, with ListAgents for who is live.

When a field cannot be kept true, remove the field rather than maintaining it.

That is the end of the ladder this repository has been climbing all week: declare a constant rather than deriving it; bound it when it must be derived; constrain it when it must be read from the thing it polices; and delete it when none of those can hold it honest. A field nobody can keep true is a lie with a maintenance schedule.

A document's version stamp is the one claim in it nothing can check

docs/GUIDE.md opened with "Every command here was run against 0.2.0 (20260728 16:40)" and was published, unchanged, for twenty-six releases. Nothing was broken by it and no gate could see it: ./check.sh proves the release sequences agree across five documents, mkdocs --strict proves every link resolves, and tools/template_drift_gate.py proves the template matches its archive — none of them reads a sentence claiming a command was run.

The stamp is not a small lie. It is the sentence a reader uses to decide whether to trust the other eight hundred lines, and it was wrong in the direction that costs most: it claimed more verification than had happened. Nine output blocks had drifted behind it, and the two most misleading were the ones a reader would act on — a euro estimate quoted 30% high, and three separate statements that only one surface in Pinakes can spend money when pnk ask --deep had been the second for four weeks.

The pattern is that a stale doc rots fastest exactly where it was most specific. Prose survives a release; a quoted output block does not, because it pins a number, a version string and a wording all at once, and any one of the three moving falsifies it. Nine blocks drifted while the paragraphs around them stayed true. That is an argument for keeping worked examples — they are what catch the drift — and against ever writing one without a way to re-run it.

Two things this pass could not do, and said so rather than faking. The paid --deep transcript would cost real money to reproduce, so it is kept and labelled as a 0.24.0 run with the one figure that has since moved named explicitly. And re-running the two-KB link walkthrough would change every ULID in the section without making a sentence truer. The distinction that matters is between an output nobody re-ran and an output nobody re-ran and did not say so. Only the second is a defect.

A near miss worth recording. The first draft of the label said the paid block's command "is now quoted at €0.20" — the figure measured at -k 2. The block's own command passes no -k, and at the default passage count it quotes €0.21. One cent, and the wrong kind of wrong: two invocations collapsed into one number, which is precisely the invented precision the label existed to prevent. Caught by re-reading the command the block actually shows instead of the one measured beside it.

A gate can be green because it is reading the list next door

The Published versions row drifted four releases while a check named after it ran green on every commit. The check was real, the row was wrong, and both statements were true at once: the gate's sixth sequence reads the Published on PyPI prose, and the row is a different list forty lines down in the same file, under the same heading.

What made it invisible was the naming, not the code. A reader auditing "is the PyPI list checked?" finds a sequence called the Published on PyPI prose, sees it green, and stops. Nothing in that report says which of the two lists under that heading it read. The lesson generalises past this file: when one document holds two lists of the same thing, a gate covering one of them reads as covering both, and the coverage claim is what needs the test — not the ordering.

The corollary for within. Scoping a pattern to a region introduces a failure the line-anchored sequences never had: the anchor can match somewhere else. Zero matches is safe, because an empty sequence trips the floor and the floor's message fits. Two matches is not, because reading the first is a silent choice — the anchor deciding for itself which region it meant, which is the "derived, never declared" mistake this module refuses everywhere else, one layer down. So it is a refusal, not a heuristic.

And a fixture proves scoping only if it would break without it. The tree these tests build writes version numbers after the enumeration and in descending order, mirroring the real cell. A within that stopped scoping reads them into the sequence and the tests go red. Without those trailing numbers every test here would pass over a bare unscoped pattern — the fixture would be asserting the thing it was built to assume.

On the floor: an exact count is declared, and still wrong here. Setting the floor to the 41 versions present would be a literal, not a derivation — but check() skips a sequence whose floor failed, and check_membership then skips it too, so deleting one release would report "the pattern has stopped matching what it names" and never say which release went missing. The precise diagnostic is worth more than the redundant one. Declared-not-derived is about where a number comes from; it does not settle what the number should be.

A constant bounds the damage; a relation catches the start

The first fix made the row a sequence with permission to lag, bounded by MAX_VERIFICATION_LAG. That is a real check and it would have caught both drifts — eventually. Counting the history says how late: 29 commits sat in the window where the row was behind and the bound was silent, and both drifts passed through it on the way to being caught.

The bound is a constant, and the thing it approximates is a relation. The row may lag the release documents because an entry waits for index verification; it may not lag the other list recording that same verification, because there is no interval during which one is true and the other is not. Written that way the rule needs no tolerance at all, and the measurement bears it out: zero false positives in 67 commits, red at the first commit of the drift rather than the eleventh.

The general form: when a bound needs a tolerance, ask what the tolerance is standing in for. A number chosen to be loose enough for the legitimate case is by construction also loose enough for some illegitimate ones. If the legitimate case can be named — here, "both lists are waiting on the same verification" — the relation that names it is both tighter and simpler than the number.

Keep the constant anyway. It is not redundant: it catches a drift in which both lists are forgotten together, which the relation cannot see. Two checks over one sequence, catching different failures at different moments, is the correct outcome — and the test that exercises the deep drift now asserts both fire, so neither can be deleted while it stays green.

Committing the mutation batteries — a design statement nobody had measured (20260823 07:34)

HIGH — the sentence that decided this was an assumption wearing a docstring. tools/mutate.py shipped in 0.27.0 saying "a battery is a per-increment working file, not a portable artifact", in a paragraph whose actual subject was where the pytest key lives. Nothing had measured it, and it then decided the question for two months by being read rather than by being right. plans/20260821_0745-mutation-harness.md enumerates what it deliberately left open and persistence is not among them — checked, not assumed. So this fills a gap rather than overturning a position.

The measurement took twenty minutes. Of 81 mutants across six increments, 78 anchors still resolved exactly once a day to a week later, and every one that broke was one whose own code had changed. The failure mode of keeping a battery is a refusal, not a wrong answer — mutate.py resolves every anchor before it writes anything and refuses on 0 matches and on 2. That asymmetry is what made the question decidable. An argument about whether a claim outlives its proof would not have resolved; what does being wrong cost? did.

HIGH — and the measurement was aimed at the half that had never broken. docs/RETROSPECTIVES.md already recorded which half rots, twice in one increment: "a battery is source that goes stale like any other, and a SURVIVED row is a claim about a pair — the mutant and the selector — either half of which can be wrong." Both my 81-mutant recovery and a peer's five refactor scenarios stress-tested anchors. The selector is what historically went stale. The gate resolves both, and the half that matters is the one the headline number says nothing about. Check the record for which half failed before designing a measurement of the other.

HIGH — the repair loop produced two broken mutants under best-case conditions, and that is the strongest evidence against the model, not for it. Two anchors were repaired here: same day, by the author, with full context. Both widened old from one line to two and left new at the one line it used to replace:

Repaired mutant	What the edit actually did
the prose list loses its exemption	deleted minimum=15, and flipped the flag — two changes under a name promising one, and the floor it fell back to sits on the boundary of that sequence's length, so the test may have gone red for the floor
the prose sequence is read in the wrong direction	deleted starts_at and duplicated minimum=15, — keyword argument repeated, a SyntaxError at import

The second read KILLED, in a batch reporting 0 errored. classify treats a collection error as the invalid mutant, and this module is imported inside the test rather than at module scope, so the SyntaxError arrived as an ordinary assertion failure. A confident row about a question nobody asked — the one thing the tool exists to refuse — produced by the tool, in its own corpus, on the first day. n=1 in the direction an adversary predicted, and it is the honest cost of the model.

refuse_a_mutant_that_cannot_compile now catches the syntax half, before the first write, and it earned itself within the hour: restructuring check_anchors staled two mutants in this increment's own battery and it caught both. ast.parse would not have — it accepts f(a=1, a=2) and compile() does not, which is exactly the case that got through, so a test pins the discrimination rather than leaving the choice of compile looking like a detail. The other half has no guard: an edit that still compiles and changes two things under a name promising one is invisible to everything here. Read the diff a mutant produces, never its anchor.

MEDIUM — a corpus with no inventory can shrink to nothing with every gate green. None of the first draft's checks was a count, and the sanctioned repair for a property that has genuinely gone is to delete its mutant — which makes deletion the cheapest path to green during a refactor. Split a 739-line gate six weeks from now, watch thirteen anchors resolve zero times, delete thirteen blocks: ./check.sh green, the battery reads 28 of 28 killed, and nothing anywhere recorded that it was 41. mutants = N per battery, validated in load_battery, makes deleting a row a two-line edit that says so.

MEDIUM — the argument for keeping them was weaker than it first looked, and saying so changed which argument to make. The first draft argued the claim outlives its proof. It does not: re-deriving 41 mutants against a gate is an afternoon. What is unrecoverable is the reasoning about which mutants were worth writing — and the durable channel for that already existed. A per-mutant table can go in a retro.d/ fragment, and docs/RETROSPECTIVES.md has exactly one, against 93 commits that speak of a mutation pass. The channel existed, worked, and was used once. That is the case for a directory, and it is a better one than there was no channel.

MEDIUM — consolidation is a check that scattering cannot perform. Merging six scratchpad batteries into three files surfaced what no individual battery could: one battery entirely superseded by a later copy of itself, and two mutants byte-identical in file, old, new and kills under different names — the same edit counted twice in a report reading 29 killed.

MEDIUM — a directory nothing reads is the failure this repository keeps recording. A template version that meant different bytes in every commit; a verification table where 61 of 98 test paths did not resolve; a docs/GUIDE.md whose commands no longer ran. So tests/test_batteries.py reads them, in 0.05s. Its cost, stated rather than discovered later: a refactor that moves a mutated line turns ./check.sh red until the battery is repaired — measured at 2 repairs per 81 mutants per week, minutes each. The assertion names re-anchoring before deletion, because deleting a mutant to green a gate is the silent damage this arrangement invites.

MEDIUM — the corrections a review pass made to this file's own claims. Each was a number stated with more confidence than the data carried:

• "the single comment-bearing anchor in the set was the one that went ambiguous" — wrong on both halves. Three of the 81 anchors carry a comment; the one that went ambiguous carries none. One comment-bearing anchor did rot, in the superseded battery. The generalisation drawn from it — that the fragile shape is knowable in advance — went with it.
• The mcp battery was described as spanning six files. It spans seven; check.sh was in neither enumeration.
• The section for tools/fragments.py was labelled 0.28.0; the property shipped in 0.27.1.
• Its header was a comment that had migrated during consolidation, claiming "the two gates" and a control that had gone to the other file. A comment attached to the wrong mutant is silent, and the comments are the reason these files exist.
• A mutant named "ci.yml guards the gate with || true" appended && echo ok, which does not swallow the failure the name promises. The name is the only guard against an anchor that still resolves while the semantics moved, so a name that misdescribes its own edit disarms it.

MEDIUM — "regression asset" is the one thing this cannot be, and the README now says so. report() exits 0 when mutants SURVIVE, deliberately — "It is not a harness failure, so this exits 0 — read the rows." Nothing here changes that, and the committed gate never runs a battery: it checks that strings resolve. Anything CI-shaped built on top needs its own check on the survivor count. And the denominator belongs beside the claim: four batteries, four targets, all under tools/; no module under src/ has one, and no invariant in docs/INVARIANTS.md is covered. The covered files change 3–11 times a month, cli.py 58 and sync.py 40. A coverage index with a hidden denominator is the exact defect this repository keeps catching in its own gates.

LOW — --check-anchors reads the working tree, not HEAD, and the difference is the point. A run refuses an uncommitted target so git checkout <file> stays a safe recovery after a hard kill. Nothing is written in check mode, so inheriting that refusal would have bought nothing and disarmed the check in the one moment it is wanted. Two guards that look alike had different reasons, and copying the reason across would have made the second useless.

LOW — three ways the first draft of a "report every failure" contract reported fewer. The test-file refusal raised inside the per-battery loop, discarding the problems already collected and never reading the batteries after it — which is the run's behaviour, and the thing this mode is defined as not being. The claim set keyed on the battery path as typed, so one battery named twice under two spellings was a false double-claim. And stdout was flushed only on the problem path, so a refusal printed above the summaries it contradicted.

LOW — the flagship test passed when half of what it pins did not happen. The two stale mutants were named gone and also gone, and "gone: …" in stderr is satisfied by the also gone line alone. Reporting only the last problem passed a test whose docstring calls reporting every problem "the difference from a run that matters most". Two fixture names where one is a substring of the other is a silent half-assertion, and the same shape hides anywhere a test asserts on substrings it also chose.

LOW — the test helper that wrote both batteries to the same filename. battery() hard-coded battery.toml, so the first "two batteries" test built the second by calling it again and copying the result — silently overwriting the first. The overlap test passed anyway, because both files claimed the same target either way; the control — two batteries claiming different files — is what exposed it. A control earns its place by failing for a reason the pin cannot.

Extracting CLAUDE.md — four agents each edited correctly, and the set was wrong (20260823 13:52)

HIGH — every defect in this increment was a neighbourhood defect, and none was a content defect. Four agents wrote four destination sections in four files. Each one did its own job correctly: right heading, right anchor, every source fact carried, matching voice, no existing heading renamed. The review then found five defects, and all five were about what the surrounding text already claimed:

Where	What it still said
docs/RELEASING.md:3-5	"the traps … stay in CLAUDE.md" — one had just moved into this file
docs/RELEASING.md:7	"Extracted … 20260801 … Nothing was dropped" — a provenance note now silently covering content added 20260823
docs/INVARIANTS.md:6-9	described the file's two structures; there were now three
docs/README.md:83	the routing table still described the pre-extraction layout
docs/BUILDING.md:168	pointed at CLAUDE.md's text, which is now the compressed version

docs/README.md § Conventions has said audit the neighbourhood, not the diff since it was written. A per-file agent cannot follow that rule, because the neighbourhood is the thing outside its file — and the one file that most needed editing, the routing table, was assigned to nobody and so was the only docs/ page absent from the diff. If work is fanned out per file, something has to own the seams, and it is not any of the writers.

HIGH — the extraction nearly created a second home for a rule, which is the failure it exists to prevent. A section was written into RELEASING.md for naming unbuilt work — and docs/README.md § Conventions already owned that rule in full, including the same date, the same STATUS.md pointer and the same historical-records carve-out. Worse than duplication: the pointers formed a loop in which no file admitted README held it. README said the live names are in CLAUDE.md; the new section said the rule and the table live in CLAUDE.md; CLAUDE.md pointed at the new section. A reader following the pointer from CLAUDE.md would never reach the copy that actually owns the rule, and the two could drift with every command reporting success. The section was deleted and CLAUDE.md now points at README. Before writing a destination, grep for the fact — a relocation into a fact's existing home is a duplication, not a move.

MEDIUM — and checking that duplicate turned up a stale claim nobody was looking for. docs/README.md § Conventions still named the deep release as live unbuilt work. It left that table at 0.26.0 — its final cut — three releases ago, and CLAUDE.md's own table had been correct all along. The duplicate was not merely redundant; the two copies had already drifted, which is the concrete form of the harm "one fact, one home" is stated to prevent.

MEDIUM — four dead links, written while creating pointers whose entire purpose is to be followed. CLAUDE.md sits at the repository root, so a relative ](RELEASING.md#…) resolves to /RELEASING.md, which does not exist. Four were written that way. mkdocs build --strict cannot catch this class: CLAUDE.md is not part of the built site, so its outbound links are checked by nothing. The check that found them is three lines of Python — parse every ](path.md#anchor) out of the file, slug every heading in the target, compare — and it belongs in any future extraction.

LOW — the honest outcome is 220 lines, not 150. The guideline is ~150 and this file is at 220 after everything with a genuine home elsewhere was moved. Going further would mean deleting rules the user set, and rule 6 says explicitly that the guardrail is "the trigger to extract sections and leave pointers — not a hard cap that justifies deleting information." Reporting the shortfall is the correct outcome; compressing past it would have been the defect the rule warns about. What remains is dense by construction: a role rule, an ownership table, a naming table where every row is a breaking change, and four sections of user-set working mode.

The ungated link surface, and three instruments that lied (20260823 14:38)

HIGH — a regex cannot gate Markdown links, and both agents on this repo built one first. The coder's first scan reported 82 broken links inside docs/, a surface mkdocs --strict guarantees clean; it had blanked inline code with re.S, so one unbalanced backtick in a 2 616-backtick file swallowed every heading after it. Mine then reported 18 where the true number was 11 — twice, for two different reasons. Rendering answers the question directly: if a link is inside a code span the parser emits no <a> at all. No stripping, no heuristics, no reimplementation of block parsing. The final instrument is markdown.markdown() plus this repo's own _github_slugify, run through the same uv run --no-project --with-requirements requirements-docs.txt invocation make docs uses, so it cannot drift from the site.

HIGH — a control leg validates only the behaviours it exercises, and a green one is not a clean bill. The coder's control caught its re.S bug. Mine did not catch either of mine: 485 links across docs/, zero broken, green — while the instrument was wrong twice. docs/ contains no links to a directory, so Path.is_file() rejecting docs/graph/ and ../plans/ was never exercised; that produced six false positives. And docs/ has no indented code block containing a link, so treating one as prose was never exercised either. The second false positive produced a wrong edit before it was caught: retro.d/README.md:37 sits in a four-space block and is the README teaching fragment authors what to write — a bare #anchor, which resolves only once the fragment is spliced into docs/RETROSPECTIVES.md. It was repointed at ../docs/RETROSPECTIVES.md before the surrounding prose was read, turning an instructional example into one that teaches the wrong form. Reverted. Read what a line is for before fixing what a tool says about it.

MEDIUM — an adversarial pass that overturns nothing has said nothing. Re-verifying the 20260807 audit's 39 findings against c45ffa8 ran eleven agents plus an adversary per cluster, each prompted to refute every already fixed verdict, since that is the direction that silently drops a verified defect. The adversary overturned zero. Hand-checking all seven afterwards found one false: docs/DESIGN.md:711 asked to drop two phrases and only one had gone — a manifest and a template drift silently still stands while doctor.py prints the drift it denies. The adversary was prompted for exactly that shape, corrected in one place while an equivalent claim survives, and upheld it anyway. A zero from a checker is a claim about the checker. The same pass also corrected two predictions made from reading the code without reading the document: DESIGN.md:715 and KB-UPDATES.md:24 were called more wrong than before on the grounds that pnk upgrade had since shipped, when both had in fact been rewritten and are genuinely fixed.

MEDIUM — mkdocs --strict is a link gate, not a rendering gate. It resolved every link in docs/RETROSPECTIVES.md for weeks while one of its lines rendered as a broken run of <code> tags on the published site. The defect class — a backslash-escaped backtick inside a code span, which CommonMark does not honour — produces valid HTML that is simply not the HTML anyone intended, so nothing that checks structure can see it. Catching it needs a rendering comparison, which is a different check from link resolution and was found only because a link fix forced the line through a parser.

The ungated Markdown surface, and two checkers that were confidently wrong (20260823 14:36)

HIGH — a checker with no control leg reports plausible nonsense, and reads as authoritative. The first draft of tools/markdown_link_gate.py reported 82 broken links inside docs/ — a surface mkdocs build --strict guarantees is clean. The cause was a one-flag mistake: inline code spans were blanked with re.S, so in docs/ROADMAP.md (2616 backticks, zero fences) a single unbalanced backtick swallowed every heading after it. Nothing about the output looked wrong — missing anchors, real filenames, plausible slugs. It was caught only because the scan was run over a corpus whose correct answer is known independently of the checker. Run a new checker against a surface something else already guarantees, before believing anything it says about the surface nothing guarantees.

HIGH — a control leg validates only the behaviours its corpus exercises. A peer wrote an independent scanner over the same surface and got 18 where this one got 12. All six extras were its bug: it resolved targets with Path.is_file(), so every link to a directory failed — docs/graph/, ../plans/, ../retro.d/ — each correct as authored, since GitHub renders a directory link as a listing. Its control leg over docs/ stayed green throughout, because that corpus contains no directory links. So the two lessons are halves of one: a checker with no control reports nonsense, and a green control certifies only what it covered. Both instruments were wrong, in different directions, and each was found by something the other did not do.

HIGH — a false positive gets acted on, so the skips matter as much as the catches. The count was reported as twelve and was eleven. retro.d/README.md:37 sits in a four-space indented code block — neither this checker nor the peer's stripped those — and it is the README's own example teaching fragment authors which anchor form to write. The peer repointed it before reading what the block was for, turning an instructional example into one that teaches the wrong form, then reverted. Nobody argues with a false positive the way they argue with a false negative; they fix it, and the fix damages the document. A link checker's skip list is a correctness surface.

HIGH — when a cheap instrument is suspect, do not argue about it: render, and measure the gap. The right response to "a regex cannot parse Markdown" was not to defend the regex or to take the dependency, but to build the renderer once as an oracle and compare. Python-Markdown over all 114 tracked files emits 894 links; the extractor now finds every one of them and invents zero, the single difference being a bare <https://…> autolink that is external and never resolved anyway. That measurement is what licenses staying stdlib-only — ./check.sh and the CI job need no install and no network — and it is kept as a test that runs wherever the docs toolchain is present and skips with its reason where it is not. The argument was unresolvable; the measurement took one command.

MEDIUM — a false negative is the failure mode a link checker actually has. The second bug in this one forbade newlines in link text, which silently skipped every link whose text wraps — in a repository that wraps prose at 100 columns, and where CLAUDE.md wraps seven of them. A false positive gets argued with; a false negative is a clean bill nobody earned. The control leg cannot see it, because a control proves the absence of false positives only. The gate therefore counts the link syntax it saw and fails on anything it could not parse, rather than skipping it.

MEDIUM — the two documents most worth gating are the two the gate cannot fix. All eleven findings sat in planner-owned files (CHANGELOG.md, plans/, retro.d/README.md), so the gate was buildable and the defects it found were not — by the same ownership rule that makes the split work. Coordination, not escalation: the findings went over as a list with evidence and line numbers, and the fixes landed on the other side of the boundary while this branch built the gate.

LOW — six of the eleven were quotations, and that is a design constraint rather than noise. plans/20260807_2143-docs-audit-findings.md quotes other documents' links verbatim; they are correct relative to docs/ and dead relative to plans/, and correcting the path would falsify the quotation. A gate that flagged them could only be satisfied by corrupting the quote. Code-span a quoted link and it is inert — which the audit file already did in one place before anyone asked.

Landing beside a peer — the overlap tool answers a different question (20260823 14:57)

MEDIUM — a check named for a problem is not a check for that problem. tools/shared_file_overlap.py --fetch --strict reads as the collision check between concurrent sessions, and two sessions ran it that way for a full afternoon. It compares a branch to origin/main. It never looks at another branch, so it cannot see a peer at all: it reported none while a coder held 20260823_1424-markdown-link-gate. It was not wrong — the file intersection genuinely was empty — but it could not have been right, because it was answering "will this merge cleanly" and being read as "is anyone else in these files". The remedy is a comm -12 over the two branches' file lists, which is three lines and answers the actual question.

MEDIUM — the landing order can be forced, and asking your peer will not reveal it. The convention is that complete, gated work lands first and the other rebases, which frames order as something two sessions agree. Here it was determined by a gate. The coder's new tools/markdown_link_gate.py reported 11 broken links and exit non-zero against main, so its own ./check.sh could not go green until the planner's link fixes landed — its branch was unable to land, not merely second in a queue. Neither session knew. The coder was running the gate on its own tree, where it passed; the planner found it by copying the gate out of the peer's branch and running it against main and against its own worktree. Run the peer's gate, in both places. A peer's report of its own state is honest and still blind, because the failing configuration is the one it never runs.

LOW — the same window bites twice, so it is a property of the procedure and not of one tool. Both sessions' link checkers crashed with FileNotFoundError on a file git ls-files lists and the disk does not have. That state is not exotic: docs/RELEASING.md step 1 splices fragments and deletes them, and ./check.sh runs before the commit that removes them from the index. Any gate enumerating tracked files and opening each one meets it on every release. A tracked-but-absent file is a normal mid-release state, and the handling is a skip.

The gate that read everything except its own output (20260823 15:11)

tools/fragments.py --check had run in check.sh since 0.24.0. It parsed every pending fragment — category, slug, emptiness, a leading --- fence — and asserted nothing about the document --apply produces. Three malformed regions sat in CHANGELOG.md through every green run, and the one that names the class was found by reading a release precedent, not by any gate.

The shape is worth naming, because it is not a missing check — it is a checker pointed one step short of the thing it exists to protect. Every other gate here reads rows: mkdocs --strict resolves links, release_order_gate reads sequences, markdown_link_gate reads link targets. A duplicated heading is a property of adjacency, so all of them walk straight past two identical headings in a row. Nothing in this repository had ever read the assembled file at all.

The bug this increment shipped, and what caught it

The first draft anchored its fence detector at column zero. CHANGELOG.md contains no column-zero fence; its only fenced block is indented two spaces inside a bullet. So the scanner skipped nothing in the one document it most needs to read, and a changelog entry demonstrating a Markdown heading would have been refused as document structure.

It was caught by the test written for it, before landing — the false-positive test, not the true-positive one. Both were written in the same pass; only one of them found anything. A passing true-positive says nothing about what the checker is failing to look at. That is the argument for writing the over-firing case even when the under-firing case is the one the item describes: a gate refusing a correct document is the failure that gets acted on, and markdown_link_gate's own docstring records a peer rewriting an example into the wrong form to satisfy a false positive before disbelieving it.

What the false-positive probe found instead

Probing the - rule against Markdown a changelog author might plausibly write turned up a defect that has nothing to do with this gate. The 0.24.0 front-matter residue — --- / category: added / ---, spliced verbatim before check refused that shape — is not inert text. A --- after a text line underlines it into a setext H2, so each residue renders as a heading titled with its own metadata. There are five, not the three this module recorded: three in CHANGELOG.md under ## [0.24.0] and two in docs/RETROSPECTIVES.md, which the original count missed because it looked only at the stream the refusal was written for. The published site carried <h2 id="category-lesson">category: lesson</h2> twice — page, permalinks and search index. All five were removed at 9718aaa, in the same afternoon and by the agent that owns those documents.

Every instrument is green on it. mkdocs build --strict resolves links and a spurious heading is not a broken link; markdown_link_gate reads link targets; and this increment's own gate misses it, because it reads ATX headings and this is setext. Which is the same sentence as the item it closes, one level up: a defect that exists only in the assembled document is invisible to every instrument pointed somewhere else in it. The documents are the planner's to repair and a setext rule is a plan decision, so this increment recorded the measurement and changed no behaviour.

The mutation pass, and the survivor that was not mine

Twelve mutants, and the first run reported ten killed and two survived. Both survivors were real.

Mutant	First run	Why
the front-matter refusal no longer reaches --apply	SURVIVED	0.27.1's row, de-fanged by this increment. The document gate gave --apply a second reason to refuse that same fragment: a spliced --- is not a - list item either. Every assertion in the test stayed true through the mutant
the bullet rule reaches the free-form stream	SURVIVED	my own discriminating row. The fixture was built from ## headings and the bullet rule only ever reads a ###, so the scoping guard could be deleted with the test still green

The first is the one to remember. Adding a guard can silently un-pin a different guard, when both refuse the same input and the test between them asserts only that it was refused. Nothing about the front-matter check changed; a test that had killed its mutant for three releases stopped, because a second correct behaviour started covering for it. A test that cannot say which guard fired pins neither, the moment a second one exists. It now names the refusal.

The second says the ordinary thing about fixtures: a fixture that omits the construct the rule reads exercises nothing. The real docs/RETROSPECTIVES.md carries thirty-four ### headings, every one opening with prose — the fixture now carries one. Re-run: 12 killed, 0 survived.

What the adversarial pass found, and the shape all three share

Five lenses over the diff. Three real defects, none of them in the two rules the item decided — all three in the machinery around them, and every one reproduced end to end (written, fragments deleted, exit 0, follow-up --check green on the wreckage) before it was touched.

Defect	Why it was invisible
_merge_into_section spliced an entry inside a fenced code block	The checker skips fences; the splicer did not. A column-zero ``` holding ### Added was a heading to one and not the other
splice would take a quoted ## [Unreleased] as the insertion point	Same disagreement, one function up — it would bury every future release inside a code block
--apply was not atomic across streams	Refusing on the second stream wrote the first and deleted its fragments, then exited 1 saying "Nothing written, no fragment deleted"

The first two are one sentence: a checker and the code it checks must agree about what a heading is. Before this increment nothing read the document, so the splicer's fence-blindness had no second party and was merely latent — genuinely harmless, for eleven releases. Adding a gate that claims --apply cannot leave the document malformed is what converted a latent asymmetry into a false claim, and that is why closing it belonged in this change rather than in a new item.

The third is the plainer lesson and the one with teeth: a refusal message is an assertion about the world, and it was false. "Nothing written" was true of the stream that failed and false of the run. A release step is one step or none.

The check stopped being read-only, and that is a cost worth naming

The item's stated defect is that --check "asserts nothing about the result of --apply". The first draft read the file on disk — which asserts something about the result of the previous --apply, and nothing about the next one, the one the sentence names. So the narrow draft satisfied a weaker item. --check now validates the assembly the pending fragments would produce.

The evidence is the replay. Both instances the item cites came from a fragment whose body opens with its own ### Fixed. Run against the tree as it stood at each of those commits, the widened check exits 1 at the commit that added the fragment — at 0.6.0, whose duplicate was hand-repaired seven minutes after release, and at 0.28.3, twenty-two days before anyone noticed. Both exited 0 before, and the failure surfaced at release time — which is exactly when docs/RELEASING.md has somebody weighing whether to hand-edit a document to get a release out.

And the cost, which a peer named and this increment had not. --check is no longer a read-only validator: it simulates the write. So --check and --apply must agree about assembly forever, and a disagreement between them would be silent — --check green on an assembly --apply would never produce. That is this increment's own failure mode, reintroduced one layer up. It is not an argument against the widening; it is the argument for making the two the same code rather than two paths that agree: main calls prospective, the function --check validates through, and a test imports the module by path and compares the predicted bytes against what the real subprocess writes. A refactor that gives either path its own assembly turns it red.

Every new guard buys a new invariant. Three of this increment's four defects were the previous instance of that: a checker and a splicer that disagreed about what a heading is, harmless for eleven releases because nothing read the document, and false the moment something did.

The hole that was pinned rather than closed

The decided rule is adjacency, because that is the shape the evidence had. A heading that repeats after intervening entries is not caught, and there is now a test asserting so, naming the plan item. Widening the rule fails that test, which reopens the decision instead of quietly changing the gate. A named hole is a decision; an unnamed one is a discovery waiting to happen.

The assembled document nobody reads, and an extent computed after it was destroyed (20260823 16:18)

HIGH — a defect can exist only in the assembled document, where no instrument is pointed. Five blocks of --- / category: <x> / --- survived a splice into CHANGELOG.md and docs/RETROSPECTIVES.md. They are not inert: --- after a text line is a setext underline, not a thematic break, so each rendered as an H2 titled with its own metadata — and two of them have been served from the published site with live permalink anchors. Every gate was green. mkdocs build --strict resolves links and a spurious heading is not a broken link; tools/markdown_link_gate.py reads link targets; and the new fragments document-gate reads ATX headings, so setext is outside it by construction. Three instruments aimed at these two files, and the defect sat between all of them. The same shape as the item closed hours earlier — fragments.py --check validating what it reads and never what it writes — which is the tell that the class is the assembled document, not any one checker.

MEDIUM — a bottom-up edit destroys the markers a top-down extent depends on. The repair had to delete each residue and re-bullet the entry it preceded, so each entry's extent ran from the residue to the next structural marker. Deleting bottom-up is the standard way to keep earlier line numbers valid — and here it was wrong, because a later residue is an earlier entry's stop marker. The first pass deleted it, the extent then ran on into the following entry, and the result indented a top-level changelog entry into a nested sub-item of its predecessor. Nothing failed. No assertion fired; the residues were gone and the spurious headings were gone, so every check the repair had written for itself said yes. It was caught by reading the rendered structure afterwards and seeing one NESTED! where a BULLET belonged. Compute every extent against the untouched document first, then apply. And the older rule that would also have caught it: read the diff an edit produces, never its anchor.

LOW — the count was five and the record said three, because the record counted one stream. tools/fragments.py's docstring named three residues in CHANGELOG.md. There were also two in docs/RETROSPECTIVES.md, missed because the note was written from the refusal that motivated it, and that refusal is changelog-shaped. A count taken from the thing that prompted the search inherits the search's scope.

A closing note credited a fix that predated the defect, and the defect recurred twice (20260824 00:23)

tools/fragments.py spliced two consecutive ### Fixed headings into CHANGELOG.md at 0.6.0, and again at 0.28.3, twenty-two days later. Between the two, the item describing it was closed. The closing note named a mechanism that could not have fixed it, and the evidence for that was already in git log when the note was written.

When	What	Verified by
20260729 04:20	_merge_into_section introduced (0.3.0)	9f7ed5c, the only commit git log -S'_merge_into_section' returns
20260801 10:53	0.6.0 ships with two ### Fixed — three days later	68ca96b; two headings under ## [0.6.0] at that commit
20260801 11:00	hand-repaired, seven minutes after release	5920f41
20260803 22:21	item closed, crediting _merge_into_section	5bef91f, into plans/open-corrections.md
20260823 04:13	0.28.3 ships the identical defect	2f633a8

The closing row reads: "Fixed with a test (tests/test_fragments.py). _merge_into_section reuses an existing ### Category heading, bounded to the anchor's own section…" — true of that function, and irrelevant to the defect, because the row conflates two mechanisms:

what _merge_into_section fixes   the section's *prose* already carries the heading
what actually kept happening     a *fragment body* opens with its own `### Fixed`,
                                 which `render` then wraps in a second one

Both recurrences are the second mechanism, and it is checkable in one command:

$ git show 68ca96b^:changelog.d/fixed-two-frozen-yaml-behaviours.md | head -1
### Fixed
$ git show 2f633a8^:changelog.d/20260823_0233-fixed-published-versions-row-…md | head -1
### Fixed

Three instruments that reported the previous truth, in one exchange

The same afternoon produced three more, all of the same shape — an instrument reporting the previous truth is indistinguishable from one reporting the current truth, unless you know which it is. None of the three is exotic; all three are the everyday tool used the everyday way.

Instrument	What it actually answers	How it misled
git fetch then reading origin/main	what did I last hear from the remote	Reported a peer's landing as absent. origin/main is a local cache; only git ls-remote origin refs/heads/main asks the remote
git log -S <string> -- <path>	history of this path, stopping at renames	Returned only the rename commit for a plan file, dating a closure 20260804 instead of 20260803. In a repo that renamed every plan to YYYYMMDD_HHMM- retrospectively, every -S in plans/ inherits this; --follow or the pre-rename path is the answer
awk '…{print NR}' fileA fileB	cumulative record number across all files	Reported residues in docs/RETROSPECTIVES.md at lines 9718/9748 — the real lines plus CHANGELOG.md's 3957. FNR is the per-file number, and a line number carrying another file's length in it looks exactly like a line number

The third was mine, in a message asserting a peer's work had not landed. Both halves of that message were wrong, from two different instruments, in the same sentence — and the report read as confident because each number was real. A wrong number that was measured is more persuasive than a guess, which is what makes this class expensive.

The cheap defence is the same in all three: name what the instrument reads before trusting what it says. A cache is not a remote, a path is not a file's history, and NR is not a line number.

The lesson, which is about closing notes rather than about splicing

A wrong closing note is worse than an open item. An open item is a standing invitation to look; a closed one with a plausible mechanism beside it is a reason not to. It spends the evidence — the instance, the diagnosis, the commit that would have shown the dates — and buys nothing, and the next occurrence has to be rediscovered from scratch. This one was: 0.28.3's duplicate was found by reading a release precedent, not by any gate and not by the note.

The check that would have caught it costs one command. Every closing note names a mechanism; a mechanism has a commit; a commit has a date. Is the fix older than the defect it claims to fix? git log -S answers it in seconds, and here it would have returned a date three days on the wrong side. A fix that predates its defect is not a fix — it is a coincidence with a plausible name.

The generalisation, and it is the third instance of one shape in a single afternoon: a green signal is only evidence about the thing it actually measures. A closing note measures whether somebody was satisfied, not whether the defect is gone. A passing true-positive test measures the case it constructs, not what the checker fails to look at. And a test asserting that something was refused stops pinning why the moment a second guard can refuse it too — all three were found here, and all three had been green for weeks.

Two precedents that agree are not a pattern (20260825 00:11)

HIGH — a decision arrived pre-framed by two real precedents, and the framing was refuted by the three it did not mention. The question was whether the owed release was 0.31.0 or 0.30.1. It was handed over as a genuine fork with evidence on both sides: 0.27.0 was MINOR for a new tool file; 0.28.3 was PATCH for extending an existing one; 0.27.0's precedent is a new file, this is not. Every clause of that is true. It is also the two cases out of ten that agree with each other, and the criterion it implies — new file versus existing file — appears nowhere in the governing SemVer rule, which says MINOR = new skill, command, capability, or feature and PATCH = bug fix, doc update, or dependency/component change.

Measured across the ten tooling-only releases (those whose src/ diff is __init__.py alone):

Claim	Verdict
new tool file ⇒ MINOR	refuted. 0.22.2 shipped a brand-new tools/release_order_gate.py as PATCH; 0.27.2 shipped a brand-new tools/wheel_import_gate.py as PATCH, with an ### Added section
an ### Added section ⇒ MINOR	refuted. Seven PATCH releases carry one; 0.18.0 is a MINOR carrying none
existing tool ⇒ PATCH	4 of 4 — and the set is not vacuous, which is the check that mattered

The load-bearing check was whether that last row was an artefact of confounding. If all four had been ordinary bug fixes to a tool, the axis would have been "fixes are PATCH" and would have said nothing about adding a capability to an existing tool. It was not: 0.27.1 added a brand-new refusal to tools/fragments.py — this same file — and was cut PATCH under ### Fixed, and 0.28.1 did the same to release_order_gate.py. The one apparent counterexample, 0.29.0 putting --check-anchors on the existing mutate.py under a MINOR, is confounded: that release's headline was five new battery files and --check-anchors is not among its Added bullets.

The generalisation is about the shape of the offer, not about versioning. A framing assembled from two agreeing examples is the most persuasive form a wrong answer takes here, because no sentence in it is false. The defence is not scepticism about the examples — they survive checking — it is enumerating the population they were drawn from. The test set is the one you did not choose.

MEDIUM — the categorisation was the real question, and the implementer is the worst-placed judge of it. The fragment was named added-…, and changelog.d/README.md makes the filename authoritative for the assembler. But it was chosen by the session that had just built the thing, labelling its own increment — the position with the least distance on whether what it built is a feature. The entry was re-filed under ### Fixed on grounds independent of the version choice, so the reasoning does not close on itself: it closes an item from a corrections plan, its body is a defect narrative, and it is replayed against history to show it would have caught two past defects, which is a fix with a regression test. A PATCH carrying ### Added has happened once in the modern convention (0.27.2); this does not make it twice.

MEDIUM — %an cannot distinguish the agents in this repository, and the commit body can. All 1029 commits are authored Luca Gattoni <…>, committer identical. Measuring those fields and generalising to "git cannot answer who wrote this" was wrong, and it was wrong in the specific way this repository keeps rediscovering: the instrument was named after what it was hoped to answer rather than after what it reads. The trailers are not uniform — 20+ distinct Claude-Session ids and four Co-Authored-By models — and they are in %B, which no identity-field query touches.

git show -s --format='%B' <sha> | grep Claude-Session

That turns "was this closing note written by the session that did the work, or by a later one reconstructing it?" into a checkable question, which is precisely the class behind the two defects closed the day before: a closing note crediting a mechanism three days older than the defect, and a message asserting a peer's work had not landed. Both were written by someone reconstructing, and nothing in the repository could say so.

MEDIUM — a section heading is a status claim, and no gate reads it. A ### item in plans/20260731_1202-open-corrections.md sits under ## Live while its own body, 26 lines down, carries **CLOSED 20260824 00:35**. grep '^## ' — how anyone triages that file — returns Live and stops. It cost a real collision: a coder session read it as live and was minutes from rebuilding a landed increment. The convention it should follow already exists two files away — T6 and T8 in plans/20260804_1016-template-release.md carry · **CLOSED <date>, <disposition>** in the ### heading itself — so the repair is conformance rather than design. Not done yet, deliberately: the item is partial, not closed (the body-rule widening is explicitly untaken), so moving it into ## Closed — recorded so nobody reopens them would file an undecided design question under a heading promising the opposite.

LOW, and it recurred twice in one session — a gate is only a gate when its exit status is what the next command reads. make docs 2>&1 | tail -12 followed by echo "DOCS_EXIT=${PIPESTATUS[0]}" printed DOCS_EXIT= — empty. The value read was the pipeline's, not the build's. Both instances were caught and re-run bare, so what actually gated the release was ./check.sh and make docs redirected to a log with $? read directly. The rule was already written, in CLAUDE.md, and had been read that same session. A rule that is known and still broken twice in four hours is a rule whose failure mode is invisible at the moment of committing it, which is the argument for the redirect-and-read-$? form being the only form ever typed, rather than the one used when it seems to matter.

LOW — a count word and the list beside it drift independently. docs/STATUS.md held two instances of forty-seven and exactly one was wrong: the other sits inside 0.30.0's own published-verification entry, where it was accurate when written. A replace_all would have silently falsified a historical record. Same family as the | head truncation that shipped a broken anchor a day earlier — the edit whose anchor matches more than once is the one that needs the count taken first.

LOW — a peer renaming itself is indistinguishable from a peer dying. This session's name changed from pinakes-d1 to pinakes-7c mid-work. A coder session correctly observed that pinakes-d1 had left ListAgents, correctly inferred a usage-limit kill, and wrote a careful handover describing a release as stranded mid-cut — while it was still moving. Nothing was lost, because the branch had been committed and pushed at every step, which is the actual defence. But ListAgents cannot distinguish gone from renamed, so a peer's absence is evidence about the listing and not about the peer.

LOW, and found by a number that failed to move — tools/markdown_link_gate.py reads git ls-files, so a file that is written but not yet staged is invisible to it. A full ./check.sh run immediately after writing this very fragment reported 84 ungated file(s), 301 link(s) — byte-identical to the run before the fragment existed. Staging it and re-running gave 85. So the ordinary sequence — write a fragment, run the gate, commit — produces a green that says nothing whatsoever about the new file, and the first thing to actually read it is CI, after the push. It is the same sentence as the item this release is about, one directory over: a green signal is only evidence about the thing it actually measures, and here the thing measured is the index rather than the working tree. Nothing was wrong with the fragment; the gate simply had not been shown it. git add before the last gate run, not after — which is also the only ordering under which the gate's answer describes the commit about to be made.

HIGH — a sweep for one direction of error is blind to its mirror, and the mirror is where the confident mistakes live. A peer session swept plans/ for work filed as done that is not. Its single strongest-worded finding was the opposite kind and it was wrong: sidecar.py:125-127 really does accept a uniformly non-string-keyed mapping and silently coerce it, and Decision 19 really is marked shipped in 0.5.0 — but the decision's own sentence reads "Top level only", so a nested mapping is outside its scope by construction, and the behaviour is separately recorded at plans/20260731_1202-open-corrections.md:205 under ## Not to be fixed — recorded so nobody tries. Behaviour filed as never-to-be-done was read as undone.

## Not to be fixed is exactly as load-bearing as ## Closed, and strictly easier to miss: a sweep hunting false closure has no reason to open it, and a vocabulary grep has no token for it. The confidence came from having read the code, which was accurate; what decided the question was the decision's own sentence, which had not been read. Withdrawn by its author on being shown both lines.

HIGH — the filter that decides what gets examined is itself unexamined. That sweep classified each ## section LIVE / CLOSED / MIXED from its heading, and sent only the first two for a semantic read. Nine of twenty files were dropped unread, and at least one wrongly: plans/20260804_1844-decision-parent-child-arity.md's ## The decision was classified CLOSED while holding the file's only outstanding work — a measured ceiling required before G5's gate runs. A section named for a decision is not a section whose decision has been discharged.

Correction, 20260825 13:12 — the lesson holds, the example was wrong. The full re-read (X3) confirmed ## The decision does hold live work, so the finding above stands. But it is not the ceiling: that was measured the same day and is recorded far earlier in this file, under the parent-child arity entry — locate it by that heading, not by a line number — at 53.42 parent-child rows per chunk, 94.7% of stored edges, +113.4% index growth. The live item is requirement 3 — if the ceiling is alarming, the immediate-parent variant is the arm to measure — and no such arm exists (tools/graph_matrix.py LEGS has seven legs and none is one). The sweep that found a heading-level claim was wrong then made a body-level claim that was also wrong, and it survived into a retrospective, which is the same defect one layer down. The number worth carrying forward is therefore 11 of 20 files read, not the finding count; a report's coverage is a property of its filter, and a filter that reads headings inherits every defect that headings have. That is the same shape as the item three paragraphs up — one instance found and generalised — one level higher: the classifier looked at eleven files and silently declined nine.

MEDIUM — the stale-section-under-a-current-banner defect has a second, larger instance. plans/20260729_0256-links-and-graph.md's entire ## Baseline asserts a world that ended at 0.11.0, while the file's own banner at line 8 already says so. A reader going top-down meets "G3 is the next increment to build" (:69), "Do not start any of them" (:82), "One cut remains … it is blocked, not pending" (:103) and "G3 is the increment to pick up" (:124) before reaching the correction. Verified against the tree rather than against the plan: store.py:28 reads SCHEMA_VERSION: Final = 3 and the schema carries both nodes and edges, so G3, G5 and G6 all shipped. None of it matches any status vocabulary; a grep finds nothing here.

And the fourth class is now demonstrated rather than hypothesised — work silently overtaken. :642 states "Its exit criterion is not discharged by any test." It is: tests/test_manifest_compat.py::test_a_manifest_requiring_a_newer_pinakes_names_the_version exists and asserts exactly it. Something closed the requirement and never came back to say so, which no search for a status word can reach — the sentence is a claim about the test suite, and only the test suite can refute it.

The check that was not git — and the mutant that was not a test (20260825 00:14)

HIGH — a substring test stood in for git's ignore semantics, and it was wrong in both directions. pnk init decided whether a KB's .pinakes/ could reach a remote with ".pinakes/" not in gitignore.read_text(). Four failure modes, all reproduced by running rather than by reading:

.gitignore contains	git ignores .pinakes/?	init said
.pinakes (no slash)	yes	⚠️ warns	false positive
.pin*	yes	⚠️ warns	false positive
.pinakes/ then !.pinakes/	no	silent	false negative
#.pinakes/	no	silent	false negative, with a consequence

The two false positives train a user to disregard the warning. The false negatives are worse because they are silent, and the commented-out line is the cheapest way to reach one: comment it out to debug, re-run init, and the ledger and every transcript are tracked with nothing said.

The general shape: a cheap proxy for an authority, where the authority was one subprocess away. git already knows what git ignores — it reads nested ignore files, .git/info/exclude, the user's global excludes and a parent repository's rules, none of which any amount of reading one .gitignore can see. The proxy was not a simplification of the rule; it was a different rule that agreed with it on the common case.

MEDIUM — the measurement trap that makes this easy to get wrong twice. git check-ignore .pinakes returns not ignored for the pattern .pinakes/ whenever the directory does not exist on disk: a trailing slash only matches a path git can already see is a directory. At init time .pinakes/ has never been created, so the obvious probe answers "unprotected" for the very pattern init itself writes. A path inside the directory carries the directory in its own name and answers correctly either way. Anyone re-testing this by hand must mkdir .pinakes first, or a correct pattern will look broken.

MEDIUM — check-ignore exits 0 when any argument is ignored, not when all are. Probing three paths and reading the exit status would have called a .gitignore naming only the ledger full protection, while the index stayed tracked. The matched paths have to be counted.

HIGH — the survivor was the useful half of the mutation run, and it found redundant code rather than a missing test. Seven mutants; six killed. The survivor deleted an explicit startswith("#") comment-skip from the offline fallback, and every test stayed green. Not a coverage gap: "#.pinakes/".rstrip("/") is "#.pinakes", which already fails a whole-line equality test, so the skip never decided anything. The code claimed a care it did not exercise, and the docstring said so in as many words. The loop became one any(...), and the row now mutates the property that is really load-bearing — whole-line equality against ".pinakes" in text, which is the shipped defect. A SURVIVED row is a claim about a pair, and this time the wrong half was the code.

MEDIUM — the tests answered from the developer's machine. check-ignore honours a global core.excludesFile and a system config. A developer with .pinakes/ in their personal global ignore file — a sensible thing to have — turned three of these tests red and made two others pass for a reason the code had nothing to do with (measured: 3 failed, 9 passed). An autouse fixture points GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM at os.devnull. The production code still honours those files, because a global ignore really does protect the ledger; only the tests must not depend on which machine runs them. A test that shells out inherits the whole environment, and the environment is not part of the fixture unless it is made to be.

HIGH — the replacement's first draft was a regression, and only an adversarial pass found it. It probed three named files: index.db, ledger.jsonl, deep/transcript.json. A .gitignore carrying *.db and *.json — an ordinary thing to write — ignores all three, so the new check reported protected while index.db-wal stayed tracked, and in WAL mode that file holds megabytes of verbatim document text. The substring test being replaced warned there. The general shape is worth more than the instance: a check that answers about three filenames cannot answer a question about a directory, and the docstring's careful reasoning that every probe must match made the answer sound complete when the probe set was the incomplete part. A sound argument over an unstated domain — every clause true, the population unexamined, which is the same skeleton as the version framing that opened this session and as the status-word grep that swept plans/. Three instances in one night, in three unrelated places. The probes are now opaque random paths only a directory-covering rule can match.

MEDIUM — a reviewer's fix is a hypothesis exactly like the code it corrects. The suggestion was two opaque probes, flat and nested. Measured against ground truth — git add -A then git ls-files --cached, so the oracle never goes through check-ignore — that set still reports protected for .pinakes/* with !.pinakes/cache, which tracks the entire extraction cache. Four probes, two of them under cache/extract/ and deep/, is what passes all eleven configurations. The review was right about the defect and wrong about the fix, and only measurement separated those two — a distinction that disappears if a reviewer's remedy is applied because the reviewer was right about the problem.

MEDIUM — a hand-rolled fallback disagreed with the thing it was standing in for. Outside a repository the text scan read .pinakes/ followed by !.pinakes/ as protection — silently, on the very input the in-repo tests assert must warn — and warned about /.pinakes/, which git honours. Two definitions of one answer will diverge; the fallback now copies the .gitignore into a throwaway repository and runs the identical probes, and reading the file is reached only when there is no git at all.

MEDIUM — the increment had quietly widened its own trigger, and justified it with something false. The check was made to run whether the .gitignore was adopted or written, on the reasoning that an ancestor repository could negate what init wrote. It cannot: git resolves ignore rules by directory depth, so the file written in the KB wins over any ancestor. What the widening did reach was a path already in the index, where check-ignore reports not-ignored and the warning would tell the user to add a line their file already contains. Narrowed back, and the message that case produces was fixed rather than left: the printed remedy must never instruct an action that would not change the verdict, which the old text could not violate — under a substring test the string's presence was the verdict, so a redundant remedy was impossible. Asking git split "unprotected" from "the line is missing" into two facts and the message went on assuming they were one. A capability arrives with the failure modes it makes reachable, and they are invisible from inside the change that adds it. Scope creep that survives is usually scope creep with a plausible comment attached.

HIGH — two mutation survivors, and both were about the test rather than the code. The first deleted a startswith("#") comment-skip and nothing went red: "#.pinakes/".rstrip("/") already fails whole-line equality, so the skip decided nothing and the docstring claimed a care it did not exercise. The second was len(matched) > 0 surviving against test_ignoring_only_part_of_pinakes_is_not_protection — a test whose name is that assertion. Its input had been a .gitignore naming only the ledger, which discriminated while the probes were named files and stopped discriminating the moment they became opaque: no probe matches, so "any" and "all" agree. A test can stop pinning the property it is named for without changing a character, when what changed is the code it points at.

MEDIUM — the tests answered from the developer's machine. check-ignore honours a global core.excludesFile and a system config. A developer with .pinakes/ in their personal global ignore file — a sensible thing to have — turned three of these tests red and made two others pass for a reason the code had nothing to do with (measured: 3 failed, 9 passed). An autouse fixture points GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM at os.devnull. The production code still honours those files, because a global ignore really does protect the ledger; only the tests must not depend on which machine runs them. A test that shells out inherits the whole environment, and the environment is not part of the fixture unless it is made to be.

MEDIUM — shelling out brought six failure modes that had nothing to do with ignore rules. No timeout, so a wedged core.fsmonitor hook blocks init forever — after pinakes.toml is written, leaving a KB the next pnk init refuses as already a KB. An inherited GIT_DIR answering for an unrelated repository, which git exports to every hook it runs. Exit 1 read as an authoritative not ignored when git had in fact printed a warning. text=True decoding strictly inside subprocess.run, past except OSError. An inherited stdin. And a non-UTF-8 .gitignore aborting with a traceback. Each has a test. The cost of asking an authority is that you inherit its whole process model, not just its answer.

HIGH — 7/7 killed is a ratio with an unstated denominator, and it was read as an answer. The battery was green about the code that had been written, not about the code that should have been; the regression above sat outside every mutant in it, because the mutants were derived from the same understanding that produced the defect. It was reported as "7/7 killed, fragments written" and read as completeness by both the implementer who wrote it and the planner who received it. Neither asked what the battery covered. A green battery is a claim about the mutants someone thought of. The defence is not that someone should have been more careful — it is that a ratio whose denominator is unstated should read as a question rather than as an answer, and this one names the same skeleton as everything else here: a sound argument over an unexamined domain.

This is the first mutation battery under src/. Every other one covers tools/, which tools/batteries/README.md records as a limitation rather than a design — "no module under src/ has one, and no invariant in INVARIANTS is covered." The decision that keeps an index, a spend ledger and a set of verbatim questions off a remote is a reasonable place to stop being able to say that. Updating that sentence surfaced that it was already stale: it said "Four batteries, four primary targets" with five on disk. The paragraph that exists to prevent a hidden denominator had one.

Scope, deliberately held. Only the existing check's correctness changed — and its trigger, once widened, was put back. Whether pnk doctor should carry a recurring check, and whether it reports a warning or a note, is a decision reserved to the planner and the user, and this measurement changes it rather than settling it: a detector that misfires on .pinakes and .pin* is a poor candidate for firing on every doctor run, and a correct one is a much better one.

The diagnostic that printed nothing, and a review of a moving target (20260825 00:53)

HIGH — the fix for "a remedy that changes nothing" was an instruction that shows nothing. The message printed git check-ignore -v .pinakes/index.db. That command reports only positively matched paths, and the branch printing it exists precisely because the path is not matched, so it emitted nothing at all. One defect replaced by the same defect one step along, inside the commit whose subject line forbade the first.

The test is why it shipped, and the shape is general. It asserted the message contained the string git check-ignore -v. Asserting that a message mentions a command is not a test that the command helps. The replacement extracts the suggested line out of the printed warning, runs it, and asserts both that it produces output and that the output contains the negation the user has to remove. A test that reads a string can only ever verify the string.

MEDIUM — and the sentence describing the replacement overclaimed in turn, which is the fourth instance today. The proposed docs line said the command shows "the ! that defeats the rule". It shows the lines that name .pinakes — so when the defeating pattern is spelled some other way (!.pin*, !.pi?akes/, a broader !*, all of which reach the same branch and were verified to) the line that actually causes the problem is not printed. Caught by a reviewer running a case the author had not chosen, which is what caught the three before it. The typical case described as though it were the only case — the same shape as the probe set, the version framing and the status-word grep, in four unrelated places in one night.

Left deliberately, and named here so it is not rediscovered as a defect. A diagnostic that always shows the conflicting rule is a larger design question — cat -n .gitignore? a check-ignore -v against a path git does match? — and it belongs with the rest of the reserved gitignore work rather than in a release being held. The shipped command prints something real in every case the branch fires, which is the improvement; it is not guaranteed to print the cause.

MEDIUM — a review is a measurement with a timestamp. The adversarial pass took long enough that the branch moved five times under it, and reviewers measured different commits: several findings labelled critical were already fixed when they were written, and the adjudicator had to re-run two survivor claims to find both already killed. Its own caution is the lesson — a long review of a moving branch measures a tree nobody will ever ship — and the response is not to review faster but to re-verify every finding against the tree as it stands before acting on its severity.

MEDIUM — the same test-hygiene defect, one layer further out. An earlier fix in this line pointed GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM at os.devnull so a developer's global excludes could not decide the answer. It closed the config leak and left the ancestor repository open: with pytest --basetemp inside a checkout that ignores .pinakes/, three tests failed and two of them are named outside a repository while running inside one. The fixture covered the half that had been thought of — the same sentence as the mutation battery two hours earlier, about a different artefact.

MEDIUM — two things were in one list because they share a prefix. GIT_DIR and GIT_WORK_TREE redirect git at a different repository and must never be honoured here. GIT_CEILING_DIRECTORIES only limits upward discovery, so honouring it makes this check agree with the user's own git add and scrubbing it made the two disagree. Naming a constant for what its members look like rather than for what they do is how the wrong one gets in.

And a note on the review arrangement itself, because it worked in both directions. The planner amended a docs/CLI.md sentence specifically to stop it overclaiming what the code does — and approved the correction by comparing the prose to the implementer's description of the code rather than running the command. A description-versus-code check performed entirely on descriptions. Caught within the hour by an independent pass, which is the argument for having one.

Planning the exposure question — a check that cannot see the case it is for (20260825 07:55)

HIGH — the 0.30.2 gitignore detector reports protected for a repository that is committing the user's verbatim questions, and the reason is the fix that made it correct. The detector probes opaque random tokens under .pinakes/, chosen on 20260825 so that no realistic pattern could target them — which had been a real regression when three named files were covered by an ordinary *.db/*.json pair. But a path that does not exist cannot be in the index, and git check-ignore consults the index: on a tracked path it answers not ignored, and only --no-index answers ignored. So the probes can ask exactly one question — would a new file here be ignored? — and never the other one — is anything in there tracked right now?

Measured, not reasoned: a repository with .pinakes/{deep/op1.json,ledger.jsonl,index.db} committed, then a correct .gitignore added, still lists all three in git ls-files, still commits the edited transcript under git commit -a — and _ignored_by_git returns True.

The general shape, and it is the fourth instance in two days: a fix that removes a false positive can install a permanent blind spot, and nothing reports the trade. The opacity that made the detector right about patterns is exactly what makes it blind to the index. Neither property was written down, so the boundary was invisible to review.

MEDIUM — a correct .gitignore does not untrack an already-tracked file, and Pinakes says so nowhere. grep -rniE "rm --cached|untrack|already[ -]tracked|already in the index" over *.py *.md *.toml *.sh *.yaml returns only tools/template_drift_gate.py, tools/mutate.py and a retrospective about the wheel. This matters because init.py already states the rule it breaks — "the printed remedy must never instruct an action that would not change the verdict" — and for a tracked-and-unignored KB, "add this line" flips the verdict to protected while changing nothing about the exposure.

HIGH — a section heading is a status claim, and it produced the same collision twice in two days. plans/20260731_1202-open-corrections.md carries an item under ## Live whose body says CLOSED 20260824 00:35 twenty-six lines below its own **Decided.** line. On 20260824 a coder session read it as live and was about to rebuild a landed increment. On 20260825 a second, freshly-cleared coder session did the identical thing and offered to build it again. Both times the save was a peer message; neither time was it a gate. grep '^## ' returns Live and stops.

MEDIUM — sed -i '' "s/X\bY/" on macOS silently matches nothing, and a success message computed before the operation will not notice. Renumbering four colliding decision IDs, the loop printed "7 line(s) rewritten" — which was grep's pre-count, not sed's result. BSD sed does not implement \b. The file was unchanged and the log said otherwise. Reading the file back afterwards is what caught it; perl -pi -e is what fixed it. A count taken before an operation describes the input, never the outcome.

HIGH — the proposed fix for that blind spot failed in the blind spot's own direction, and two sessions caught it only by running it. The first draft specified git ls-files -- .pinakes, with a relative pathspec. Run from a subdirectory of a repository that is tracking three files under .pinakes/, it returns rc=0 with zero rows — and a genuinely clean repository returns rc=0 with zero rows as well. pnk init can be run from anywhere, so the two states are reachable, identical at runtime, and the wrong one reads as safe. The absolute pathspec returns three rows from the same directory. A check that answers clean because it was asked the wrong question is exactly what the increment it corrects existed to remove, so the fix has to make the pathspec structural — build it from the resolved root, and test it from a subdirectory, because the failing case is invisible from the root.

MEDIUM — a remedy whose steps are right and whose order is wrong fails silently. In a repository with no ignore line, git rm -r --cached .pinakes followed by an ordinary git add -A puts the file straight back in the index (measured: one file, immediately). The ignore line has to come first. The reverse order looks like it worked and reverts on the user's next add.

HIGH — knowing a hazard is not what avoids it; running the command bare is. The boundary table above was first read through ... 2>&1 | head -3, which reported the outside-a-repository case as rc=0 instead of 128 — the pipe's status, not git's. This repository had already recorded that exact hazard, twice, and the session that walked into it had read the entry that morning, within the hour. The entry is therefore restated in its stronger form: the rule does not fire on recognition. The only thing that separates a real exit status from a plausible one is running the gate bare and reading $?.

HIGH — the plan wrote a constraint from a measurement taken in the wrong place. § X1 required the tracked-check's pathspec to be absolute and prescribed the test that would prove it: "run it from a subdirectory — the case that fails is invisible from the root." The property is real. The prescribed test cannot fail, because _ask_git passes cwd=root to subprocess.run (init.py:94) and every call site passes root — so the process cwd never reaches git and a relative .pinakes resolves against root regardless. The reproduction behind the constraint was a shell fact, run with cwd=<subdir>, a state this code never produces. It was promoted to a claim about a code path without anyone opening that code path's cwd.

The domain nobody examined was the instrument. Every other item here is a claim whose population went unstated; this one is a test whose ability to observe anything went unstated. The argument for the property was sound and stays sound — the absolute form survives as defence. What was wrong was the belief that a named test would pin it. A prescribed test is a claim about a failing test, and whoever prescribes it owes the same evidence as whoever writes it. The increment's own retrospective carries the mutation run that settled it, and the finding is recorded there rather than here.

MEDIUM — the planner produced the day's own lesson while writing the plan about it. I told a coder that 36 unrowed test_fragments.py tests were a gap to fill. It counted the population instead: 2023 tests defined, 1106 named in docs/VERIFICATION.md, 917 with no row — and the document's own scope section says it maps promises to tests, naming six modules that predate the table. A gap of 917 cannot all be holes. The coder declined, correctly, and cited a peer message is coordination, never permission. Two honest counts of that same population differed by twelve (1106/917 against 1094/929) purely because the matching rule differed, and neither number carries its rule.

LOW — zsh does not word-split unquoted parameter expansions, so set -- $pair inside a for loop passes one argument, not two. The anchor assertion caught it before any write. Worth keeping only because the assertion is the reason: the guard that looked like ceremony was the thing that turned a silent no-op into a stop.

X1 — building the tracked-KB check, and an instrument that could not fail (20260825 08:16)

HIGH — the specification asked for a test that cannot fail, and only writing it revealed that. The plan bound the implementation to three constraints, the first being "the function takes the resolved KB root and builds the absolute pathspec itself… its test must run it from a subdirectory — the case that fails is invisible from the root." That constraint was written from a real, reproduced measurement: git ls-files -- .pinakes run from a subdirectory exits 0 with no rows against a repository tracking three files. Two sessions reproduced it independently.

It is still not reachable through this code, and the test it prescribes proves nothing. _ask_git passes cwd=root to subprocess.run, so the process cwd never reaches git: a bare .pinakes resolves against root and returns the identical verdict from a subdirectory, and with a relative root. Measured rather than argued — with the pathspec mutated to ".pinakes", 68 of the 69 tests in tests/test_init.py pass. A behavioural subdirectory test would have been green against the exact form it existed to forbid.

So the constraint moved to where it can fail. The pathspec is built by its own function and the test asserts the pathspec — :(literal) prefix, absolute, ending in .pinakes — which is the one test the relative form kills. The absolute-and-literal form stays, because correctness of the relative form depends entirely on _ask_git keeping cwd=root, a coupling no caller can see, and because a KB root may legally contain * or [. It is defensive rather than a bug fix, and the docstring says so instead of implying coverage it does not have.

The general form is worth more than the instance. "Pinned by test X" is a claim about a failing test was already the rule here. This adds the case the rule does not obviously cover: a constraint can name a real property, be derived from a real measurement, and still prescribe an instrument that cannot observe it. The argument was sound; the domain it quantified over — this call path, rather than the shell where the measurement was taken — was never examined. Neither session checked it. It surfaced only because the implementer refused to write test_…_from_a_subdirectory without first watching it go red.

A second gap the same discipline found. pinakes_tracked is a bool, so None and False collapse into one field and no test through init can tell them apart — an implementation returning False outside a repository (exit 128, fatal: on stderr) passed every test written. The constraint that unknown must never read as clean is now asserted against _tracked_by_git directly. A field's type can erase a distinction the specification depends on.

And the pipe hazard fired again, on the person who had just written it up. The full gate was run as ./check.sh > log 2>&1; echo "EXIT=$?" | tee -a log; tail -20 log. The harness reported exit code 0; the gate had exited 1, on two reportPrivateUsage errors. The $? was captured correctly and then discarded by the commands after it — a gate is only a gate when its exit status is what the next command reads, missed by someone who had recorded that sentence twice the same evening. Recognition is not the mechanism. Writing the status to its own file, and reading that file, is. Three instances of one class in one evening, each caught by running something rather than by reading it — and this one caught only because the log was read after the notification said the opposite.

MEDIUM — and the fourth instance of this increment's own shape: a CI watcher that was green for a reason nobody had checked. Its exit condition was "all visible runs are complete". That is unsound in general — with staggered run creation it exits before the remaining runs exist and reports success for a run that was never created. Here it gave the right answer, because exactly one run was ever going to be created: docs is path-filtered to docs/**, mkdocs.yml, mkdocs_hooks.py, requirements-docs.txt and its own workflow, and this increment touched none of them. But that fact was established after the watcher was armed, not before. The instrument was sound by luck over a domain its author had not examined — written an hour after the same author wrote that the instrument is the part nobody checks.

A second watcher, on the planner's side, failed the same way and was not saved by luck: gh run list --commit <short-sha> returns empty, so an until [ ... = "completed" ] loop could never terminate and would have polled in silence until timeout. --commit requires the full sha. Silence from a watcher is indistinguishable from still running, which is why both were rewritten to emit on every terminal non-success state rather than to wait for a success that may never arrive.

Design review passes 1–7 (pre-implementation)

Seven adversarial passes over DESIGN.md before any code was written — 58 findings resolved (11 HIGH, 32 MEDIUM, 15 LOW). Moved here 20260728 16:40 from DESIGN.md §10, so that all project history lives in one file and the design document is specification only.

The headline lesson, visible only across the whole sequence: passes 2 and 4 fixed defects that passes 1 and 3 had themselves introduced. That is the argument for looping a review rather than running it once — and the same argument the per-increment retrospectives above rest on.

Pass 1 — 6 HIGH, 15 MEDIUM, 5 LOW resolved. HIGH: sqlite-vec wrongly described as an ANN index (verified false upstream — §3.1 rewritten and the tiering rationale corrected to bounded memory); reverse cross-KB links specified against the other KB's gitignored index, impossible after clone (now scans committed sidecars, §6.2); pnk:// URIs used local aliases, breaking on share (now KB ULIDs, §2.2); rename/orphan/duplicate-ID sync semantics unspecified (§6.4 added); per-operation budget cap claimed a guarantee it could not deliver post-hoc (now pre-call reservation, §5); v0.1 omitted pnk sync, pnk doctor and hooks though every other section depended on them (§8). MEDIUM: MCP tools renamed kb_* → pinakes_* for namespace safety; multi-hop scope stated as single-KB in v0.1; "no network" qualified against first-use model download and weights moved to the shared HF cache; embedding storage described two ways, unified on a float32 BLOB; confidence signal recast as calibrated with term-coverage demoted to a tiebreak; token limits validated against the model's own tokenizer; template versioning decoupled from package version; install line corrected to uvx --from "pinakes[st]" pnk with core-only behaviour defined; sync partial-failure semantics and failures table added; WAL/read-only/lock concurrency policy added (§6.5); orphaned-sidecar deletion made opt-in; paths fixed as KB-root-relative; index migration policy stated as rebuild-only; ledger privacy and append atomicity specified; pnk build unified into pnk sync --rebuild. LOW: budget window timezone; FTS5 external-content triggers; RRF k=60; latency claim replaced with a measured 2.25 ms at 50k×384; golden-set size and coverage targets.

Pass 2 — 1 HIGH, 7 MEDIUM, 5 LOW resolved. Several were introduced by pass 1's fixes, which is the argument for looping rather than reviewing once. HIGH: the --rebuild swap added in pass 1 renamed a WAL-mode database without checkpointing, leaving a stale -wal beside a new index.db — a corrupt read. Now checkpoint-truncate, clean close, then rename (§6.5). MEDIUM: "operation" undefined for the per-op cap, letting an N-step --deep loop spend N× the limit (§5); §4.2 referenced calibration thresholds the manifest had no field for (§2.1); the links schema could not represent a reverse link, whose source doc lives in another KB (src_kb_id + origin enum added); §3.1 presented three tiers as if all shipped, with v0.1 behaviour above 50k chunks undefined; duplicate-content files made hash-based rename detection ambiguous with no tie-break (§6.4); MCP server boundary and prompt-injection posture unstated (§4.7); FTS5 / enable_load_extension treated as universally available — verified present on uv-managed CPython 3.13, now probed by pnk doctor. LOW: a single top_k covered three different cut-offs (split into candidates_per_source / fusion_top_k / final_k); max_tokens sat under [embedding] though §4.6 treats it as chunking; [[links.kb]] present from v0.1 but unused until v0.3, now labelled; what publishing a KB repo exposes; reverse-link origin provenance.

Pass 3 — 1 HIGH, 3 MEDIUM, 4 LOW resolved. HIGH: §6.3 said --rebuild "discards .pinakes/", which would delete ledger.jsonl — the spend history §5's rolling budget is computed from. A routine maintenance command would have silently reset the budget. Rebuild now replaces index.db only; cache/ clearing is opt-in. MEDIUM: the server's staleness check read meta.build_id through its own open connection, which after a rename still points at the old inode and would report the old id forever — replaced with a per-request stat() on the path (§6.5); per_operation_eur served as both the confirm threshold and the hard ceiling, making the confirmation prompt unreachable (split into confirm_above_eur + per_operation_eur, §2.1/§5); §6.4 framed pairing as ordered per-file rules, but rename and duplicate detection require the whole before/after set — restated as an explicit two-phase algorithm. LOW: v0.1's pnk doctor list omitted the environment probe §3.1 depends on, and pnk serve was referenced in §4.5 but absent from the release list; "aliases … never stored" contradicted the manifest that stores them (clarified: never inside a URI); the reservation formula reused the name max_tokens, which [chunking] already claims; "not in v0.1 but present from day one" reworded.

Pass 4 — 2 MEDIUM resolved, both self-inflicted by pass 3. The rebuild bullet still ended "readers detect the new build_id and reopen" — directly contradicting the stat()-based detection added three lines above it in the same pass (§6.5, now reconciled; build_id is retained for provenance only). And pnk://self/… was left unexpanded, so a sidecar copied into another KB would silently retarget its link at the new KB — self is now expanded to the owning KB's ULID on write, like every other alias (§2.2). A grep sweep confirmed no stale kb_* tool names, pnk build, or bare top_k references survive outside the log.

Pass 5 — 0 findings. Verified by re-reading §§1–10 in full and grepping for every identifier renamed across passes 1–4. No section contradicts another; every external claim (sqlite-vec is exhaustive not ANN, FTS5 + extension loading on uv-managed CPython 3.13, pinakes free on PyPI, 2.25 ms at 50k×384) was measured or fetched in-session rather than recalled; every locked constraint is honoured; every capability in §1 maps to a release in §8. Review complete.

Pass 6 (20260725 09:28, implementation-readiness review) — 2 HIGH, 2 MEDIUM, 1 LOW resolved; the two product calls were decided by the user, not the review. HIGH: the reranker was simultaneously a v0.1 default (rerank = "local" in §2.1, "on by default" in §4.1, its scores the substrate of §4.2's confidence signal, "rerank precision" in §7's v0.1 CI) and a v0.5 deliverable in §8 — a freshly-inited KB would have defaulted to a stage that didn't exist, and v0.1 would have shipped with no defined confidence signal. Resolved: the reranker ships in v0.1; default BAAI/bge-reranker-base (user decision — same id on both backends beats the smaller ms-marco model's provider-specific ids), a [rerank] manifest block mirroring [embedding], fitted_for added to [retrieval.confidence], and a CI HF_HOME cache so ~1.4GB of weights download per cache key, not per job (§2.1, §4.5, §8). And §8's v0.1 had no CLI query surface at all — pnk search existed in §4.2's escalation story, the CLI stub and the README, but not in the release that claims "end to end". Added explicitly (§8). MEDIUM: the post-commit hook wrote sidecars, dirtying the tree it had just committed — every document commit would trail an untracked .pnk.yaml forever. Resolved with a three-hook split: pre-commit mints and stages sidecars for staged documents only, post-commit/post-merge touch the index only (§6.3). And a stale sync.lock from a killed sync silently disabled hook-driven freshness forever ("a second sync exits immediately" had no liveness story). Resolved: the lock records pid/host/start-time; dead-pid locks are reclaimed with a warning, cross-host locks refuse with --force-unlock as the human path, pnk doctor reports held locks (§6.5). LOW: the sidecar's content_hash duplicated documents.content_hash, was read by nothing, and guaranteed a two-file diff on every document edit while going stale whenever sync hadn't run — dropped from the sidecar (user decision); change detection is index-only, stated in §2.2.

Pass 7 (20260725 09:52, surfaced while adversarially reviewing plans/20260725_1317-v0.1.md — the implementation plan's review loop reads the design fresh each pass, which is how these escaped passes 1–6). HIGH: §4.5 claimed model weights go to the shared HF cache on both backends — false for fastembed, which defaults to $TMPDIR/fastembed_cache (verified upstream): CI's HF_HOME cache would never hit and pnk doctor's weights check would probe the wrong directory. The fastembed backend now passes an explicit cache dir under HF_HOME, making the claim true by construction. MEDIUM: a sidecar-only edit (tags/title/links changed, document untouched) fell through §6.4's "path and hash unchanged → Skip" and was never re-indexed — documents.sidecar_hash added (§3) and the sidecar-only change class stated (§6.4); soft delete left chunks and embeddings searchable — removal on soft delete stated, identity row retained (§6.4); rename+edit in one sync had both the adoption and deletion rows firing for the same ID with no stated winner — sidecar adoption now wins, no soft delete emitted, and the sidecar-didn't-travel case is reported at sync time (§6.4).