Updating an existing KB — design note¶

Status: mostly proposal. Decided 20260728 18:39. Its minimum — the requires_pinakes pre-pass (§4, §7) — shipped in 0.6.0, as G4 of plans/20260729_0256-links-and-graph.md; every field rule is in MANIFEST.md and the reasoning in DESIGN §2.1. The template-drift gate (§6) shipped in 0.17.0, along with the version bump §9 puts ahead of it, so the doctor check this note calls dead now fires. 0.18.0 made pnk doctor report how far a template has drifted — a computed line count, not a diff — and pnk upgrade, which prints the lines themselves, is merged. What remains a proposal is --apply: adopting a change into a manifest. Template-release work either way. STATUS.md is the authority on what exists.

This note answers one question the build plans had not asked: when Pinakes changes, what happens to a KB somebody already has?

1. The problem¶

A KB is two things with opposite update stories. .pinakes/ is derived — throw it away and rebuild, free and deterministic. pinakes.toml, docs/ and the sidecars are committed, hand- edited, and belong to the user; nothing may rewrite them casually.

The design handles derived state well and committed state not at all. Every mechanism below exists for the first category. For the second, detection shipped in 0.17.0, pnk upgrade in 0.19.0, and adoption — --apply — is merged (see the header above). This section was written when none of them existed. What adoption does not reach is still the honest gap: it writes pinakes.toml and nothing else, so a template's README.md and its starter eval/questions.yaml remain yours to refresh by hand, and a KB recording a version whose content was never archived has no baseline to adopt against.

2. The four drift axes¶

#	What drifts	Mechanism today	State
1	Index schema	`schema_version` mismatch → refuse to open, name `pnk sync --rebuild`. No migrations, by design (`store.py:205`)	✅ shipped
2	Embedding model	Index built by another model/revision → queries refuse rather than return garbage	✅ shipped
3	PDF extractor	Fingerprint mismatch → free backend refuses; paid marks `stale_extraction` and warns	✅ shipped (I5)
4	Manifest + template	`[kb] requires_pinakes`: a version floor read in a pre-pass, so a refusal can name the version needed (G4, shipped 0.6.0). Detecting template drift shipped in 0.17.0 — a bumped `notes@1.1`, a CI gate that makes the bump impossible to forget, and a `pnk doctor` WARN that now fires. Adopting it landed in T4: `pnk upgrade --apply` writes the hunks that fit into the user's manifest, after printing every one of them, and refuses the whole run if any conflicts	● closed for `pinakes.toml` — a template's other three files are still the user's to refresh by hand

Axes 1–3 share a shape: detect, refuse, and point at a free remedy. That works because the remedy is always "rebuild derived state", which costs nothing and destroys nothing.

Axis 4 cannot borrow it. The remedy is "change a file the user owns", which is neither free nor safe by default — so the mechanism has to be different in kind, not merely deferred.

3. The gap is live, not theoretical¶

Two live cases on main today, and one this note got wrong:

The PDF explanation, not the glob. pnk init stamps include = ["**/*.md", "**/*.txt"] and still does — the template deliberately leaves **/*.pdf out, with a comment above include telling the reader to add it. That comment shipped in 0.2.2, and a template change reaches new KBs only, so it appears in no KB created before it: their owners get 0 indexed on a PDF with nothing in their own manifest explaining why. This note originally claimed the glob itself was added to the template and that existing KBs were left "PDF-blind permanently". That never happened — the glob was never added, so the drift is in the explanation, not the behaviour.
I6a's budget keys. daily_eur and max_price_age_days landed with defaults, so existing KBs keep working — but a KB whose owner sets one is then unreadable by any earlier Pinakes (§4).
~~The one drift signal that exists does not fire.~~ Closed in 0.17.0. doctor._template (doctor.py:205, comparing at :219) compares declared version strings only — which was worthless while notes declared version = "1.0" through eleven releases of changing content. notes is now 1.1 and §6's gate makes the next bump impossible to forget, so the check discriminates. The comparison is no longer a version string. 0.18.0 renders both archived versions and reports how many lines separate them; pnk upgrade (0.19.0) prints the lines themselves, and pnk upgrade --apply (0.20.0) adopts them — the hunks that fit, after printing every one of them, refusing the whole run if any conflicts. What is still missing is a baseline, not adoption: a KB recording notes@1.0 has no archived content to compare against, because 1.0 denoted eleven different template contents. A KB stamped from notes@1.1 onward is compared automatically.

4. Compatibility posture¶

Verified behaviour when a file contains a key the running Pinakes does not know:

File	Behaviour	Direction
Sidecar	Preserved verbatim under `extra` and written back (`sidecar.py:35,106`)	Forward-compatible
Manifest	Hard error (`_toml.py:184`)	Forward-incompatible
Index	`found != str(SCHEMA_VERSION)` (`store.py:205`)	Refuses both directions

Demonstrated against main (20260728 18:39) with a hypothetical future key:

REFUSED: [budget]: unknown key(s): `weekly_eur`
REMEDY : Unknown keys are rejected rather than ignored — a typo would otherwise leave you
         with default behaviour while believing you had configured something.

The refusal is correct; the diagnosis is wrong. The user's problem is an out-of-date Pinakes, and the message tells them they made a spelling mistake.

Decided¶

Downgrade is unsupported. A KB may be opened by the Pinakes that wrote it, or newer. An older one refuses, naming the version required. This makes explicit what store.py already does.
Strictness is unchanged. Unknown keys stay a hard error — the typo protection is worth more than graceful degradation, and cross-version sharing is not a goal.
[kb] gains requires_pinakes — built in G4 — e.g. ">=0.3", so the refusal states the remedy instead of the symptom. A floor only: >= is the sole operator, since the posture above has no ceiling to express. Absence means no floor declared:

error: this KB requires pinakes >= 0.3 (this build is 0.2.1)

The cost, accepted: it couples the KB format to package version numbers, which the project deliberately avoided for templates. The reasoning that makes it acceptable is that the template decoupling exists so a package upgrade never silently changes a KB's blueprint — a compatibility floor changes no blueprint. An actionable error was judged worth the coupling.

5. `pnk upgrade`¶

Diffs the KB's recorded template version against the installed one.

Does:

print the diff, always, before doing anything;
with --apply, write the changes that fit into pinakes.toml, preserving comments. Built in the template release, and not the way this section proposed: the hunks are applied as text, so comments survive because nothing ever parses them away, and no tomlkit was added to core. F2 is why — no template change has ever added or removed a key, so the unit of drift is the rendered text and a key-level writer would have been the wrong tool as well as the costlier one. One key-level write remains, [kb] template, and it is a bounded textual rewrite that refuses rather than guessing;
never update requires_pinakes — corrected 20260808, and the sentence it replaces said the opposite. D-11 (taken 20260804): nothing in Pinakes writes that key, in any direction. Writing it would make a KB unreadable to every build before 0.6.0 in order to record that it adopted a comment, and no version floor could be computed honestly anyway: nothing here maps a manifest key to the release that introduced it. --apply names the keys it introduced and leaves the floor to the user.

Must never:

touch anything under docs/ — not a document, not a sidecar;
renumber or regenerate any ULID;
re-chunk, re-embed or re-extract as a side effect. A changed include glob means new documents exist to index, and that is pnk sync's job, invoked separately and explicitly;
apply anything without --apply, or without having printed it first.

The precedent it follows is pnk doctor --prune: print every path, then act only on request.

6. Detecting template drift¶

Shipped in 0.17.0 as tools/template_drift_gate.py — seven legs, run by check.sh and by its own template-drift CI job. It hashes the template directory and fails when content changed without a version bump. template.toml's version stays the human-readable contract; the hash is what makes the contract enforceable.

Scope: everything under templates/<name>/ except three exclusions — anything under a _versions/ component (the archive is not live content, and hashing it would make the live hash depend on its own history), template.toml itself (it carries the version being compared), and anything git ignores (asked of git rather than kept as a list of junk filenames, because that list is never finished). Inverting the list is deliberate — an explicit include-list would need extending whenever a template gains a consumed file, which is the same rule-without-a-gate failure this gate exists to prevent. Fail-safe: a new file is covered by default.

README.md is in scope, not exempt. This note originally exempted it as prose. It is not prose: copy_extras copies it into every KB, so it is a consumed file, and exempting it would let the copy in a user's KB drift with no bump to say so. Held by tests/test_template_drift.py::test_editing_the_template_readme_fails_the_gate.

One key of template.toml is folded back in: files. The exclusion above was scoped to what that file held when it was written — a name, a version and a description, none of which change what a KB is stamped with. files (below) does exactly that, so leaving it out would let a template change the set of files written into every new KB with no version bump: the property this gate exists to enforce, defeated by a key sitting one file to the side of it. Only the list is hashed — name, version and description stay out, so a version bumped with no content change remains detectable. An absent key contributes nothing, so every hash published before this rule is unchanged and no ledger row needed migrating.

6.1 What a template declares — `files`¶

template.toml may list the files the template writes into a new KB:

name        = "notes"
version     = "1.1"
description = "Plain Markdown notes: the smallest useful knowledge base."
files       = ["README.md", "eval/questions.yaml"]

An absent files means ["README.md", "eval/questions.yaml"] — never none. That is what copy_extras copied before the key existed, and every template shipped so far declares nothing, so absent has to keep meaning what it always did. An explicit files = [] is a different statement and copies nothing.

Entries are relative to the KB root, and each is refused — before any file is written, so a bad declaration never leaves a half-stamped KB — when it:

Refusal	Why
names `_versions` as a path component	An archived version is the frozen record of what a reference meant. Copying one into a KB stamps content from a version nobody released under the name of one they did. Containment cannot catch this: the path lands inside the KB
is absolute, or empty	Entries are relative to the KB root
lands outside the KB	`../../evil.md`, and a symlinked directory in the target — which is what adopting an existing directory can present
reads outside the template	A symlinked directory in the template tree lands its destination inside the KB while its source escapes, copying a file the template does not own into a KB that is then committed and published

The last two are separate layers and neither covers the other: an escaping destination reads from inside the template, and an escaping source writes to inside the target.

A template is packaged data, which is not the same as trusted data — pnk init --template names whatever is installed, and that can have arrived from anywhere.

The gate runs at commit time, so it produces no warnings in any user's KB. Its history leg needs a full clone, and says so when it has been skipped — a skip is not a pass.

7. Implementation constraints¶

requires_pinakes must be read in a pre-pass, before strict validation. Otherwise the parse dies on the first unknown key and the good error never fires — the field would be unreachable in exactly the case it exists for. This is the one non-negotiable ordering requirement.
Its absence means compatible, never an error. Every KB in existence lacks the field; a missing floor is "no floor known", not a refusal.
The template-drift gate needs no runtime support in a user's KB — it compares repo content against a declared version, entirely inside CI.

8. Open questions¶

Does an increment's manifest addition oblige a requires_pinakes bump? Additive keys with defaults do not break a newer reader, but strictness means an older reader fails on them the moment a user sets one. A rule is needed: probably "bump when a key is added", accepting that this tracks feature releases closely.
What updates requires_pinakes on a KB whose owner never runs upgrade? Nothing does, so the floor reflects the last write. That is honest but means the field is a lower bound, not a promise.
~~Should doctor report available template upgrades?~~ Answered yes, shipped 0.17.0 — detection was cheap and report-only, and it makes the gap visible without waiting for pnk upgrade. It reported only a version string until 0.18.0, and pnk upgrade now prints the diff itself.
Multi-template ecosystem (the template release) multiplies all of this by the number of templates.
Small follow-up: the unknown-key remedy still points at docs/DESIGN.md §2.1, whose field tables moved to MANIFEST.md in 0.2.1. It should also offer the second hypothesis — that the manifest may have been written by a newer Pinakes — because today it only ever suggests a typo.
The one-key instance of the first question. Decided by the user 20260825 18:16; folded in here 18:46. A 20260805 decision proposed that shipping a new [chunking] value must set a requires_pinakes floor. It is superseded rather than withdrawn, and it belongs under this heading because the general question above is older than it and still open. Its premise failed: nothing writes the floor. D-11 (taken 20260804) settled that pnk upgrade --apply never writes it, and pnk init does not stamp it either — that half resting on D-6, a standing recommendation rather than a taken decision, a distinction kept because the practical bar and the decision status are not the same thing. Either way the clause specified a floor with no writer — and D-11's accepted-cost paragraph had already put exactly this harm to the user, who accepted it. The user-facing remedy is not missing: GUIDE.md § Troubleshooting answers the collaborator case directly. All four now sit in one place — this bullet, the question above it, D-6 and D-11 — which is what folding it here was for.

9. Scope — undecided¶

Deliberately not assigned. The cheapest useful subset, in dependency order:

Step	Cost	Buys
~~Bump the `notes` template version whenever its content changes~~	one line	Built (0.17.0). Makes the shipped `doctor` check fire at all
~~The template-drift CI gate (§6)~~	small	Built (0.17.0). Makes that bump impossible to forget
~~`doctor` reports what changed, not just that something did~~	small	Built (0.18.0, and `pnk upgrade` for the lines themselves). Makes the WARN actionable without writing to anyone's config
~~`requires_pinakes` + pre-pass read (§4)~~	small	Built (G4). Turns a misleading refusal into an actionable one
~~`pnk upgrade` + `--apply` + `tomlkit` (§5)~~	medium	Built (0.19.0 print, 0.20.0 `--apply`) — and without `tomlkit`. Existing KBs adopt new defaults. The `tomlkit` half was never needed: the rewrite preserves comments because nothing parses them away (§5), so no TOML round-tripper entered core

The withdrawn row. This table used to promise that "doctor reports manifest keys the installed template sets that this KB lacks" would close the PDF-glob gap. It would not: the template never sets **/*.pdf (§3 case 1), so there is no key for such a check to find. What §3 case 1 actually describes is a missing comment, which no key-level diff reaches — only a content diff does.

Every row in this table is now built, and the sentence that stood here — "the remaining two would close the live gap in §3, neither is assigned" — was wrong twice over: one row remained, not two, and it shipped in 0.19.0 and 0.20.0. What §3 still describes is closed by none of them. Case 1's missing PDF comment and case 2's budget keys both reach new KBs only, and adoption cannot help a KB whose recorded template version has no archived content to adopt against. That is a baseline problem, and it is the one the template release still owns.