mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-11 21:39:01 +00:00
c4d429d14c94de941ac761333cdac0a33895a05f
338 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xarmian
|
c4d429d14c |
fix: three codex round-6 findings, one premise corrected (TASK-2878)
Round 6 confirmed round 5 and found three. All three are real; one arrived
with an account of its own cause that the test refuted, and the corrected
route is narrower than the report.
## A NON-STRING RELATION DEFAULT, AND WHERE IT ACTUALLY GETS IN
The finding: injected defaults are never type-checked, so `42` or `[]` can
persist in a relation field. True, but not by the route described.
`MigrateFields` injects destination defaults ITSELF, so in the ordinary case
the key is present when `ValidateFieldsDetailed` runs and its type IS
checked — a numeric default lands in needs_value with "must be a string",
which is correct behaviour. My first test asserted the wrong thing and
FAILED against the fixed build, which is how I found this out.
The unchecked route is narrower: a NULL OVERRIDE deletes the key after
MigrateFields filled it, so validation injects the default itself — and its
own injection branch `continue`s PAST the type check. That is the one way a
non-string reaches a relation field unchallenged. The late-default pass owns
values that arrive from defaults, so it reports this one:
`invalid_shape`, a new reason, because every existing reason describes a
lookup that never happened.
Retargeting the test then exposed a second defect IN MY OWN FIX: the late
pass's `if len(late) == 0 { return nil, nil }` discarded the non-string drops
it had just recorded. The value vanished from all three buckets. Green on the
first route, silent on the second.
THE PARITY GATE FROM AN EARLIER COMMIT CAUGHT THE NEW REASON: adding
`invalid_shape` failed TestCopyPreflightDropReasonsAreRenderedByTheDialog
until the TypeScript union and CopyItemDialog learned it. That gate exists
because `referent_not_portable` shipped unrendered in BUG-2674, and it just
did its job on its author.
## A WHITESPACE-ONLY VALUE IS "NO REFERENCE", NOT A BAD ONE
The store resolver trims and ignores `" "`. The server wrapper checked the
UNTRIMMED string, so the value fell through to the visibility loop — and
since round 1's vanished-target arm turns a missing lookup into a refusal,
`" "` came back as not_found instead of an empty field. A defect my own
round-1 fix introduced: before it, that path did `continue`.
## A MALFORMED CARRIED VALUE IS NOT "NOT PORTABLE"
The cross-workspace branch dropped every carried value without looking,
including non-strings, and labelled them `referent_not_portable` — a false
account of why the value is going. It is not a reference at all. Left in
place now for ValidateFields to reject on shape, which is what the
SAME-workspace branch already did with it: the two modes disagreeing about
one malformed value was the defect.
## Counterfactuals
Skip non-string defaults again -> TestCopyEndpoint_NonStringRelationDefault
DETECTED. Restore the untrimmed skip -> TestRelationDoors_WhitespaceOnly
DETECTED. Both build-checked, and the first form of the second mutant did NOT
build (unused import) — reported as such rather than scored, since a
non-compiling mutant produces no failures and reads as survived.
Gates: internal/server ok 324.8s · internal/store ok 344.5s · internal/mcp ok
16.9s · go vet clean · gofmt clean · make lint 0 issues · npm run check 0
errors. Postgres green on the parent commit (store 596.1s, server 299.4s);
re-running on this tree.
|
||
|
xarmian
|
8fedc5487c |
fix: three codex round-3 findings — oracle, bulk defaults, required (TASK-2878)
Round 3 confirmed round 2's fixes and found three more. Codex still could not execute anything, so all three were static reads I verified myself. ## `wrong_collection` DISCLOSED EXISTENCE The store emits `wrong_collection` when a value names a LIVE item outside the field's declared target, and the server's visibility layer skipped any key that already carried an issue. So the message — "is not an item in collection X" — told a caller the value EXISTS, distinguishably from the `not_found` a nonexistent value gets. An existence oracle for anyone who cannot see that item, and the exact shape the `not_found` collapse exists to prevent. Collapsed to `not_found` when the requester cannot see the target, and KEPT otherwise: "you linked a task where a person belongs" is the useful half of this reason, and blanket-collapsing would have passed the security leg while destroying it. Both legs are in the test for that reason. Needed one new store export, `ResolveRelationTarget`: deciding what to disclose about a live item requires the item, and the resolver is the only thing that maps a value to one under the no-slug rule. ## BULK STATUS / PRIORITY BYPASSED RELATION DEFAULTS `bulkFieldUpdate` resolves only the keys `changes` names — correctly, since re-litigating stored values would freeze legacy items — but `ValidateFields` runs BEFORE it and INJECTS schema defaults. A defaulted relation was persisted raw: never canonicalised, never checked against its collection. The same late-arrival the migrate doors hit, reached by a different route, and the narrow pass built for them covers it unchanged. `dropped_fields` now rides every bulk op's activity row, not just the move branch: a status or priority change can discard a relation default too, and a drop nobody records is the defect BUG-2674 closed. ## A REQUIRED RELATION COULD END UP ABSENT AND REPORTED VALID The late pass deletes a key AFTER validation passed, so nothing re-checked required-ness: a required relation whose default did not resolve left the item with the field absent, and the preflight reported `valid: true`. Re-running validation is not the fix — it would re-inject the same broken default. There is no valid value for that field, so the doors refuse: `missing_required_fields` on move and bulk move, a FieldValidationError on the copy, and on the preflight a `needs_value` row with `valid: false`. That split is the one this pair has everywhere else — the preview says what is wrong, the copy refuses. ## Counterfactuals Remove the oracle collapse -> the visibility leg DETECTED. Remove the bulk-update late pass -> DETECTED. Remove the required-relation refusal at the copy door -> DETECTED. All build-checked first. Gates: internal/server ok 352.1s · internal/store ok 358.7s · internal/items ok · internal/mcp ok 21.0s · go vet clean · gofmt clean · make lint 0 issues. Postgres green on the parent commit (store 596.7s, server 294.7s, private container port 5481); re-running on this tree. |
||
|
xarmian
|
2e9df62c80 |
fix: two codex round-2 findings — late defaults, UUID disclosure (TASK-2878)
Round 2 confirmed the three round-1 fixes and found two more. Both real,
both verified in the code before acceptance; codex still could not run
anything ("the read-only filesystem prevented Go cache/temp creation").
## THE DESTINATION-DEFAULT FIX ONLY COVERED HALF THE WAYS IN
The migrate doors resolve BEFORE they validate, and that order is
load-bearing: the required-field check has to see a value referent
resolution dropped, or a dropped value in a REQUIRED relation field would
store the item with the field absent instead of refusing. But
`ValidateFields` INJECTS schema defaults, so a default can land after the
resolver has finished. Two ways in, both now closed:
- a NULL OVERRIDE deletes the key and the default fills the hole, arriving
uncanonicalised — a `PEOP-1` default reached the row as the literal
string;
- a default the resolver DELETED as unresolvable is put straight back by
validation, and `StillDropped` then suppresses the warning about it.
Dropped, reported, restored, and reported as not-dropped.
`ResolveLateRelationDefaults` is a narrow second pass over exactly the keys
validation added. Reordering the two would have traded this defect for the
required-field one; a second pass costs one lookup in the rare case a
relation field declares a default and nothing otherwise. A default is
asserted by nobody, so an unresolvable one is DROPPED and reported, never
refused — the same disposition the main pass gives
RelationOriginDestinationDefault, which is what this is: the same origin,
arriving late.
THE SNAPSHOT IS TAKEN AFTER THE MAIN PASS, NOT BEFORE, and getting that
wrong is what the second sub-case caught. Snapshotting before the pass
treats a key the pass DELETED as already examined, so the late pass skips
the very value validation just put back — the arrangement that hid it. My
first version did exactly this and the second sub-case failed on it.
## A REFUSAL FOR AN INVISIBLE ITEM DISCLOSED ITS UUID
The store resolver rewrites a supplied ref into its target's id before the
visibility check runs, so a message built from the resolved value handed
back the canonical UUID of an item the requester may not see — confirming
both its existence and its identity. That is the existence oracle the
`not_found` collapse exists to prevent, reopened by the message. Every issue
this function raises now quotes what the CALLER sent.
## Prior findings, per codex
Override visibility: fixed. Bulk move dropped list: fixed. Nil-target race:
no dereference. Destination defaults: this commit.
## Counterfactuals
Neuter the late-default pass -> TestCopyEndpoint_LateInjectedRelationDefault
DETECTED on both sub-cases. Restore the canonical UUID in the message ->
TestCopyEndpoint_InvisibleRelationOverrideIsRefused DETECTED. Both mutants
build-checked first.
Gates: internal/server ok 346.3s · internal/store ok 351.3s · internal/items
ok · internal/mcp ok 22.9s · go vet clean · gofmt clean · make lint 0 issues.
Postgres green on the parent commit (store 534.5s, server 268.5s, private
container port 5481); re-running on this tree.
|
||
|
xarmian
|
ed03d488da |
fix: four codex round-1 findings — origin, visibility, bulk reporting (TASK-2878)
Codex round 1 named four; three were real P1s and I verified each in the
code before accepting it. Codex could not run anything ("Go could not
create its build cache because the workspace is read-only"), so every
finding here is a static read that I confirmed and pinned.
## A THIRD ORIGIN, not two
`items.MigrateFields` injects the DESTINATION schema's defaults for keys the
source item has nothing for. My classifier split on `supplied` versus
everything-else, so a destination default was filed as CARRIED — and on a
cross-workspace copy every carried relation drops without a lookup. The
destination's own default was discarded and reported `referent_not_portable`,
which is flatly false about a value the destination chose.
There are three origins: SUPPLIED (refuse on failure), CARRIED from the
source item (cross-workspace: drop as not-portable), and DESTINATION DEFAULT
(resolve against the destination in BOTH modes; drop with the resolver's own
reason on failure, because nobody in this request typed it). Telling the last
two apart needs the source field map, which all four migrate doors have as
`currentFields`, so it is now a parameter.
Empty values are skipped at every origin. An empty relation is a cleared
field, not a referent, and reporting it as dropped tells a user they lost
something they never had.
WHAT THE FIRST VERSION OF THIS TEST PROVED: nothing. The mutant that reverts
the classifier SURVIVED it. `ValidateFields` re-injects the default after my
resolver deleted the key, so the value comes back either way and
`StillDropped` filters the false report out — the end state is identical
unless the default is a REF. A UUID default is already its own canonical
form, so "resolved" and "dropped then re-injected raw" produce the same
bytes. With `PEOP-1` as the default the mutant is DETECTED, because only a
resolved default lands as the id.
## SUPPLIED OVERRIDES AT THE MIGRATE DOORS SKIPPED THE VISIBILITY CHECK
The four write doors go through `s.resolveRelationReferents`, which adds
`checkItemVisible` on top of the store resolver. The migrate doors called
`store.MigrateRelationReferents` directly — it is a store function and cannot
answer a request-scoped question — so their SUPPLIED half, which this unit's
own rule calls an ordinary write, resolved against the database alone. A
caller able to edit both collections could point a relation at an item they
cannot see.
The ROLE is the part worth getting right. For `move` it is `workspaceRole(r)`.
For copy and preflight it is the caller's role in the DESTINATION, and
`CrossWorkspaceAccess.Role` is exactly that — its own doc says never to
substitute `workspaceRole(r)`. So `resolveRelationReferents` now takes the
role explicitly (`resolveRelationReferentsAs`), and the new
`refuseInvisibleRelationOverrides` runs at all three doors with the right one.
At the copy it runs in the HANDLER, before the store call: a pre-write
refusal must not open a transaction to roll it back, and the preflight runs
the identical check — DR-6's "the preview IS the copy" only holds if both
doors refuse the same request.
## BULK COLLECTION MOVE DISCARDED FIELDS SILENTLY
`bulkMoveCollection` has populated `result.Dropped` since MigrateFields
existed and NOTHING read it — the only reference in the file was my own
append. BUG-2674 fixed the single-item door and left this one, so a bulk move
discarded values with no record anywhere. Pre-existing, and routing relation
drops into the same dead list is what made it mine to fix.
Reported on the activity row, same key, same joined-string shape and the same
BUG-2628 reason as `handleMoveItem`, filtered against the final map so the
report is true when written. Threaded as an out-parameter, deliberately: only
this branch produces drops, the caller needs them for ONE activity row per
item, and a third return value would put `nil` in fourteen unrelated returns.
## THE P2, AND WHAT IT IS NOT PINNED BY
`resolveRelationReferents` did `if item == nil { continue }` after the
visibility read — "treat a race as someone else's 404". It now refuses with
the same `not_found` the resolver would have given moments later. This whole
unit exists to keep a dangling referent out of the blob, and a target that
vanished mid-request is the one case where waving it through would have been
deliberate.
NO TEST. Reproducing it means deleting a row between two reads inside one
request, and a test that faked that would pin the fake. Stated here rather
than left to look covered.
## Counterfactuals
Every fix has a mutant that its own test detects, each build-checked first:
classifier reverted -> DETECTED (destination default); empty-skip removed ->
DETECTED; visibility helper neutered -> DETECTED at both the move door and
the copy/preflight pair; bulk-move report removed -> DETECTED.
Gates: internal/server ok 224.6s · internal/store ok 261.5s · internal/items
ok · internal/mcp ok 15.1s · go vet clean · gofmt clean · make lint 0 issues.
|
||
|
xarmian
|
34861f8658 |
feat(mcp): ToolSurfaceVersion 0.29, and the drop-reason renderer it exposed (TASK-2878)
PLAN-2857 U1. The bump, its documentation sweep, and the consumer this
change turned from a rare wart into a routine one.
THE BUMP, at 0.29 rather than 0.28. Rebasing onto main found
|
||
|
xarmian
|
ee607047dd |
feat(copy): the two cross-workspace doors resolve referents, with their pin (TASK-2878)
PLAN-2857 U1, doors seven and eight. `migrateCopyFields` and `handleCopyItemPreflight` now take their relation decision from `store.MigrateRelationReferents` — the same function the four write doors and the two move doors already call, which is the entire reason it exists. The defect this closes: MigrateFields matches on key and TYPE, so a same-named `relation` field carried a SOURCE-workspace item id across the boundary and the preflight reported it as a clean carry. What landed in workspace B was a value naming a row in workspace A — unrenderable, and indistinguishable on read from a legitimate reference. Provenance decides, as at the move doors. A CARRIED value on a cross-workspace copy is dropped without a lookup (no id from A can mean anything in B) and reported through the `dropped_fields` channel BUG-2674 established; a SUPPLIED override is an ordinary write and an unresolvable one is refused, 400 validation_error on both doors, rendered by the same `store.RelationIssuesMessage` so one refusal cannot acquire two phrasings. `internal/server`'s `relationIssuesMessage` now delegates to it: the eighth door refuses from inside `store`, so the sentence had to be reachable there. Two things are threaded rather than re-derived, and both are load-bearing: - The TRANSACTION, not the pool. `migrateCopyFields` becomes a method taking a Queryer, and `copyItemAcrossWorkspacesTx` passes its `tx`. That function has held a transaction since its second statement, so a pool read from inside it can wait for a free connection while every pooled connection is blocked on this transaction's locks — the starvation shape BUG-2409 fixed for the attachment planner and this repo keeps a deterministic test for. This is what the day-70 handoff named as the reason these two doors were not wired with the other six. - The MODE comes from the `scope` MigrateFields was already given, not from a second boundary test. Two independent answers to "is this crossing a workspace" is how one request gets migrated one way and validated the other, and this path also serves a copy whose target IS the source workspace, where relations resolve and survive exactly as on a move. The destination workspace id is the resolution scope: a supplied override is a write into B and must name something that exists there. THE PIN, and why it is not a per-door table. These two doors sit in different PACKAGES and the code at both sites says so is how they drift unnoticed. A table with a row per door can be fully green while the two disagree about one request, which is the defect rather than a gap in coverage of it. So every case sends ONE body to BOTH endpoints: - carried relation — must drop on both, and the preflight must say referent_not_portable rather than the generic no_target_field, which is false here (the destination DOES declare the key, so that answer sends the reader to fix a schema that is fine); - supplied + unresolvable — both refuse, same status, same code, both name the offending field and value, and nothing is written; - supplied + resolvable — the positive control, supplied as a REF so resolution is visible in the result. Without it the first two legs are equally consistent with "relations always fail". Negative controls run, all three mutants BUILD-CHECKED first (a non-compiling mutant produces no `--- FAIL` lines and reads as survived): both doors unwired = DETECTED; preflight unwired alone = DETECTED; store unwired alone = DETECTED. Each single-door mutant failing is the pin's whole claim — neither door can be wired without the other. CONVE-23 sweep: the preflight's LIMITATION comment said this gap belonged in MigrateFields "for both callers at once". That is now false in its prescription as well as its premise — `internal/items` is DB-free by construction and cannot ask whether a string names a live item — so the comment records where the fix actually went and what of it remains open (`computed`, `terminal_options`, `unique_scope`). Gates: internal/server ok 170.0s · internal/store ok 296.1s · internal/items ok · go vet clean · gofmt clean · make lint 0 issues. |
||
|
xarmian
|
f7caba5657 |
refactor(store): thread a Queryer through referent resolution (TASK-2878)
Preparation for the two cross-workspace copy doors, landed on its own because it is independently correct and the doors are not. `migrateCopyFields` runs inside `copyItemAcrossWorkspacesTx`, which opens a transaction as its second statement. A resolver reading from the POOL there would issue pool reads while holding a tx — the deadlock this repo keeps a deterministic test for. So `ResolveRelationReferentsQ` and `MigrateRelationReferentsQ` take the executor, following the store's own convention (`GetItemQ`, `uniqueSlugQ`, `getCollectionInWorkspaceTx`); the pool-backed names stay as one-line shims for the six wired doors. Two small read helpers come with it. `collectionIDBySlugQ` returns the ID only — the referent check compares `item.CollectionID`, and the full model would pull in per-collection counts nothing here uses. `itemByRefQ` keeps `GetItemByRef`'s fallback to a bare item-number lookup, because a relation written as COLO-3 must keep resolving after its target collection is renamed, which is exactly what BUG-2873 made possible. internal/store ok (341.7s), vet and gofmt clean. WHY THE COPY DOORS ARE NOT IN THIS COMMIT. They were written and building, and I reverted them. Team CONVE-29 and the lead's condition both say the copy pair lands WITH its pin — one case driving BOTH doors, asserting identical drop-and-report for a carried relation and refusal for a supplied override — and I measured 58.9% context against a 65% ceiling, which is not enough for that pin plus the 270s server suite plus the commit. Landing the behaviour change unpinned would have been worse than landing nothing: the preflight and the store copy are the pair the code already warns will drift unnoticed, so they are the last place to accept an untested agreement. The design is complete and on the trail: derive the carry mode from the existing `items.MigrateScope` rather than a second flag, pass `tx` on the store side and the pool on the preflight side, refusals through the copy's existing validation-error channel, drops appended to `migrated.Dropped`. |
||
|
xarmian
|
9b19a78459 |
feat(store): one migrate decision for all four carrying doors (TASK-2878)
PLAN-2857 U1, third slice, on the lead's refined ruling: PROVENANCE
decides, not which door you came through.
* SUPPLIED (an explicit `--field` override on a move or copy) is a write
like any other, so an unresolvable value REFUSES.
* CARRIED (everything the source item already held) was asserted by
nobody. `internal/items` has accepted any string for a relation all
along, so most stored values are legacy — refusing them would make
those items unmovable and uncopyable. Dropped and REPORTED instead.
And carried values are not all alike, which is the refinement that keeps
this from being one rule wearing four coats:
* WITHIN a workspace (move, bulk move) the targets are still here, so a
valid relation SURVIVES the move and only an unresolvable one is
dropped, through the `dropped_fields` channel BUG-2674 established.
* ACROSS workspaces (copy, and its preflight) every carried relation is
dropped WITHOUT a lookup: the value names a source-workspace row and
v1 excludes cross-workspace targets, so no amount of resolving in the
destination changes what it means. Reported as `referent_not_portable`
— the same reason `github_pr` uses, because it is the same fact about
the same kind of value.
`MigrateRelationReferents` is one function because the four doors sharing
it is the point, not tidiness: the preflight lives in `internal/server`
and the copy in `internal/store`, and the code already carries a comment
saying those two sit in different packages and that is how they drift
unnoticed. A preflight that says "carried" while the copy drops is one
request answered two ways.
Tests drive both provenances against both modes, because the same bad
value must be a drop when carried and a refusal when supplied — a suite
that only drove carried values would pass against a build that never
refuses anything.
|
||
|
xarmian
|
977132387d |
feat(store): referent resolution for relation values (TASK-2878)
PLAN-2857 U1, first slice: the rule itself, with no door wired to it yet. `ResolveRelationReferents` canonicalises every `relation` value in a field map to the target item's ID and reports the ones that cannot be resolved — same workspace, and the collection the field DECLARES. WHERE IT LIVES was forced, not chosen. `internal/items` is DB-free by construction and keeps the shape check only. `internal/server` cannot own it either: six of the eight coercion doors live there, but the eighth is `store.migrateFieldsForCopy`, and `store` does not import `server`. Putting it here is what lets the cross-workspace copy door and the preflight door reach the SAME function instead of two implementations of one rule — those two already carry a comment saying they sit in different packages and that is how they drift unnoticed. VISIBILITY IS NOT HERE, deliberately. "Can this requester see that item" is request-scoped and needs the user, role and auth mode; the server layer adds it via `checkItemVisible`, which already exists as the context-free predicate for exactly this reason. NO SLUG FALLBACK, which is a deliberate divergence from `ResolveItem` (UUID, then ref, then slug). Found by a test failing rather than by reading: "red" resolved, because it is the slug of the live Red colour. A relation field's contract is that it stores an item ID; a slug is neither an ID nor stable, so the same stored value could point elsewhere tomorrow. Worse, "red" is exactly the free-text value the pre-U2 editor wrote into these fields, so accepting it makes the corruption this unit exists to stop indistinguishable from a legitimate write. The client refuses the same match for the same reason (TASK-2868). Exact-TITLE resolution is U6. Issues are reported in SCHEMA order, not map order, because the copy preflight is one of the callers and is specified to be safe to call repeatedly and return identical results. Unresolvable values are left EXACTLY as supplied: the caller quotes them back, and a half-canonicalised map would make a drop report lie about what the source held. Verified rather than asserted: both lookups exclude soft-deleted rows (`ResolveItem` by contrast with `ResolveItemIncludeDeleted`; `GetItem` via `getItemScanQ`, which appends `AND i.deleted_at IS NULL`). That is what keeps "target was deleted" distinguishable from "never resolved" — the read half U2 shipped. |
||
|
xarmian
|
b437cc582d |
feat: item reminders — the fire-at-an-instant primitive, and one overdue rule for all four surfaces (IDEA-2641, closes #1010) (#1244)
* feat(store): item reminders — the fire-at-an-instant primitive (IDEA-2641) Adds the storage, the scheduler tick, and the canonical event for one-shot item reminders (GitHub #1010). Nothing in Pad acted at a target time before this: a due_date makes an item show up as overdue once somebody asks the dashboard, so "revisit TASK-X on the 1st" had to live in an external cron. A TABLE, NOT A SCHEMA-FIELD ANNOTATION. The design sketch proposed marking schema date fields with a `reminds: true` key on models.FieldDef; recon overturned it. Such a key does not survive an ordinary collection edit, two independent ways: the web editor destructures each field into an EditableField and rebuilds a fresh definition key-by-key on save, so unknown keys are dropped (`pattern` and `unique_scope` survive only because two lines were hand-added for them), and models.CollectionSchema has fixed fields with no catch-all, so any Go unmarshal+marshal round-trip strips unknown properties — the hazard retargetRelationFieldsTx mutates raw JSON to avoid. Both failures are silent and both disarm a whole collection's reminders at once. It is the same defect class that moved traits out of the schema column in TASK-2657. The table also gives the lifecycle a home. A reminder is armed, then fired, then acknowledged, and a re-arm returns it to armed — per-reminder state a field definition has nowhere to keep. remind_at is an RFC3339 UTC instant, deliberately not a `date` schema value: those admit both YYYY-MM-DD and full RFC3339 and are compared against the SERVER'S LOCAL calendar day. A fire-at time cannot carry that ambiguity. The remaining timezone question for due_date is filed separately. Firing is one transaction per reminder carrying BOTH the fired_at write and the outbox insert. That pairing is the point: a fired_at committed without its event is a reminder that silently notifies nobody and can never be retried, because the row has left the armed set; an event without fired_at fires every tick forever. The UPDATE's own `fired_at IS NULL` predicate is the arbiter, so two instances ticking at once produce exactly one winner. item.reminder_due is admitted to the closed events/1 set as v1.2, with a new PayloadReminder family and no SSE name. The subject is the REMINDER, not the item: two reminders can be armed on one item, so an item-subject event could not say which fired, and the reminder id is what an acknowledgement addresses. A new payload family rather than reusing the item snapshot for the same reason — a snapshot would validate and still not answer the only question the event exists to answer. No SSE name in v1 because the poll surface is the contract; adding one later is additive, removing one is not. Ack is explicit and nothing else acks. An item reaching a terminal status deliberately does NOT ack: that would make every status write a reminder mutation, and it would silently consume a reminder set to fire after the work was done. * feat(server): reminder surfaces, and one shared overdue rule for all four Second half of IDEA-2641: the HTTP surface, the scheduler tick's wiring, and the fix for the finding that justified the unit — `ready` / `next` did no date handling at all. OVERDUE NOW HAS ONE IMPLEMENTATION. It used to live inline in the dashboard's attention loop, which meant `pad project stale` inherited it (it filters that very list) and the recommendation surface never saw it. So a deadline reached the two surfaces that REPORT on work and never the one an agent PULLS from. overdue.go is now the only place that decides, and all four call it. Two behaviour changes fall out, both deliberate: - An overdue item bypasses the orphan branch's high/critical priority gate. That gate was where a deadline quietly stopped: a low-priority item three weeks late was reported by `stale` and never suggested by `next`. - Overdue sorts above in-progress. The list is capped at three, so a rank below in-progress would not merely order the deadline lower — on any workspace with three things in flight it would keep an overdue item off the surface entirely, which is indistinguishable from not shipping this. The server-local-today comparison is UNCHANGED and known to be wrong for multi-timezone deployments; it is filed as its own item with the cloud case stated. Changing what "overdue" means on every existing instance inside a change about where the rule LIVES is the kind of behaviour change nobody reviews. Fired reminders reach `next` / `ready` two ways, from one filtered list: PendingReminders is the addressable form (it carries the id an ack needs), and a prepended suggestion is the rendered form. They are prepended AFTER the cap rather than entered as ranking candidates — a reminder is not a task competing on priority, and whether it appeared should not depend on how busy the workspace is. Terminal-item reminders are FILTERED from the surface, never acked. Acking on terminal status would couple every status write to reminder state and would consume a reminder armed to fire after the work was done. The row stays exactly as the user left it; the distinction is observable, and asserted. Three guard tests caught this change and each was answered rather than silenced: - The request-body reader guard was right: the handlers now go through decodeJSON, inheriting the NUL refusal and the size cap. - The canonical-events guard was right: item.reminder_due is admitted to the duplicated contract table as SPEC-3 v1.7, with the reminder subject kind and the new payload family. SPEC-3's own text owes the same amendment. - The NUL census asked for a decision on eight new columns. None carries caller text: ids and FKs are server-generated, four are the server clock, and remind_at is now re-parsed and re-formatted in the STORE as well as at the edge — so the stored value is always machine-produced from a parsed time and no caller bytes reach the column. The doc comment that used to say "the caller normalizes" protected nothing. Regenerating the baseline also found that GEN_NUL_BASELINE=1, which the test's own instructions name, was never implemented — the flag did nothing, so the documented path was hand-editing the file. Implemented, so the next reader gets the mechanism the instructions promise. * test(reminders): the lifecycle, the four surfaces, and 22 killed mutants Every test here was designed against a specific mutation and the mutation was RUN. A green suite proves nothing about a suite nobody tried to break, and three of the mutants I first wrote were not experiments at all. Store (10 mutants, all killed): candidate predicate <= flipped to >=; the event emission lifted out of the fire transaction; the fire UPDATE's `fired_at IS NULL` arbiter removed; the RowsAffected check ignored; re-arm clearing fired_at but not acked_at; ack losing `fired_at IS NOT NULL`; the poll surface losing `acked_at IS NULL`; normalizeRemindAt no longer refusing; it dropping .UTC(); GetReminder losing its workspace scope. Surfaces (12, all killed): the priority gate no longer bypassing on overdue; the sort no longer ranking overdue first; attention leaving the shared helper; the reason losing its OVERDUE prefix; the comparison flipped to >; terminal items no longer skipped; terminal reminders no longer filtered; the filter ACKING instead of hiding; reminders appended instead of prepended; the tick running on a far-future clock; ack answering 200 for an unfired reminder; parseRemindAt accepting a bare date. THREE MUTANTS DID NOT COUNT ON THE FIRST PASS and were rewritten. Two failed to compile (`if false` orphaned a variable; deleting a parse orphaned an import) and one had an anchor matching two call sites. A non-compiling mutant emits zero FAIL lines and reads exactly like a surviving one — it invents a hole that is not there — so the harness reports BUILD-FAIL and ANCHOR-BAD as outcomes distinct from SURVIVED. It also restores files from an in-memory copy rather than `git checkout`, which would delete uncommitted work in the tree. ONE MUTANT GENUINELY SURVIVED and the test was at fault, not the mutant: appending rather than prepending reminder suggestions was undetectable because the fixture had a single item, so the reminder sat at index 0 either way. The fixture now fills the three-item cap with in-progress work, where an appended reminder lands fourth and vanishes. Faithful mutant, weak test — checked in that order. The same lesson shapes the four-surface fixture: it is a LOW-priority open orphan, because that is the case the old code handled worst. A high-priority task would have made the ready/next leg pass against the unfixed tree, which is a green that measures nothing. Negative controls throughout: a future deadline is not overdue and does not reach the gate bypass; a tick with nothing due fires nothing; a completed item is neither overdue nor suggested. Without them a helper that reported every date, or a tick that fired everything, would satisfy every positive leg. The lead's pin is asserted in both directions: a fired reminder on a done item is ABSENT from the surface and PRESENT and still unacknowledged in the table. Asserting only the absence would pass against an implementation that consumed the row, which is the behaviour the pin exists to forbid. * feat(mcp): pad_item.remind + ack-reminder, ToolSurfaceVersion 0.28 An agent that can RECEIVE a reminder but not set one has half the primitive. The poll surface is pad_project.next / ready, both long exposed, so reminders already reached agents — what was missing is the other half: deferring a piece of work is exactly the moment an agent knows when it wants to be asked again, and it had no way to say so. Two additive actions, two optional params. Nothing existing moved, so a v0.27 consumer enumerating neither is unaffected — the v0.13 / v0.11 / v0.8 disposition, which likewise wired existing CLI verbs onto the catalog. remind_at REFUSES a bare date rather than reading it as midnight. Worth stating because the `date` schema type accepts YYYY-MM-DD and a caller will reasonably try it here: a bare date names a 24-hour span, and choosing an hour inside it would fire at a time nobody picked. Re-arm and disarm stay CLI-only. Both address a reminder by an id the agent would have to list first, and no listing action exists on this surface — a door with no handle. Adding them later is additive. Five guards had to be taught, and each was answered on its merits rather than excluded: the HTTP parity test (route mappers added, so the actions work on the remote transport rather than being advertised and unrouted), the read-only catalog's cmdhelp fixture and expected cmdPath map, the field- conflict classifier (remind_at / reminder_id are NOT field writers — a reminder is a row in its own table addressed by its own id, so listing them as classified sources would have pointed detectFieldConflicts at something that is not a field source), and the instructions.md / README action tables. That machinery is why the version bump is safe to make now, and it earned its keep on this change: every one of the five failed on the first build after the catalog entry landed. CONVE-23 sweep for prose this falsifies: - SPEC-3 (DOC-2653) amended to v1.7 in the room, recording item.reminder_due with its new subject kind and payload family — the first canonical event with no user mutation behind it, since a scheduler tick produces it. - CLAUDE.md gains the reminder routes, the CLI verbs, and the v0.28 entry. It was also stale at 0.26 with NO v0.27 entry at all: the 0.27 unit swept instructions.md and README.md and missed this file. Both added. - skills/pad/SKILL.md gains the verbs and a routing entry, including the two things an agent will get wrong — the time is an instant, so ask for a time of day rather than picking one, and finishing the item does not acknowledge the reminder. * fix(reminders): codex round 1 — four findings, all real, all with a pin Round 1 found four defects and refuted none of them. Each fix carries a test that fails against the code as it was, and each of those was mutation-checked. **P1 — pending reminders bypassed item-level visibility.** Every other dashboard section reads `allItems`, which the store already scoped to the caller's collections AND their granted item ids. The pending-reminder list is a direct workspace-wide query and inherited none of that, so a guest holding a grant on ONE item could read the refs and titles of every other item in the collection through its reminders — an item-level leak wearing a notification's clothes. Now filtered with the same `isItemVisibleToGuest` call the sibling sections use. The test's two items share a COLLECTION on purpose: a collection-level filter was already applied, so separate collections would have made it pass against the unfixed code. **P1 — soft-deleted items could starve the queue permanently.** Candidate selection ignored `deleted_at`, and `fireOneReminder` rolls back when it finds the item gone — which leaves the reminder ARMED and therefore a candidate again on the next pass. Candidates are ordered oldest-first and bounded by a limit, so enough archived reminders fill every batch and no live reminder ever fires. Silent, too: the tick reports zero fired and looks idle. Excluded in the candidate query rather than skipped downstream, so those rows never occupy a slot; the reminders themselves are kept, so restoring an item restores its reminder with it — asserted, because a fix that reaped them would pass the starvation test alone. **P2 — the pass stopped at the first failing reminder.** The per-reminder transaction exists precisely so one unfireable row cannot hold back the rest, and `return fired, err` made that comment false — with candidates oldest-first, one persistently broken old reminder blocks every newer one forever. Now continues and joins the errors, so a pass that fired seven and failed three reports both halves rather than reading as clean. The loop is split behind an injected seam because a real mid-transaction failure is not reachable from outside: the database refuses the corrupt rows that would cause one (verified — invalid JSON in items.fields is rejected by the schema). **P2 — suggestions dropped the reminder id.** The docs tell an agent to acknowledge what it sees in next/ready, and the payload carried no handle: a stateless poller could read the reminder and had no way to retire it, so it would be shown the same item forever. `DashboardSuggestion` now carries `reminder_id` (omitempty), `pad project next` prints the exact ack command, and the test acks with the id the surface handed out rather than merely checking the field is populated — a wrong-but-present id satisfies equality with itself. Four mutants, four killed; one was rewritten first because its anchor matched two call sites and was therefore not an experiment. * fix(reminders): codex round 2 — four findings, all real **`--rearm` was unusable.** `ExactArgs(1)` forced an item ref that the rearm branch then ignored, so the flag could not be reached without supplying a ref that was silently discarded. Now `MaximumNArgs(1)`, with each mode checked explicitly: a ref is required to arm, and a ref supplied ALONGSIDE `--rearm` is refused rather than ignored — it names an item the reminder may not even belong to, and quietly dropping it is how a user learns nothing about the reminder they just moved. **`unremind --format json` emitted plain text**, breaking the parseable-output contract every sibling command honours. **The MCP `ref` param did not list `remind`.** Agents read that flat description to decide what to send, so an action missing from it is an invalid call waiting to happen. It now also says what `ack-reminder` takes instead, and why: a reminder is addressed by its own id because an item can carry several. **Fractional seconds fired early.** `time.Parse` accepts `09:00:00.900Z` and `Format(RFC3339)` drops the fraction, so it was stored as `09:00:00Z` and fired 900ms BEFORE the moment the caller named — silently, having rewritten their value on the way in. Seconds are genuinely the stored resolution (the column is compared as a string against a whole-second clock, and the tick runs every 30s), so the only question was which way to resolve it, and truncation resolved it the wrong way. `NormalizeInstant` now rounds UP: at most a second of lateness, in exchange for a guarantee that can be stated — a reminder never fires before the instant it was set for. Late is a reminder; early is a wrong answer. Whole seconds round-trip exactly, which is asserted, because an implementation that added a second unconditionally would otherwise pass. Three mutants for this round, three killed (round-up→truncate, round-up→unconditional-add, MaximumNArgs→ExactArgs). Thirty across the unit. Two fixes carry no dedicated test and it is worth being explicit rather than implying coverage: the `--format json` branch on `unremind` is a one-line output change with no server-free way to drive it, and the MCP `ref` description is prose the drift tests do not read — they assert an action is DOCUMENTED, not that a param's sentence lists it. * docs(reminders): the ack id is on the surface an agent polls, not only on the arm response CONVE-23 follow-through on the round-1 fix. Both agent-facing docs told a caller to acknowledge a reminder with the id "returned when you armed it" — true, and useless to the caller that matters: a poller reading next/ready never armed anything. The suggestion now carries reminder_id and `pad project next` prints the exact ack command, so the docs say that instead. The prose was written before the fix existed, which is exactly the case CONVE-23 is about: a change that makes an instruction stale without touching the file the instruction lives in. * test(reminders): bind the tick LOOP to the work, not just the pass (CONVE-19) Every other test in this file calls runReminderTick directly. That vouches for the component and says nothing about whether anything ever calls it — a tick that is never started is indistinguishable, from those tests, from one that is. It is the convention's exact case, and the failure I recorded on my own identity doc three times in one unit: I test the component and not the binding. Driven through the injectable tick channel so the assertion pins a SPECIFIC pass instead of racing a 30-second ticker, and polled to a bounded deadline so a loop that never runs FAILS rather than hanging the suite. Mutant: drop `s.runReminderTick()` from the select and this goes red while every direct-call test stays green. Killed. The idempotence leg exists because a second Start spawning a second loop would leave one running after Stop, making the BUG-842 drain invariant false for this sweeper specifically — the one property a copied lifecycle is most likely to get right by accident and least likely to be checked. The cmd/pad call site (cmd_server.go, alongside StartTokenReaper) stays verified by inspection: a source-scanning guard for it would be an instrument asserting facts about source, which is code with an adversary and not worth it for one line that sits in the middle of five identical neighbours. * fix(reminders): codex round 3 — a deferred reminder fired anyway, and the poll surface was unbounded **A re-arm mid-pass did not stop the fire.** The candidate scan selects an id; before the UPDATE runs, a `--rearm` can move that reminder into the future. Re-arm clears `fired_at`, so a predicate checking only `fired_at IS NULL` still matched — the pass fired a reminder the user had just deferred and emitted its event. The re-arm cannot undo that: it can clear the mark, but the event is already on the outbox and at-least-once means a consumer has seen it. The fire UPDATE now revalidates `remind_at <= nowTS` against the SAME nowTS the candidate scan used. Same-value deliberately: the arbiter and the scan must agree about when this pass is, or a reminder could pass one and fail the other for no reason but clock drift inside a single pass. **The poll surface was unbounded.** Every fired-and-unacknowledged reminder was loaded and turned into a suggestion prepended to a list that is otherwise capped at three, so a workspace with five hundred unacknowledged reminders returned five hundred suggestions — in the dashboard response, the hottest read in the product, growing until somebody acknowledged them. Two bounds, because they are two different guarantees: the query takes a window (default 50, oldest-fired first, so it holds what has waited longest), and the prepended suggestions are capped at 5 so `suggested_next` stays a recommendation rather than a second inbox. The full set stays addressable in `pending_reminders`. Truncation is REPORTED as a boolean, not a count. A count would have to be post-visibility-filter to be true for the caller reading it, and the store cannot compute that — the filter runs per item, above. "There are more than you can see here" is the strongest claim the data supports, so it is the one made. Four mutants; two killed outright, two survived and were run down under CONVE-28: - **Uncapped suggestions survived because the fixture had ONE reminder** — capped and uncapped are the same list at n=1. That is the SECOND time a single-item fixture hid a count-or-order property in this file. Fixture now arms eight; it also asserts all eight remain in `pending_reminders`, so the cap is pinned to the recommendation and not to the data. - **Removing the SQL LIMIT survived, correctly, and the test comment now says so.** The Go slice cap bounds the PAYLOAD; the SQL LIMIT bounds the DATABASE'S work. Only the first is observable at this level — with the LIMIT gone the response is still bounded, while the query silently goes back to materialising every pending row before discarding most of them. That is a memory and I/O property with no assertion available here, so it is stated as a coverage boundary rather than papered over with a green that would not have measured it. * docs(reminders): the fire predicate arbitrates against two actors, not one CONVE-23 inside the file the round-3 fix touched. The comment described the UPDATE as an arbiter for concurrent TICKS, which is what it was written for and is why I did not re-read it when asked whether a user edit could race the pass. It now says what it actually defends against, and names the general shape: an arbiter is only an arbiter with respect to the writers it can see. * fix(reminders): codex round 4 — the round-3 bound recreated the round-1 starvation Round 3 bounded the poll surface. Round 4 caught what that bound did: the query took the first N rows and the dashboard then discarded the ones it could not show — hidden items, unauthorised items, completed items — so N such rows hide a visible reminder behind them indefinitely, with no continuation to reach it. That is the SAME defect I had removed from the fire path one round earlier, reintroduced in the read path within the hour. The general form is worth stating because I clearly did not hold it: **a bounded window is only safe when the discarding happens BEFORE the bound.** Filtering above a limit is a starvation every time, and it does not matter what the filter is for. Two halves, because the two filters are not the same kind of thing: **Visibility is now scoped IN SQL**, using the same collection-id / item-id sets every other dashboard section gets through `allItems` — the same three-way shape as ItemListParams, where holding both collection grants and item grants is an OR. Invisible rows no longer occupy the window at all, which is strictly better than filtering them out afterwards and is what the sibling sections have always done. **Terminality is paged**, because SQL cannot evaluate it — a collection's schema defines which statuses are terminal. The collector refills from the next page when a page comes back short, bounded by a max scan so a workspace full of completed items cannot turn a dashboard read into a table scan. The bound is 10x the window: the common shape fills on the first page, and the pathological shape terminates in a fixed number of indexed reads. Stopping at the scan bound reports truncation, which is honest — there may be more, and we did not look. The empty-scope case is a THIRD state that reads like the second: nil CollectionIDs means unrestricted, a non-nil EMPTY slice means this caller sees no collections. Without an explicit guard they collapse, because the switch matches none of its cases at length zero and adds no clause at all — so "nothing visible" would return the whole workspace. Three mutants, one survived: the empty-scope guard, because no dashboard-level test produces that state (callers that would are refused earlier by workspace access). Faithful mutant, missing test — it now has a direct one, with a sanity leg so a build returning nothing cannot pass it by accident. A guard for a state nothing exercises is exactly the one that rots. * fix(reminders): codex round 5 — the MCP action I shipped did not work over stdio **P1: local stdio MCP `remind` was unusable.** cmdhelp derives positionals by regex from a command's `Use` string, and `<instant>` inside `remind <ref> --remind-at <instant>` matched — it became a second REQUIRED positional, so dispatch failed with `missing required argument "instant"`. The action was advertised on a transport where it could not run. **The MCP catalog's own tests did not catch it, and the reason is the finding.** That suite builds its cmdhelp document BY HAND: I wrote `Args: mkArgs("ref")` in it, so the fixture agreed with what I meant rather than with what the CLI says. Five parity and drift tests passed against a document I authored to match my own intention — the "a test that agrees with whatever the table says is not a test of the table" shape, which the canonical-events test warns about in its own comment two packages away. The new test reads the REAL command tree via cmdhelp.Build, which is the only thing in this repo that can disagree with me about what the CLI declares. **P2: `pad project ready` withheld the ack handle** that `next` prints. Showing a fired reminder on the surface an agent polls while withholding the id it needs to retire it means the same entry comes back on every poll, forever. **P2: suggestions asserted a collection they did not have.** The orphan branch admits ANY collection — its own comment claimed it gated on tasks "mirroring the active-plan branch", and that comment was simply false — while the output hardcoded `Collection: "tasks"` and the reason said "Open task". Pre-existing for high-priority items since BUG-1082; my overdue bypass widened it to any overdue item, which is how it surfaced. Fixed by carrying the item's REAL collection rather than by narrowing the branch: narrowing would silently drop the non-task items this has surfaced for a year, and the defect is the mislabelling, not the inclusion. The false comment is replaced with what the code actually does. The first version of that test used an overdue IDEA and SKIPPED — ideas use `new`, and the branch requires `open` or an active status, so it never became a candidate. A test that cannot fire is a failed reconstruction, not a pass; the fixture is now a bug-like collection whose vocabulary contains `open`, which is the population the defect can actually reach. Three mutants, three killed. Forty-one across the unit. * fix(reminders): codex round 6 — reminders fired from soft-deleted workspaces **P1, and the only defect in this unit whose consequence leaves the process.** Workspace soft-delete deliberately keeps items for the 30-day restore window, so the candidate query's filter on the ITEM's deleted_at found nothing wrong — and the tick kept firing, emitting outbound webhook events for a workspace whose owner had deleted it, possibly while deleting their account. Both queries now join workspaces and require `w.deleted_at IS NULL`. Nothing is destroyed: a restored workspace resumes firing, which the test asserts, because "stops firing" and "is destroyed" are very different answers to someone who restores a workspace and only one of them is right. That test first failed for the WRONG REASON and the fixture was at fault: it counted every outbox row in the workspace, and item creation writes its own, so the assertion was satisfiable by the fixture itself and discriminated nothing. Scoped to the reminder event type. **`Use: "remind <ref>"` declared a requirement the command contradicts.** cmdhelp derives the machine-readable arg spec from that string, and `--rearm` takes no ref — so the published contract said "required" for something optional. The requirement is CONDITIONAL, which cmdhelp cannot express, so the honest declaration is `[ref]` plus the explicit check that names both call shapes. The round-5 test grew a `required` column, which is what makes this observable at all: asserting only the arg NAMES would have passed. **The pad_item tool description omitted both new actions.** The params were declared and the actions dispatched, but the prose an agent reads to decide what a tool can do did not mention them — discoverable only by someone who already knew to look. It now describes both, including the two things an agent gets wrong: remind_at is an instant, and nothing but an explicit ack retires a fired reminder. Three mutants, three killed. Forty-four across the unit. * fix(reminders): codex round 7 — one predicate for the scan and the arbiter Third instance of one class, so this fixes the SHAPE rather than the instance. The class: the candidate scan filters on something the fire transaction does not revalidate, so a change committed between them fires a reminder that no longer qualifies. Round 3 was a re-armed instant. Round 1's soft-deleted item was the same thing caught from the other side. Round 7 is a workspace deleted between the scan and the fire — the round-6 fix added the condition to the SCAN only, and the arbiter went on not knowing about it. Fixing those one at a time is what let the third happen. `reminderFireable` is now a single string that both sites reference: the scan asks it and the fire UPDATE re-asks it, so they cannot disagree, and a fourth condition is one edit in one place rather than two edits someone has to remember are paired. Written as a correlated EXISTS on item_reminders.item_id rather than a JOIN precisely so the identical text is valid in both a SELECT and an UPDATE, and the scan drops its table alias so the two uses are the same characters. What deliberately stays outside it: `fired_at IS NULL` and `remind_at <= ?` live on the reminder row itself, are already spelled identically at both sites, and folding them in would need a parameter order the shared form cannot express. Said in the comment so the omission reads as a decision. Both directions are now tested at the arbiter — a workspace deleted mid-pass and an item deleted mid-pass — because the item case previously relied on the item load coming back nil, and someone simplifying the EXISTS down to the workspace check alone would otherwise still see green. Three mutants, three killed: the arbiter dropping the shared predicate, and the predicate dropping each of its two halves. Forty-seven across the unit. * fix(reminders): codex round 8 — workspace export silently dropped every reminder WorkspaceExport is a hand-maintained field list, so a new table joins it only if someone remembers. Reminders did not: a backup/restore, or a SQLite→Postgres migration via `pad db migrate-to-pg`, dropped every pending reminder with nothing in the destination to show anything had gone. The line that list has always drawn is item-scoped workspace CONTENT (comments, links, versions — exported) versus per-user state (stars, watches — not). A reminder has no user column and hangs off an item, which puts it on the exported side. Stating the rule rather than just adding the field, because the next person adding a table needs to know which side they are on. LIFECYCLE MARKS ARE CARRIED, not reset. A fired-and-unacknowledged reminder is still owed to whoever armed it, so it arrives pending; an armed one whose instant has passed fires once on the destination's first tick, which is what would have happened had the workspace never moved. Re-arming everything on import would invent a schedule the user did not set. NULL rather than empty string for the unset marks — the lifecycle is defined by NULL-ness, and "" would make a never-fired reminder read as fired at "". TestMigratedTablesCoversTheExport caught the second half, which I would have missed: `pad db migrate-to-pg`'s NUL preflight decides what to REFUSE on from MigratedTables, so a table the migration copies and the preflight does not know about is a gap in exactly the guard that exists to prevent one. Added there too, with the reason it can never actually fire — every column is machine-produced, so it is listed for coverage rather than expectation — and the "six tables" prose it falsified is now seven. Two mutants, two killed: export dropping the block, and import discarding the marks. Forty-nine across the unit. * test(reminders): state the fire-path invariant and pin it from the invariant The lead's read on why rounds 4 and 7 were the same class: the fire path had no stated invariant, so each fix defended an instance. This states it, and derives the pin from the paragraph rather than from the bug history. THE INVARIANT: the candidate scan is a hint and may be assumed to prove nothing. Every condition that made a row a candidate is re-asserted inside the transaction that marks it fired, in the same statement that does the marking, so checking and writing are one atomic act. Worded as "the scan proves nothing" rather than as a list on purpose — a list invites the next person to add a condition to the scan and stop, which is exactly what happened four times here. TestFirePathInvariant is the pin: one table, one row per scan-side condition, each invalidating that condition in the window between the scan and the fire and asserting the same three things — nothing fires, no event leaves, the reminder is not consumed. The earlier per-defect tests are folded in as rows; they said the same thing one instance at a time, which is how four of these shipped. Adding a fifth condition to the scan without a row here should feel like an omission. It carries a positive control, because four cases that all assert nothing happens would pass against a build that never fires at all. The matrix immediately falsified a claim in the paragraph I had just written. I wrote that the item load inside the transaction is "for the payload, not for the check"; removing the item half of reminderFireable alone changes no observable behaviour, because the load then returns nil and the deferred rollback undoes the write. Item liveness is defended TWICE and a single-mutant experiment cannot say which guard is carrying it — removing both is what kills the test. Both are kept, the predicate is named as primary (the row never matches, so no write happens at all), and the asymmetry is stated: workspace liveness has no second line, which is why dropping ITS half does fail the pin. Six mutants: five singles plus the pair. Five killed alone; the item single survives by design and is documented as such rather than left as an unexplained green. Fifty-five across the unit. * fix(reminders): codex round 9 — one legacy row could hide every reminder **P1: items.item_number is NULLABLE and I scanned it into an int.** Migration 006 added the column to existing rows, so a pre-numbering item still carries NULL — and scanning NULL into an int fails the Scan, which fails the QUERY, which degrades the whole pending-reminder section. One old row, and the feature is dark for everyone in that workspace. ListWatchesForUser, which this query was modelled on, uses sql.NullInt64 for exactly this column. I copied its shape and dropped the part that handles the column's actual nullability — the same way of being wrong as the round-5 cmdhelp fixture: borrowing a form without borrowing what it knows. The legacy row now carries no ref rather than a fabricated "PREFIX-0", which would name a different item. **P1: export shipped reminders that import could only discard.** The items section filters on deleted_at IS NULL, so a soft-deleted item is not in the bundle and its reminder can never be reunited with it. My comment claimed the item_links rationale — round-trip the raw graph so a restore reunites them — which is true for links and false here, because links keep soft-deleted endpoints in the bundle and items do not. A link is a row ABOUT two items; a reminder whose item is absent is a dangling schedule. **P2: import wrote remind_at raw.** Import is a writer, and a bundle is not necessarily one this server produced — hand-edited, or from another instance. A local offset or a bare date would land in the one column every comparison downstream treats as a UTC instant, firing early, late, or never. It now normalizes like every other door. An unparseable value is SKIPPED with a warning rather than failing the restore, matching the lenient import-side precedent already in this file, and the raw value's LENGTH is logged rather than its content. Three mutants, three killed; two needed rewriting because the single-line form did not compile — reverting the nullable scan also requires reverting the render, and dropping the normalization orphans a variable. PROCESS FAULT, recorded because it makes this round's findings weaker than they look: I edited the tree while this review was reading it — committed the invariant work and ran five mutation experiments, which write and restore source, over the same files. A review binds to the tree it read and I moved it underneath. Every finding above was re-verified against the current tree before being acted on, and the next round runs with no concurrent edits. * fix(reminders): codex round 10 — one orphaned item aborted a whole restore An ORPHANED item — one whose collection is missing from the bundle — still gets an itemMap entry. It has to: the entry is written before the skip because parent resolution inside the same loop reads the map for items it has not reached yet. So `itemMap[x] != ""` is satisfied by an id that names no row, and inserting a foreign key to it fails (SQLite enforces FKs here via the DSN's `_pragma=foreign_keys(on)`; Postgres always does). The pre-existing mapping is the sharp edge. The aggravating half was mine: this loop treated a failed reminder insert as FATAL, where item_links and item_versions both skip, so one orphaned item carrying a reminder rolled back an entire 900-item workspace restore. A reminder is the least critical thing in a bundle and it had the strictest failure handling in the file. Both halves fixed: the loop gates on items that actually landed, and a failed insert warns and skips like its siblings. TWO GUARDS THAT ONLY DIE TOGETHER, and this is measured rather than assumed. Reverting either alone leaves the test green — with the map gate restored the skip survives the FK failure, and with the fatal return restored the gate means the insert never fails. Removing both is what fails it. They are kept as a pair because they defend the same failure at different depths (prevent the bad write / survive a bad write arriving some other way), and the pair is recorded in the code so a future reader does not delete one as dead after watching its mutant survive. Second time this shape appeared today; the first was item liveness on the fire path. The bundle in the test is hand-built, because ExportWorkspace cannot produce an orphan — which is the reason it needed a test. That shape only arrives from a hand-edited or foreign bundle, and surviving those is what import is for. Three mutants: two singles that survive by design, plus the pair that kills. Sixty-one across the unit. * fix(reminders): codex round 11 — four contract slips, one of them another unit's **suggested_next returned up to eight entries against a cap of three.** Round 3 prepended reminders PAST the list's own cap, reasoning they should not compete for slots. Every consumer — the web dashboard, `pad project next`, `pad project ready` — is written for three. Worse, it silently falsified a decision recorded elsewhere: BootstrapDashboard deliberately has no suggested_next_overflow_count BECAUSE this list is capped at three upstream, and its comment names raising that cap as the moment to add one. My change made another unit's reasoning wrong in a file I never opened. The combined list is now trimmed back to three, reminders still leading — a reminder can push a task suggestion out, which is the right way round, and the full set stays addressable in pending_reminders. My first version of that trim used `limit`, which is REASSIGNED above to len(candidates) — so on a workspace whose only entries are reminders it would have truncated to zero, killing precisely the case the surface exists for. Caught by reading the surrounding lines before running anything; it has its own test now. **pending_reminders was uncapped in the bootstrap projection.** BootstrapDashboard embeds *DashboardResponse, so every new field joins the boot payload automatically — here, a window of up to 50, which is the budget PLAN-1410 spent a unit trimming. Capped at 5 with an overflow count, under its own constant rather than borrowing bootstrapAttentionCap: they answer different questions and a future change to one must not silently move the other. **Truncation was reported from the wrong question.** The collector used the store's `more` flag, which answers "is there another PAGE", not "did I read all of THIS one" — so a window filling part way through the final page reported that the caller had seen everything while unread rows sat behind the fill point. The paging bounds are now injectable so the case is testable at all: building it with a window of 50 needs ~75 rows in a specific pattern, with a window of 3 it is four. **Import accepted acked-without-fired**, which is not one of the lifecycle's three states. Such a row fires, is excluded from the pending surface because it is already acked, and can never be acknowledged because AckReminder requires acked_at IS NULL — an event emitted into permanent invisibility. The acknowledgement is dropped and the schedule kept, since an ack of something that never fired means nothing. Five mutants, five killed (one rewritten — removing the flag orphans a variable). Sixty-six across the unit. * fix(reminders): codex round 12 — a read is not a hold; scope the arm; ack from the ack Four P2s from round 12 (two independent runs, both landing on the same line of the fire path), each closed at the layer where it lives: - fireOneReminder pins the item and workspace rows FOR NO KEY UPDATE on Postgres before the arbiter UPDATE. reminderFireable re-asserted liveness at the predicate's instant and nothing held it to the commit instant; under READ COMMITTED an archival could commit in between and the event left the process about a deleted resource. Same idiom and same lock strength as CreateAttachmentForLiveItem; SQLite is excluded by its BEGIN IMMEDIATE, not skipped for convenience. Two PG-only pins verify "blocked" in pg_stat_activity, not by elapsed time; the pin-removed mutant fails both. - CreateReminder asserts "live item of THIS workspace" in the INSERT's own SELECT and returns ErrReminderItemGone otherwise. The table had an FK and no same-workspace constraint; a mismatched pair fed another workspace's title to this one's dashboard and webhooks. Handler maps it to 404. - AckReminder matches every fired row (COALESCE keeps the first ack, updated_at moves only when acked_at does), so a no-match means exactly "not fired at the instant of the ack". The handler no longer decides 409-vs-200 from the row it read before the UPDATE. - The invariant paragraph gains its missing sentence: "at that instant" means the commit instant, and the pin is what makes the predicate's instant and the commit instant the same one. Round-12 caveat carried: both runs were static reads (sandbox blocked Go's build cache), so "four" is a floor, not a measurement. Refs IDEA-2641 * fix(reminders): codex round 13 — a reminder's workspace must agree with its item's, at every read Every reader scoped by r.workspace_id and then joined the item without asserting the two agree. No door writes a disagreeing row today (CreateReminder derives the pair from the item; import maps within the workspace), and the table has nothing that forbids one — so a hand-edited bundle, a future move door, or a direct write would carry one workspace's item into another's dashboard, export, and webhooks. The identity goes into reminderFireable (scan + arbiter), the Postgres row pin, ListPendingReminders and the export query. One test writes the row raw — the only way one can exist — and asserts it is inert at each site; the predicate-removed mutant scans and fires it. Refs IDEA-2641 * fix(reminders): codex round 14 — the by-id and by-item reads assert the same identity as every other read GetReminder scoped by the row's own workspace_id and ListRemindersForItem by item_id alone, so a row whose two columns disagree — the class rounds 12 and 13 closed at the scan, the arbiter, the pin, the pending surface and the export — was still readable through the two reads that reach a single row. reminderOwned is that identity on its own, without the liveness half those two reads must not have (a fired reminder on an archived item is history worth showing). The write paths reach a row only through GetReminder, so scoping it scopes them; a row no door can write needs no door to delete it. ListRemindersForItem now takes the workspace its caller already resolved the item in. The raw-row test asserts both reads refuse the row from both sides; the reminderOwned-removed mutant surfaces it through GetReminder. Refs IDEA-2641 * fix(reminders): codex round 16 — an archived item's reminders are readable, and its verbs say "archived" The doors resolved the item live. Listing an archived item's reminders answered 409 from a GET, and ack/re-arm/delete answered a bare 404 for a reminder that exists on an item that exists — while the store, since round 14, deliberately keeps that history readable. The API already has a posture for archived items: GET reads them, mutations answer 409 "archived … restore it before editing" (writeItemResolveError). The list now follows handleGetItem; the lifecycle verbs load the item include-deleted, run the visibility check first, and then answer the same 409 every other item mutation does. One test walks archive → list 200 / ack 409 / arm 409 → restore → ack 200 on the same rows. Refs IDEA-2641 * fix(reminders): codex round 17 — one suggestion per item, the archived 409 by slug, and the door courtesy named Three findings on the server pass. (1) An item that was both a fired reminder and an ordinary candidate appeared in suggested_next twice; the ordinary entry is dropped, the reminder entry (which carries the ack id) stays, and two reminders on one item remain two entries. (2) Round 16's 409 for an archived item's reminder was written by re-resolving item.Ref, which is derived and empty for a legacy item with no item_number — so the class most likely to be legacy fell through to a bare 404. The slug is handed over instead. (3) The archived check in resolveReminderForWrite is check-then-write, and an archive landing in between lets the verb through: accepted and documented — it is the posture of every item mutation here (UpdateItem's UPDATE has no liveness clause), the outcome is benign, and putting liveness in AckReminder's WHERE would re-create the no-match ambiguity round 12 removed. Refs IDEA-2641 |
||
|
xarmian
|
e94e9afbea |
Merge pull request #1240 from PerpetualSoftware/fix/bug-2850-field-coercion
fix(server,mcp,cli): type field values server-side; carry the fields object natively (BUG-2850) |
||
|
xarmian
|
9ecc59af1e |
fix(store): refuse a schema with trailing content instead of truncating it (BUG-2873)
Codex round 3, one P2. `json.Decoder.Decode` stops at the end of the FIRST value and ignores whatever follows, where `json.Unmarshal` refuses it — so a stored schema with junk after the object would be silently truncated by the rewrite. It is now treated as unparseable and left alone, the same posture as any other schema this migration cannot faithfully reproduce. **The Postgres gate then failed the new test, and the failure is the finding.** It failed at the SEED, not the assertion: `ERROR: invalid input syntax for type json (SQLSTATE 22P02)`. `collections.schema` is TEXT on SQLite (`005_collections.sql:10`) and JSONB on Postgres (`pgmigrations/001_initial.sql:114`), so a value with trailing content cannot be STORED on Postgres at all. The state this guard defends against is reachable on one dialect and forbidden by the column type on the other. So the test skips on Postgres with that reason recorded. Asserting there would be asserting about a state that cannot exist — and reading WHICH LINE failed is what separated "my test is not portable" from "the product is broken on PG". **Second instance of the mutation harness reporting a false survivor**, same cause as the last: deleting the guard leaves `io` unused, the mutant fails to compile, and counting `--- FAIL` lines sees zero. With `_ = err` in place of the return it dies immediately. Twice in one unit makes it a harness defect, not bad luck: a runner that counts test failures must check the BUILD separately, or every non-compiling mutant reads as a hole in the tests. Mutation matrix 7 of 7 killed. Gates: `gofmt` clean, `go vet ./...` ok, full `go test ./...` green on SQLite, full `internal/store` green on Postgres (502.2s, private container at 127.0.0.1:5473, detached with a sentinel). |
||
|
xarmian
|
499387e99d |
fix(store): never regress a migrated token; keep large integers intact (BUG-2873)
Codex round 2: four findings, two fixed here and two filed as their own items.
**Migrated siblings' OCC tokens could REGRESS.** A sibling updated between this
rename's timestamp and the scan already holds a newer `updated_at`; stamping the
rename's value on it moved the token BACKWARDS — breaking the strictly-increasing
invariant the transaction above exists to maintain, and re-validating a token the
client should have lost. Each rewritten row now takes `max(current + 1ns,
renameToken)`, computed in Go from the value read under the row lock rather than
by comparing timestamp TEXT, which the existing comment warns is never safe.
**Large integers in unknown properties were corrupted.** Round 1 fixed the typed
round-trip dropping unknown keys, but decoding into `interface{}` turns every
JSON number into float64, so `9007199254740993` came back CHANGED. A rename would
silently damage a property it exists only to carry through. `UseNumber` keeps the
literal text.
## Filed, not absorbed — both because their dependents are not the relation feature
- **BUG-2875** — a collection CREATED during a rename escapes the scan's
`FOR UPDATE` and keeps a relation aimed at the old slug. Closing it means
`CreateCollection` takes the workspace lock, which changes the concurrency
behaviour of every collection creation on the instance. Same reasoning that
split IDEA-2874 out; this unit's reviewability rests on affecting zero live rows.
- **IDEA-2876** — migrated siblings emit no `collection_updated` event, so an open
page keeps the pre-rename schema until reload. Handler/event layer; the store
publishes nothing.
## The mutation harness was reporting a false survivor
Counting `--- FAIL` lines treats a mutant that FAILS TO COMPILE as one that
survived — zero failures either way. Removing the token guard leaves
`rowUpdatedAt` and `renameToken` unused, so that is exactly what happened, and it
read as "the guard is untested". With a compiling mutant (`_ = rowUpdatedAt`) it
dies immediately. Worth stating because the failure mode is silent and points the
wrong way: it invents doubt about code that is fine, and would equally hide a
real survivor behind an unrelated build break.
Mutation matrix 6 of 6 killed.
Gates: `gofmt` clean, `go vet ./...` ok, full `go test ./...` green on SQLite,
full `internal/store` green on Postgres — 460.9s, private container at
127.0.0.1:5473, run detached with a sentinel.
|
||
|
xarmian
|
6428e7db31 |
fix(store): lock, stamp and preserve on the relation retarget (BUG-2873)
Codex round 1: four findings, three P1, all real. **The migrated siblings' concurrency token was not advanced.** `collections.updated_at` doubles as the OCC token (BUG-2265), so rewriting a sibling's schema without touching it left a client holding the PRE-rename schema — and a token that still matched — able to write it straight back and undo the migration. Every rewritten row now takes the rename's own token, so the whole rename shares one instant. Pinned by asserting the stale token now 409s. **The scan did not lock the rows it rewrites.** A concurrent schema update to a sibling could commit between the SELECT and the UPDATE, and this transaction would then overwrite the newer schema with its stale copy. `FOR UPDATE` on Postgres, ordered by id so the multi-row acquisition is deterministic; SQLite is covered by its BEGIN IMMEDIATE write lock. **The old slug came from the pre-transaction snapshot.** Two tokenless concurrent renames of the same collection both read the ORIGINAL slug outside the lock; the loser would migrate `original -> its own new slug` while the relations already said the WINNER's, matching nothing and stranding them at a name no collection holds. The slug is now re-read alongside the token under the row lock. This is the READ — the ALLOCATION of the new slug is still outside the transaction and still IDEA-2874's, deliberately. **Re-marshaling through `models.CollectionSchema` dropped unknown properties.** That struct has fixed fields, so unmarshal+marshal silently erased anything it does not declare — a rename would quietly strip forward-compatible metadata from every relation-bearing schema in the workspace. It now edits the raw decoded JSON, touching only `fields[i].collection`. ## Two instruments that were not instruments Both found by mutation, not by reading: - **The deadlock test passed against its own mutant in 0.44s.** Two unsynchronised goroutines never collided. With a start barrier and 40 rounds it now fails in 1.4s with `ERROR: deadlock detected (SQLSTATE 40P01)` — so Rook's hazard was reproducible, not theoretical. - **The pre-tx-slug mutant survived the first matrix**, because nothing forced the interleaving. Rather than call it untestable, it is pinned by an end-state invariant that holds under ANY interleaving — whatever slug the collection ends up with, every relation aimed at it points there — over 40 concurrent rounds. It fails at round 1 under the mutant. Mutation matrix 4 of 4 killed. Gates: `gofmt` clean, `go vet ./...` ok, full `go test ./...` green on SQLite, and the full `internal/store` suite green on **Postgres** — 448.6s on a private container at 127.0.0.1:5473, never the shared 5445 a sibling seat may tear down. Run detached with a sentinel after the first attempt was killed at a turn boundary; the harness kills backgrounded tasks, it does not kill disowned ones. |
||
|
xarmian
|
4687c46f94 |
fix(store): migrate relation fields when their target collection is renamed (BUG-2873)
`models.FieldDef.Collection` holds the target's SLUG, and it is the ONLY pointer a relation field carries — there is no id beside it to fall back on. `UpdateCollection` re-slugifies on rename and nothing migrated the definitions aimed at the renamed collection, so every relation field pointing at it was stranded: the picker filters on a slug that resolves to nothing and the field silently stops being fillable. `retargetRelationFieldsTx` re-points them in the SAME transaction as the rename, for the reason the field-value migrations already run there: a failure must roll the rename back rather than commit collections pointing at a slug that no longer exists. **It parses instead of string-replacing.** A schema's JSON contains the old slug in places that must not move — a text field's `default`, a select's `options`, a label. Only `FieldDef.Collection` on a `relation` field is a reference. Export's `remapFieldIDs` gets away with a blind replace because it substitutes UUIDs, which cannot collide with prose; a slug is a word. A control test pins that. **The renamed collection is included deliberately** — a relation targeting ITSELF needs the same rewrite — and the rewrite lands after the caller's own `schema` write in the transaction, so a simultaneous schema edit composes rather than being reverted. Both have tests. ## The deadlock hazard, and why the existing comment does not cover it The lock-order comment above this transaction is a Codex P1 fix that orders the workspace lock against ONE collection row lock, because until now nothing took more than one. This change writes SIBLING collection rows, so two concurrent renames of mutually-referencing collections take those locks in opposite orders. **Reproduced, not theorised:** with the serialization removed, the test fails in 1.4s with `ERROR: deadlock detected (SQLSTATE 40P01)` on Postgres. Renames now take the workspace lock — previously acquired only when `len(input.Migrations) > 0` — BEFORE the row lock, which closes it without inventing a second ordering rule to keep in sync with the first. **The first version of that test was not an instrument.** Two unsynchronised goroutines passed against the same mutant in 0.44s, having simply never collided. It takes a start barrier and 40 rounds to be evidence. ## Scope The out-of-tx slug allocation (`uniqueSlugExcluding(s.db, …)` at :420, before `s.db.Begin()` at :503) is deliberately NOT touched — filed as IDEA-2874. Its dependents are every collection rename in every workspace, not the relation feature, so it does not belong in a change whose reviewability rests on affecting zero live rows. `UNIQUE(workspace_id, slug)` makes today's behaviour loud rather than lossy, so it can wait. A census found ZERO relation fields across all 11 accessible workspaces on this instance, and no shipped template declares one — this repairs the rename path before PLAN-2857 creates the population, which is why a migration is not needed. Gates: `gofmt` clean, `go vet ./...` ok, `go build ./...` ok, full `go test ./...` green on SQLite, and the full `internal/store` suite green on **Postgres** (private container on 127.0.0.1:5473, never the shared 5445 a sibling seat may tear down). Pin written and run BEFORE the fix per team CONVE-29: 2 propagation tests failed, the control passed. |
||
|
xarmian
|
ae793e6fa6 |
fix(server,store,items): coerce field values to their declared types server-side (BUG-2850)
The write doors disagreed about what `key=value` means. The CLI has coerced by schema type since BUG-1125, and local stdio MCP inherits that by shelling out to the binary — but the remote /mcp transport builds its field map in ingestFieldKVP with `dst[key] = val`, so every value arrives as a string. validateFieldType then correctly refuses a string for a declared number or json field, and the net effect was that an MCP agent on that transport could not write those fields AT ALL: every attempt a 400, not a mis-typed value. Measured before writing anything (repro table on BUG-2850's trail): CLI and stdio MCP store 42 and an array; the HTTP door 400s on both; an UNDECLARED key is stored as a string on every door. items.CoerceFields(fields, schema) converts strings to the declared type — number via ParseFloat (NaN/±Inf refused, because json.Marshal cannot encode them and the ignored downstream error would silently drop the whole payload), json/multi_select via Unmarshal, checkbox via ParseBool — and is applied immediately before every Validate* call. Three deliberate non-behaviours, each with a test: - A value that will not parse is left as the string for the validator, so the existing "must be a number" error still fires. Coercion invents no error path, and cannot turn a currently-PASSING write into a failure. - Non-string values pass through untouched; an int stays an int. - Text-typed fields holding "42" stay strings. Coercing anything that parses would retype real data while fixing the bug. Not folded into ValidateFields, though that would be the single call site: a function named Validate that mutates its input is a trap, and two callers re-marshal the map they pass. THE POPULATION IS 8 CALL SITES, and finding them took two sweeps. The first was scoped to internal/server and found 7; the copy path validates in internal/store (items_cross_workspace_copy.go), which only a repo-wide sweep sees. The preflight and the store-side copy now carry cross-references to each other: the preflight exists to PREDICT the copy, they live in different packages, and that is exactly how they would drift unnoticed. The undeclared-key half of BUG-2850 is untouched and marked as a decision point in CoerceFields — refuse/warn/keep is with Dave. A test pins today's keep behaviour so the ruling lands as a deliberate change. The CLI's parseFieldFlag deliberately STAYS: it is why two of four doors are correct today, and removing it alongside its replacement would put all four at risk of one mistake. Retiring it is a follow-up. Claude-Session: https://claude.ai/code/session_011Q4b1iHtJtSyMs7BA2ySxo |
||
|
xarmian
|
e89c8c8ab6 |
fix(cli): two ways the preflight and its remedy disagreed with the migration (BUG-2810)
Codex round 9, both confirmed against the code rather than reasoned about. **PAD_DATABASE_URL was treated as proof of a PostgreSQL deployment**, so the flow this unit prescribes broke on itself. cmd_server.go opens PostgreSQL only when PAD_DB_DRIVER=postgres; PAD_DATABASE_URL is ALSO migrate-to-pg's target, and its default. An operator who follows the preflight — refused, told to run `pad db repair-nul`, with the target URL still exported in their shell — got "This deployment is PostgreSQL ... Nothing to scan or repair" and exit 0. The remedy the refusal names did nothing, which is the failure mode this unit has now produced three separate ways. PAD_DB_DRIVER alone decides. Verified by running the real command with the target exported. **The preflight refused on tables the migration does not copy.** ExportWorkspace / ImportWorkspace read six tables, and migrate-to-pg's own help says users, platform settings and auth data are not migrated — so a NUL in users.name blocked a copy that would never touch it, demanding the operator rewrite content unrelated to the migration they asked for. Refusal is now filtered to store.MigratedTables(). Those rows are still REPORTED: `pad db scan-nul` lists them, they are real, and going quiet about a broken row because this command does not care about it would be the information-discarding the preflight was already corrected for once. The table set is pinned by REFLECTION over models.WorkspaceExport's shape, not by a regex over ExportWorkspace's SQL — TASK-2825 already established that multi-line and Sprintf-composed SQL are invisible to any source-level instrument. It fails in both directions: a new export section with no entry (a miss, ending in a half-finished migration) and a spurious entry (an over-refusal). One residual, stated rather than hidden: the export also skips SOFT-DELETED collections and items, and this filter is per-table. A NUL in a soft-deleted item still blocks. Narrowing it needs a per-row deleted_at check at every candidate, which costs more than the remaining over-refusal — the operator's way out is the same single command either way. |
||
|
xarmian
|
d86a499f56 |
fix(store): the SQLSTATE extractor indexed one string and sliced another (BUG-2810)
Codex round 7. sqlStateOf searched strings.ToUpper(msg) for the marker and then sliced the ORIGINAL message at that offset. Correct only while every byte before the marker is ASCII: Unicode case mapping changes byte LENGTH for some runes, and PostgreSQL renders messages in lc_messages, so a non-English server is not a hypothetical. One string now serves both the search and the slice, which also makes the returned code uppercase without a second conversion. Mutation-verified rather than argued: against a message carrying U+0131 (two bytes, uppercasing to a one-byte "I") the old code returns "TE 22" where the code is "22P05". That garbage happens to classify as unavailable — the safe direction — but only by luck; a different offset lands on a spurious "22" prefix and turns a check that never completed into a verdict about the value. The regression leg uses a localised message shape for that reason, and the failure it produces is the one above. |
||
|
xarmian
|
6d4c3b4b75 |
fix(store): an operational SQLSTATE is not a verdict about the value (BUG-2810)
Codex round 6, and it is the round-5 fail-open one level deeper. That round split "the server answered" from "the server did not", and I implemented the first half as "does the error carry a SQLSTATE at all" — which is wrong, because 57014 (query cancelled), 57P01 (terminated by administrator), the 08 class (connection exception) and the 53 class (out of resources) all carry SQLSTATEs while saying nothing whatever about the value. Classified as verdicts, they let the preflight proceed with an UNVERIFIED suspect, which is the exact thing the three-way split was added to stop. The test is now INVERTED: only SQLSTATE class 22 — data exception, PostgreSQL's class for "this value is wrong" — counts as a verdict about the value. `SELECT $1::jsonb` produces 22P02 for malformed JSON and 22P05 / 22021 for the NUL cases. Everything else, code or no code, means the question was not answered, and the caller refuses rather than guessing. Erring toward "unavailable" is the safe direction: its cost is a refused migration an operator re-runs, against a half-finished one they have to unpick. The coverage is split deliberately, and both halves are needed. The operational codes are from PostgreSQL's error-code table, formatted the way pgx renders them, because provoking an administrator shutdown inside a unit test is not worth it. What is NOT assumed is the rendering, or the premise that class 22 is what a bad value yields: the real-server test now extracts the SQLSTATE from a genuine malformed-value rejection and asserts it is class 22 and a completed verdict, and the closed-pool test covers the no-code path. Neither half stands on its own. sqlStateOf's own edges are pinned too — a truncated "SQLSTATE 22" must not yield a partial code that then matches a class prefix, and the marker search being case-insensitive means the extraction has to be as well. |
||
|
xarmian
|
0363c139a9 |
fix(store,cli): the oracle failed open, and it over-refuses one column (BUG-2810)
Three findings from codex round 5, all real; the third corrected a claim I had
made about the design.
**The suspect path could leave data unrepaired and exit 0.** The CLI printed
SuspectsFailed and then returned nil, checking only the violation bucket. A
script sees success; an operator who trusts the status moves on. Both buckets
now decide the exit code, and the decision is extracted into
nulRepairExitError so it is testable without a database — the bug was in the
decision, not in the repair, and a test that needs a fixture to reach it is a
test nobody writes.
**The destination oracle failed open.** Connection failures, timeouts and
read-back errors were bucketed with "the destination answered, about something
else" — reported and not refused on. So an UNVERIFIED suspect passed the
preflight, which is the defect the suspect class was added to correct arriving
by a different route.
There are now three outcomes rather than two: the server answered with a NUL
code (refuse), the server answered with another complaint about the value
(report, because a NUL preflight that quietly grew into a general one would
block migrations unrelated to this bug), and the server never answered
(REFUSE). ErrDestinationCheckUnavailable carries the third, and
TestDestinationOracleFailsClosedOnAnUnusableConnection pins it against a real
closed pool — with an open-pool control first, since a classifier that answered
"unavailable" for everything would satisfy the assertion and refuse every
migration.
**The oracle is not a perfect model of the migration, and I said it was.**
Codex claimed workspaces.settings is normalised on import, so the cast
over-refuses there. Measured rather than argued, by importing the same
shadowed-duplicate value into three columns against a real server:
workspaces.settings -> import SUCCEEDS, stored as {"a": "clean"}
items.fields -> import FAILS, SQLSTATE 22P05
collections.schema -> import FAILS, SQLSTATE 22P05
CreateWorkspace runs models.NormalizeWorkspaceSettings, a map round-trip that
drops the shadowed member. So the claim was right, and my own runtime demo
earlier on this branch — which used workspaces.settings — was showing a
spurious refusal.
The cast STAYS. That row is a value Layer B refuses on every write today and
exists only because it predates enforcement, so surviving the migration is an
accident of one column's normaliser rather than a property worth preserving,
and repair-nul clears it in one command. Deriving "would this column's writer
normalise it" is a per-column enumeration, which is the shape this cluster
keeps proving unmaintainable.
What changed is the CLAIM. The file header no longer says the oracle is "exact
in both directions" — it is exact about the VALUE and is not a model of the
MIGRATION; the refusal no longer tells an operator PostgreSQL would reject the
row, only that the value carries a NUL jsonb refuses; and the measurement and
the over-refusal are written into CheckJSONBAcceptable's doc comment and
docs/backup.md, which also now states that the check errs toward refusing.
The disposition is flagged to the lead rather than settled here: skipping
normalised columns is a scope call, not mine.
|
||
|
xarmian
|
57b7ca5f48 |
feat(store,cli): ask the destination about suspects instead of dropping them (BUG-2810)
Day-54 lead ruling on PR #1233, and the ruling names the defect precisely: the scan's own SQL pre-filter already surfaces the shadowed-duplicate row as a candidate, and `ParameterRefused` then drops it. So the preflight was discarding information it was holding and going on to promise the migration would go through. I had recorded that as an accepted residual on the grounds that closing it would violate DOC-2823's one-layer rule — but that rule is about what the enforcement layers REFUSE. It says nothing about a preflight throwing away a candidate it had in hand. **The SUSPECT class.** A pre-filter hit the predicate does not refuse. Most are doubled-backslash literals — text that writes ABOUT the escape, which is the false positive this whole predicate family exists to avoid. One member is not: a NUL in a value shadowed by a LITERAL duplicate key, which a map-model decode drops and PostgreSQL refuses. Nothing here can tell them apart, so nothing here tries: `pad db scan-nul` lists them under their own heading, apart from the violations, with what resolves them. **The destination is the oracle.** `pad db migrate-to-pg` casts each suspect on the TARGET connection — `SELECT $1::jsonb`, side-effect-free, and the very cast an INSERT performs — and refuses on 22P05 / 22021. That is exact in both directions precisely because it is not a fourth opinion of ours. Measured against a real server: the literal is ACCEPTED, the shadowed duplicate is REFUSED with 22P05, and a non-JSON value fails for a reason that is reported rather than refused on, because a NUL preflight that quietly grew into a general one would block migrations unrelated to this bug. **The repair had to be measured, not assumed, and the answer changed the design.** `textguard.Repair` leaves the shadowed value completely untouched: its scanner is gated on DocumentDecodesNULAnyShape, a map-model question that answers false for exactly this shape, so it never runs. A preflight that refused the row and printed `pad db repair-nul` would have been printing a command that does nothing to it — a remedy nobody ran (PATTE-135). So the repair reaches the class through the token-level scanner, exported for this, which rewrites the shadowed escape and still leaves the literal byte-identical because it consumes escapes in order. Suspects get their own buckets in the repair report rather than being folded into Repaired, so the dry run's promise and the run's result stay the same number. **Nothing about what any layer REFUSES changed.** textguard.KnownGaps and its pin are untouched, and TestScanNULInheritsTheRecordedKnownGaps still asserts the scan does NOT detect the shape. TestSuspectsCollapseWhenBUG2812Lands fails when the token-walk makes that false, and names every file to delete — the suspect path is a second mechanism that exists only while the predicate is blind. **One defect this found that no test did.** Running the real command against a real Postgres, the refusal announced "0 stored value(s) carry a NUL; nothing was migrated" while listing one — the count used the violations only, and the tests asserted the message CONTAINED "nothing was migrated" without reading the number. Fixed, and the assertion now reads the count. The whole loop is now verified end to end: preflight refuses, `repair-nul` fixes, the migration completes. My own prose from earlier on this branch is corrected with it. ScanNUL's doc comment, the preflight's, and docs/backup.md all said this shape passes the preflight and fails mid-copy, which the same commit makes false. |
||
|
xarmian
|
178b6b5010 |
fix(server,store): two more from codex rounds 3 and 4 (BUG-2810)
**The import repair could silently change what gets imported.** It decodes into map[string]any, where a repeated object member keeps only the LAST value. The TYPED decode that runs next does not agree: encoding/json unmarshals members in order into the same struct field, so two `"workspace"` objects MERGE there and collapse here. A body with duplicate members would therefore import differently with --repair-nul than without, which is outside what a flag by that name may do. It now DECLINES such a body: returns it untouched, lets the gate judge it exactly as it would without the flag, and says why in the refusal — "the payload repeats the member X, and repairing it would change which value is imported". Detection is a token walk, because a decode is what loses the information: by the time there is a map the duplicate is gone. The detector's own test carries the false positive that matters — the same member name in SIBLING objects is not a duplicate, and a single shared set of names would decline every real export, since items all carry `id`, `title`, `slug`. Rewriting such a body faithfully wants a token-preserving pass, which is BUG-2812's token-walk and not a rider on this. A real export cannot contain duplicate members (json.Marshal does not emit them), so declining costs nothing an operator meets by accident. The tally now owns the repair — decodeJSONRepairingNUL takes it and calls Apply — so the count and the declined reason come back through one object instead of a return value a caller has to remember to record. That is the same mistake this branch already made once, when the JSON path dropped the count and the header reported 0 for an import that had rewritten a value. **A row the repair could not address was reported as a failure.** A NUL in a key column the list does not protect, on a row whose violation is elsewhere, makes the address unbindable: Layer A inspects every bound parameter, including a WHERE clause's, so the lookup is refused before SQLite is asked to find the row. It landed in Failed carrying "invalid text parameter: parameter 2" — the same information phrased as a fault in the repair rather than a property of the row. Now detected up front and reported as a skip with the reason, alongside the two skips that already existed. **One finding NOT fixed, deliberately, and recorded instead.** Round 3 raised that the scan misses a NUL in a value shadowed by a LITERAL duplicate key, so such a row passes the migrate-to-pg preflight and then fails during the copy — the exact failure the preflight replaces, surviving for one shape. That is textguard.KnownGaps: a blind spot every layer shares on purpose, which DOC-2823 forbids closing in one layer alone, because layers disagreeing about one value is the defect this cluster is made of. So it is named in ScanNUL's doc comment, in the preflight's, and in docs/backup.md for the operator, and TestScanNULInheritsTheRecordedKnownGaps pins the miss and FAILS when it stops being one — the notification that BUG-2812 has landed and those three prose sites need updating. The consequence is recorded on BUG-2812's trail. Round 2's single finding was refuted rather than fixed: it predicted TestRepairFlagReachesTheNestedAndObliqueForms would fail, on a mechanism that describes the raw-byte scanner this branch had already replaced. The test passes; the outer decode resolves the oblique spelling before the walk sees it. |
||
|
xarmian
|
49bd342e4c |
fix(store,server,cli): three defects from codex round 1 (BUG-2810)
**The import flag could not repair the column it exists for.** `--repair-nul` scanned the RAW body for a live escape, which is right for a value the gate reads at the top level and wrong for the one that actually matters. An item's `fields` blob travels through an export as a STRING: a NUL escape in the stored blob marshals into the body with a DOUBLED backslash, which a raw scan must leave alone because at that layer it is literal text — while the gate refuses it anyway, since it decodes the body and re-parses that string as the document it is. So the repair now walks the DECODED body with the same classing bodyDecodesNUL uses, one verb changed: where the gate asks textguard whether a value decodes to a NUL, this asks textguard to repair it. Two walks of one shape in one package is a real risk, and the mitigation is that they are measured against the same corpus in both directions rather than reviewed for similarity — TestBodyRepairMirrorsTheGateOverTheCorpus drives every case through the body shape and asserts refused-becomes-accepted and accepted-stays-byte-identical. Two consequences worth stating. The walk also reaches the OBLIQUE spelling — the backslash written as its own escape, so the six characters never appear in the raw bytes at all — which the scanner could not, so the test that pinned that limit is replaced by one asserting the capability. And re-encoding is now possible, so it is bounded: UseNumber, so an integer wider than float64 is not silently re-emitted in scientific notation; SetEscapeHTML(false); and a body with nothing to repair is returned byte-identical rather than round-tripped. The mutation that removes UseNumber turns 9007199254740993 into ...992, and a test says so. The header is now X-Pad-Repaired-NUL-Values, because at the decoded layer an escape is not a thing that exists any more and one nested document may have carried several. **The scan could not run on the databases it exists for.** Several protected tables carry a NULLABLE workspace_id — activities, api_tokens, mcp_audit_log — and the scan selected it into a plain *string, which fails with "converting NULL to string is unsupported" and takes the scan, the repair and the migrate-to-pg preflight down with it. Every column is now scanned as sql.NullString: SQLite also permits NULL in a declared PRIMARY KEY that is neither INTEGER PRIMARY KEY nor NOT NULL, which no other engine does, and a NULL key cannot address a row for an UPDATE — such rows are reported and skipped with the reason rather than handed a WHERE that matches nothing. Verified against the unfixed code: the scan returned `scan activities.actor row: sql: Scan error ... converting NULL to string`. It needed a VIOLATING row in such a table, which is why every fixture that planted its rows in `items` missed it. **--force by accident.** The repair skipped the running-server check whenever --from was given — and the most natural --from an operator types is the path `pad db scan-nul` just printed, which IS the live database. The check is now on the resolved path (Abs + EvalSymlinks, so a symlinked data directory or a relative path still matches), and a --from naming an unrelated backup stays unguarded, which is correct: nothing is writing it. The ordering moved with it. `store.New` runs pending migrations, so the refusal now happens BEFORE the database is opened; opening first and refusing second made the guard arrive after the thing it guards against. |
||
|
xarmian
|
63da2f4f5f |
feat(store,server,cli): count and repair the legacy NUL population (BUG-2810)
Layers A and B stop the value being written. Neither makes a row that
already carries one go away, and BUG-2810's filing is what that costs: an
affected workspace exports with a 200 and re-imports with a 400, so a
self-hoster restoring their own backup is blocked with no path forward in
the product, and `pad db migrate-to-pg` fails partway through the copy
against PostgreSQL's jsonb parser rather than up front.
This is DOC-2823's S3, on Dave's day-54 rulings: U+FFFD as the replacement,
repair standalone only with a migrate-to-pg preflight that refuses and
prints the command, `--repair-nul` on import shipping default-strict.
ONE REPAIR, beside the one predicate. textguard.Repair lives next to
ParameterRefused because four layers that agree about what is REFUSED and
disagree about what a repair PRODUCES is this bug family arriving one step
later. Its contract is a property over the same corpus, in both directions:
every refused value becomes one all four layers accept, and every accepted
value comes back IDENTICAL. The second half is the load-bearing one — a
repair that tidies values nobody complained about rewrites
`{"a":"x\\u0000y"}`, six literal characters after a doubled backslash, and
corrupts it.
The JSON arm is a string-literal SCANNER, not decode-walk-remarshal, which
is what the recon write-up proposed before it was written. Re-marshalling
changes four things nobody asked to change — object key order, insignificant
whitespace, integers wider than float64, HTML-ish characters — and silently
drops one of a document's LITERAL duplicate keys, which is a gap BUG-2812
owns and the last thing a repair should do. Scanning copies every byte it
does not deliberately rewrite, so an untouched document is byte-identical
without that having to be argued. A substring replace is not equivalent and
the test that proves it took a mutation to find: a doubled-backslash literal
ALONE never reaches the scanner, so the discriminating fixture is one
document carrying a live escape AND a literal.
THE COUNT IS COMPUTED IN GO. Measured on the read path in this worktree: a
row planted with `bad<NUL>name` reads back into a Go string with all 8 bytes
and the NUL intact, while `length(name)` in the same database answers 3.
TASK-2824 found that C-truncation and concluded no DB-side REPAIR could be
trusted; the same measurement on the read path says no DB-side COUNT can be
either. SQL narrows — `instr(col, char(0))`, plus the escape prefix on
JSON-classed columns, which is textguard's own pre-filter — and never
decides. The decision stays ParameterRefused with isJSON from the shared
86-column list, i.e. Layer B's classing.
Row addressing is read from the live schema rather than a hand-kept map:
39 tables carry protected columns, one (item_wiki_links) declares no primary
key and is addressed by rowid, two have composite keys, and five have a
single key that is not `id`. The repair checks RowsAffected because an
address that stopped selecting its row would otherwise commit an UPDATE that
touched nothing and report it as repaired — the one failure an operator
cannot see in the output.
`email_optouts(email)` is both a protected column and its own primary key.
Repairing it changes the row's identity and can collide with an existing
row, which in that table means somebody starts receiving mail again. It is
reported and skipped, with the reason.
The import flag is NOT an exemption from the gate. `--repair-nul` buys the
body one repair attempt and then runs the same `bodyDecodesNUL` on the
repaired bytes, which still decides — a decode path that skipped the check
is the door BUG-2803 spent thirty rounds closing, on the endpoint carrying
the largest attacker-controlled body in the product. Only the ESCAPE form is
repaired: a raw NUL byte makes the document invalid JSON, and widening what
parses is not this flag's job. Both doors are covered, JSON and tar.gz,
because giving them different answers is how one of them keeps being
forgotten.
Postgres is settled with evidence rather than sent up as a ruling: it cannot
hold either defect (22021, 22P05) and the four-way differential test already
pins that, so the scan reports not-applicable WITH the reason rather than
returning a zero a reader could mistake for a clean database.
Spellings settled here, per the dispatch: `pad db scan-nul` and
`pad db repair-nul` as siblings rather than `repair --nul`, matching
`migrate-to-pg`'s hyphenated compound — a repair verb that errors when given
no flag is a worse shape, and there is no second repair to share it with.
scan-nul IS the dry run, so repair-nul grows no --dry-run. It refuses while
the server is running unless --force, on the `pad db restore` precedent: the
report is a claim about a database, and one somebody else is concurrently
writing makes it a claim about a moment that has passed.
docs/backup.md's section on this is rewritten. It still said the rule lives
in the binary and not the database, which S2 made false, and it pointed at
this item for a preflight and a repair that now exist. Its import examples
also showed `pad workspace import < file`, which has never worked — the file
is an argument.
Closes BUG-2810.
|
||
|
xarmian
|
f54a0e41d4 |
docs(store,server): four comments and one log line that had stopped being true (BUG-2827)
Codex round 6. No logic finding; it confirmed the refusal ordering, the
split-budget claim and the first-tick scan as sound. Five statements in
the branch's own prose were untrue of the code as it stands:
- MaxOutboxPayloadBytes' comment still argued from 64 MiB ("two orders
below both ceilings") after the constant became 128 MiB, which is 4x
under the lowest ceiling, not two orders.
- maxOutboxClaimBytes' comment counted a scan-into-string-then-copy
transient that round 1 removed; the scan lands straight in []byte.
- maxOutboxClaimRows' comment used the item BODY mean (~2.4 KB) as the
payload mean; the measured payload mean is ~3.5 KB, so 5,000 rows is
~17 MiB, not ~12.
- emitBulkItemEventTx's early-out said the numbers the caller sees still
come from writeOutboxTx; when the early-out fires they come from the
projection, and the error says so in Measured.
- The drain's oversized-row log said "not claimed". OversizedPendingOutbox
filters only dispatched_at, so during a rolling upgrade a binary older
than the ceiling may be holding a claim on the row it names. The line
now states what the query establishes: this instance will not claim it.
The doc says why claimed_at is deliberately not a filter.
|
||
|
xarmian
|
0836808ca5 |
fix(store): drop a past-the-hop-bound event before judging its size, and correct three comments (BUG-2827)
Codex round 5. No production defect found; one ordering edge and three comments that had stopped being true. THE ORDERING. writeOutboxTx has two refusals that disagree about the mutation. The hop bound drops the event and lets the mutation stand, because only the cascade it would extend is illegitimate. The size cap fails the mutation, because there the mutation and the event are the same fact. An event that trips BOTH was judged for size first, so it failed a mutation over a row that was never going to be written. The hop drop now comes first. Unreachable today - nothing propagates a hop - which is exactly why the ordering is worth pinning before something does. TestAnOversizedEventPastTheHopBoundIsDroppedNotRefused fails with the two checks swapped back (run before the crash that interrupted this round, and again on this tree). THREE COMMENTS. The drain-limit constant said whole batches are claimed past it; the byte budget and row cap can now split one. The claim candidates' doc said every sibling; it is as many as the budget still allows. OversizedPendingOutbox's doc named the write cap while its query uses the claim ceiling, and said it ran every tick when the caller throttles it to once per five minutes. Gates: gofmt clean, golangci-lint 0 issues, full SQLite suite exit 0, full Postgres suite exit 0. |
||
|
xarmian
|
57caa7f92e |
docs,test(store): correct five comments and strengthen the shrink fixture (BUG-2827)
Codex round 4. Its one P1 does not hold, but the test it named as weak genuinely was, and five comments in this branch had drifted from the code they describe. THE P1, CHECKED RATHER THAN ARGUED. The claim measures size at candidate selection and never rechecks it, so a payload that GREW during a concurrent scrub could be claimed over the ceiling. The proposed growth path was Go's HTML escaping: json.Marshal writes < as its six-character unicode escape, where the source had one byte. Measured against Postgres 16: a one-key object whose value holds the four characters x<y>z&w, written with those characters LITERAL -> 16 bytes the same object written with < > and & as their six-character JSON unicode escapes instead -> 16 bytes Postgres parses the escapes and stores the characters, so the escaped and literal forms are the same size and the round trip cannot grow the row. On SQLite the payload is stored exactly as Go wrote it, so re-marshalling is idempotent. The rejection from round 3 stands, now on a measurement instead of an assertion about key removal. But the test defending it was weak, and codex was right about that: its fixture was one repeated ASCII letter, which cannot tell any of these encoder paths apart. It now carries <, >, & and non-ASCII, so it exercises the divergence rather than asserting past it. Mutation note worth keeping: a whitespace-padding mutant is caught on SQLite and NOT on Postgres, because jsonb discards insignificant whitespace — the mutant does not actually grow the stored row there. The faithful mutant adds a key, and that one dies on both. FIVE COMMENTS THAT SAID SOMETHING UNTRUE, all introduced by this branch: - OversizedOutboxPayloadError was documented as the write cap's error; three sites raise it, against two different limits. - "The two things Measured can name" listed three. - measuredStoredRow said "as the database stored it", but OctetLength measures what the driver hands back, which on Postgres is the ::text rendering rather than storage. - OversizedPendingOutbox described its threshold as the write cap while querying the claim ceiling. - ClaimPendingOutboxEvents still said batches are claimed whole, which the byte budget and row cap deliberately interrupt. Tests also now assert Measured at all three refusal sites — without it the field could be blank everywhere and every existing assertion still passes — and the claimability property test's refusal branch checks the row actually rolled back, so "refused" cannot be satisfied by a write that committed anyway. Gates: gofmt clean, golangci-lint 0 issues, full SQLite suite exit 0, full Postgres suite exit 0. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
213142c4b5 |
fix(store,server): honest refusal figures and a throttled oversized scan (BUG-2827)
Codex round 3. Two of three findings acted on, one rejected with an invariant test in place of the change it asked for. REFUSAL FIGURES SAID SOMETHING FALSE. The store refuses on two different measurements against two different limits — the member content before marshalling, against the write cap, and the row exactly as stored, against the claim ceiling — and both reported the number as "a %d-byte payload". For the first that is untrue: projectedBulkPayloadBytes is explicitly a lower bound, so the message named a size the payload did not have. A caller seeing two different numbers for one mutation had no way to reconcile them. OversizedOutboxPayloadError now carries what it measured, and both the error and the 413 say so. THE DIAGNOSTIC WAS THE MOST EXPENSIVE THING THE DRAIN DID, and it was most expensive when it found nothing. OversizedPendingOutbox has a non-sargable size predicate and no index to help it, so an empty result means evaluating octet_length over every pending row — on Postgres, detoasting and serializing each JSONB payload — and it ran every 5s tick. Now throttled to once every 5 minutes. Latency is the cheap thing to spend here: the rows it reports are permanently unclaimable and sit until the 7-day retention takes them, so a five-minute alarm delay changes no decision anyone makes about them. The first tick after a restart still scans, so an existing oversized row is reported promptly. REJECTED: that the claim needs to revalidate size, because a concurrent scrub could grow a payload between candidate selection and the claim UPDATE. It cannot. scrubOutboxRowTx is the only UPDATE of payload in the tree and it removes keys and re-marshals compactly, so a rewrite is strictly smaller — and shrinking is harmless, since a row judged claimable stays claimable. That is load-bearing for the claim needing no revalidation, so it is now stated at the function and pinned by TestScrubOnlyEverShrinksAPayload rather than left as an assumption for the next person adding a payload rewrite to break silently. The throttle test found its own gap on the way in. Written first against the helper, it stayed green when the call site was mutated to `if true` — a helper nothing calls is still correct in isolation. TestOutboxDrainTickConsultsTheThrottle covers the call site through the stamp the tick leaves behind. Also caught by the gate rather than by review: the field carrying the throttle clock landed on outboxDrainSettings as well as outboxDrainConfig, because the edit matched a line both structs have. Tests passed with both; lint named the dead one. Gates: gofmt clean, golangci-lint 0 issues, full SQLite suite exit 0, full Postgres suite exit 0. Mutants: growing the written payload kills TestScrubOnlyEverShrinksAPayload, removing the throttle call site kills TestOutboxDrainTickConsultsTheThrottle. One mutant discarded as unfaithful — corrupting the payload BEFORE the compare-and-swap is neutralised by the retry, which re-reads and redoes the work correctly, so it tests the retry rather than the invariant. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
46a551aa4b |
fix(store): refuse an outbox row on its STORED size, not its Go size (BUG-2827)
Codex round 2. One material defect, and the measurement that settled it
invalidates an argument the previous commit leaned on.
I had claimed the Postgres JSONB text expansion was bounded near 1.4x —
whitespace after colons and commas — which is why a 2x claim ceiling was
said to guarantee that anything writable is claimable. That is wrong.
Postgres reparses JSON numbers as `numeric` and prints them positionally,
so the expansion has no ceiling at all. Measured against Postgres 16:
{"a":1,"b":2} 13 bytes -> 16 (whitespace only, ~1.2x)
{"a":1e-100} 12 bytes -> 109 (~9x)
{"a":1e-3000} 13 bytes -> 3009 (~231x, exponent free to grow)
So no multiple of the write cap is a safe claim ceiling, and the Go-side
cap does not bound the stored row at all. Reachable rather than
theoretical: item payloads carry `fields` as a JSON *string*, whose
contents are escaped text and immune, but a bulk delta is a
map[string]any and a numeric field value from a request body arrives as
a float64 that re-marshals in exponent form. The failure it produced was
a row accepted by the write and then excluded from every claim for the
rest of its retention window — written, undeliverable, visible only as
an oversized-row log line.
Fixed where the number is actually true: the INSERT now RETURNs
octet_length of the stored payload and refuses against the claim
ceiling, rolling the caller's transaction back exactly as the
pre-marshal check does. "A row this binary wrote is a row this binary
can read back" is now established by construction instead of inferred
from an expansion argument that did not hold.
MaxOutboxClaimableBytes keeps its 2x value but loses its false
justification: its job is only to leave ordinary payloads room above the
write cap so the two rules do not fight over rounding.
Test gaps from the same round, all three closed:
- The claim-ceiling invariant was pinned only by a Postgres round-trip,
so a ceiling collapsed back to the write cap passed every default
(SQLite) run. The constants test now asserts the relation directly and
fails on either dialect.
- TestBatchSiblingQueryIsBoundedInSQL drives claimableBatchSiblings
directly: bounding the batch in the caller instead of in SQL passed
every assertion on the claim's return value while keeping exactly the
unbounded allocation the row cap was added to remove.
- The scrub's byte budget changes peak memory and nothing else, so
removing it left every outcome assertion green. A TEST-ONLY
afterOutboxScrubBatch seam makes batch count observable, which is the
one visible consequence of the budget working.
Codex also confirmed the previous round's rejected finding: keyset
paging covers every row present at the initial scan and additionally
catches later commits sorting above the cursor, so it is a superset of
the unbatched behaviour rather than a regression.
Mutation matrix, run on BOTH dialects: dropping the stored-size refusal
kills TestEverythingWrittenIsClaimable on Postgres only (correctly — it
is a Postgres defect); dropping the sibling SQL LIMIT and collapsing the
claim ceiling kill their tests on both; dropping the scrub byte break
kills TestScrubSpendsItsByteBudgetNotJustItsRowLimit.
Gates: gofmt clean, golangci-lint 0 issues, full SQLite suite exit 0,
full Postgres suite exit 0.
Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm
|
||
|
xarmian
|
37f26f5430 |
fix(store): close the outbox bound's remaining unbounded paths (BUG-2827)
Codex review of the previous commit. Four real defects, one finding answered rather than acted on, and one test that did not discriminate. ROW CAP ON THE CLAIM. A budget in bytes alone does not bound the per-row cost, and the batch scan is the path that shows it: siblings are collected past the row limit by design, so a batch of a million tiny rows sits comfortably inside 64 MiB while its ids, maps, OutboxEvent structs and folded delivery do not. maxOutboxClaimRows (5,000) binds only that shape — at the measured ~2.4 KB payload mean it is ~12 MiB, well inside the byte budget, which stays operative for real traffic. REFUSE BEFORE MARSHALLING. writeOutboxTx could only see the payload after json.Marshal had built it, so a member set large enough to be refused was large enough that rendering it to be refused was its own memory event. emitBulkItemEventTx now charges the members' own bytes first. It is an early-out, not a second rule: JSON only adds, so the projection is a lower bound and can never refuse something the real check would accept, and the authoritative numbers still come from writeOutboxTx. Pinned by asserting the reported size equals the projection exactly — any weaker assertion passes against the code this guards, because the late check refuses with the same error type. ONE COPY, NOT TWO. outboxEventsClaimedBy scanned each payload into a string and then converted it to []byte. On the largest single row a pass may take, that was the difference between one copy and two. A COMMENT THAT WAS FALSE. ScrubOutboxUserRefsTx's note called its subject_id arm indexed. Migrations 081/082/083 index (occurred_at,id), dispatched_at, (workspace_id,occurred_at), batch_id and (claimed_at,occurred_at) — nothing on subject_id. Corrected to say the scan it actually is. CONSIDERED AND REJECTED: that keyset paging over random uuids can miss rows. It cannot miss a row that existed when the scan began — ids are fixed, the walk is ascending over every matching row above the cursor, and a row stops matching only once scrubbed. It changes concurrent commits in the SAFE direction: the single query missed everything committed after it, while this catches those sorting above the cursor, so coverage is a superset of the unbatched version. Snapshotting all ids first would make the window describable without reference to uuid order, and was tried and reverted: it holds every matching id at once, which is an unbounded allocation of the same shape this change removes. The reasoning is now in the comment so the next reader does not re-derive it. Also accepted as residuals, both documented at their constants: a row taken alone because it exceeds a whole pass's budget still costs that pass its size, and the scrub reads one oversized legacy payload whole. Both are the price of delivering and erasing data that exists; refusing either is data loss rather than a bound. Gates on this tree: gofmt clean, golangci-lint 0 issues, full SQLite suite exit 0, full Postgres suite exit 0. Mutation matrix re-run for the new guards: dropping the row cap kills TestOutboxClaimStopsAtTheRowCap, dropping the early-out kills TestBulkEventIsRefusedBeforeItIsMarshalled, dropping the scrub cursor advance still kills TestScrubOutboxUserRefsTerminatesOnLikeFalsePositives. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
5b12d5eeeb |
fix(store,server): bound the outbox's unbounded reads and writes (BUG-2827)
item.bulk_updated marshals every cascaded member body into ONE
event_outbox row, and nothing bounded either the row or the drain's
reading of it. The v1 doc called the size deliberately unbounded and
named a follow-up condition; measurement met it.
MEASURED FIRST, against an 8,434-item instance:
widest wiki-title cascade 23 members / 175.6 KiB of bodies
widest wiki-ref cascade 50 members / 365.5 KiB
worst option rename 4,429 members / 7.34 MiB
(tasks.status="done")
marshal ratio ~1.46x (item.updated payload mean 3,518 B
against an item body mean of 2,416 B)
So renaming one status option on that instance emits ~11 MiB in a single
row today, from a user clicking rename in the collection editor. That
kills the obvious design: a write cap tight enough to bound the drain's
memory refuses routine work, and one loose enough for routine work
bounds nothing. The two therefore became two numbers.
WRITE CAP (128 MiB) in writeOutboxTx, the single INSERT INTO
event_outbox in the tree, so all seven emit paths inherit it rather than
an enumerated site list. Refusing FAILS the mutation: the hop bound in
the same function drops the event and keeps the mutation, correctly,
because there the cascade is what is illegitimate, while here the
mutation and the event are the same fact. 128 rather than 64 MiB so it
clears MaxItemRenameCascadeBytes — a cascade squeaking under that 64 MiB
bound marshals to ~96 MiB, and a 64 MiB cap here would let the vaguer
refusal preempt rename_cascade_too_large on the very renames that bound
describes. Surfaces as 413 event_payload_too_large, following that
precedent rather than inventing a second spelling.
CLAIM BUDGET (64 MiB per pass), spent by the primary candidates AND by
batch siblings. Spending it on siblings knowingly relaxes "batches are
claimed whole": read literally that rule makes the budget bypassable by
construction, since one large batch is an unbounded read no row limit
touches, and groupOutboxDeliveries already defines the split. The first
candidate is always taken whatever it costs, so the bound meant to keep
the drain alive cannot starve a row instead.
CLAIM CEILING (2x the write cap), and it is NOT the write cap — a
distinction the Postgres leg had to teach. The cap measures the Go bytes
json.Marshal produced; the claim measures what the driver hands to Scan,
which on Postgres is the JSONB ::text rendering, one space inserted
after every colon and comma. A 40,000-byte payload reads back as 40,001.
Thresholded at the same number, a payload written at exactly the cap was
admitted by the guard and then permanently excluded by the claim:
delivered to nobody, reported as nothing, reaped seven days later.
TestARowWrittenAtTheCapIsStillClaimable fails on Postgres and passes on
SQLite against the 1x version.
Rows above the ceiling are excluded IN THE PREDICATE, not filtered in
Go — filtering in Go leaves them occupying candidate slots and starves
everything behind them, which is the same jam wearing different clothes.
They are logged every tick and left pending for the existing 7-day
undispatched retention, not stamped dispatched_at, which would record
that an event went out when it did not.
SCRUB. ScrubOutboxUserRefsTx collected every LIKE-matching payload at
once — the same unbounded read through a different door. Now batched by
a keyset cursor on id. READ FULLY THEN WRITE is preserved PER BATCH, and
the cursor keeps the per-row UPDATEs in ascending id order across
batches, so batching does not quietly reintroduce the BUG-2409 deadlock
it was written to avoid.
Prose sweep: MaxItemRenameCascadeBytes' comment asserted this vector had
no bound, and emitBulkItemEventTx's asserted the payload was unbounded
by decision. Both now say what is true, the second keeping its original
reasoning because it still explains what the bound does NOT do.
Mutation matrix, each mutant compiled and run unfiltered: removing the
row-cap check, > to >=, removing the budget break, removing
always-take-one, removing the SQL size exclusion, removing the sibling
budget, removing the scrub cursor advance, and 2x to 1x — all die, the
last on Postgres only. One survivor recorded as unfaithful rather than
as a gap: setting the scrub cursor to the first row read still advances
monotonically over an ascending id > cursor query, so termination holds
and only the pass count degrades.
Not addressed, named rather than left to be found: a workspace that
outgrows the write cap cannot shrink its own cascade, so the refusal
leaves no recourse. The answer is chunking one bulk event across rows
sharing a batch_id, which the drain's fold already supports.
Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm
|
||
|
xarmian
|
ebe40de932 |
feat(store): add a dialect accessor for a column's scanned byte length
Dialect.OctetLength renders the byte length of a text-ish column as the driver will hand it to Scan. The "as Scan will see it" part is the whole reason this is a dialect method rather than a literal in one query. event_outbox.payload is TEXT on SQLite and JSONB on Postgres; octet_length has no jsonb overload, so Postgres needs a ::text cast, and that cast renders the PARSED value — whitespace normalized, keys reordered, duplicates collapsed — which is not the byte count that was written. Every caller of this is deciding whether it can afford to Scan a value, so the scanned count is the one they need and the stored one would quietly mislead. No behaviour change on its own; BUG-2827 is the first consumer. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
978af702f7 |
fix(store): sweep by pattern rather than a name list, and quote the dropped identifier
My round-6 "class sweep" was a hand-written list of column NAMES, which is an enumeration one level up — and round 7 found exactly the two names nobody thought to write down: event_outbox.claimed_by and oauth_connection_workspaces.added_by. Five rounds have now each named unprotected columns. So the sweep is a PATTERN (*_by, plus actor/author/source/owner) and it is a TEST, which is the difference between having done it once and it staying done. A new *_by column fails the day it is added, with a message saying which side of the decision it needs. It carries its own instrument check: if the pattern matches nothing at all it fails as broken rather than passing as clean. 262 triggers over 131 columns. Second finding, and it would have been a startup failure rather than a silent one: trigger names read from sqlite_master were interpolated into DROP TRIGGER unquoted. A legal identifier matching the GLOB — one containing a hyphen — produces a syntax error, which fails the restoration, which migrate propagates, which fails startup. The one place a stray trigger is most likely to have an unusual name is the one place this code has to survive it. Quoted, with the embedded-quote case escaped. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
5d2ae30ca2 |
fix(store): sweep the attribution-column class, and treat a stray trigger as unhealthy
Codex named eight unprotected attribution columns on documents, versions and comments. Rounds 2, 3 and 6 have now each named one or two of these, which is the signal that the reviewer is sampling a POPULATION rather than finding instances — CONVE-18's whole point. So I enumerated the class: every created_by / last_modified_by / source / actor / author / *_by column in the schema. SIXTEEN were unprotected, against the eight named. All sixteen are in, including the ones that are server-set today (granted_by, invited_by, uploaded_by), on Ruling 2's posture for the second ring: an instr trigger is near-free, and litigating each one is how the last three rounds went. 258 triggers over 129 columns. Second finding, also real: the health check asked only whether every EXPECTED trigger was present, so a database carrying an EXTRA pad_nul_ trigger read as healthy. That is not cosmetic — a stray left by a partial restore or a manual edit can abort legitimate writes, and the check meant to notice would report the database fine forever. Extras now count as unhealthy, and the existing drop-then-recreate removes them. The test installs a stray that refuses everything, proves a legitimate write is broken by it, and then measures the repair. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
214b888811 |
fix(store): normalize IF NOT EXISTS, and make the restoration report its work
[P1] The definition comparison NEVER matched, so every startup dropped and recreated all 226 triggers under an immediate write lock. SQLite stores "CREATE TRIGGER" while the generator renders "CREATE TRIGGER IF NOT EXISTS", and the comparison was verbatim. Normalized now. [P1] A schema_migrations query ERROR was folded in with "not applied" behind a nolint:nilerr, so a migrated database with a read failure would start successfully with the invariant unenforced — enforcement silently absent in exactly the case the check could not run. Only the table being ABSENT means "earlier than that migration"; a query failure propagates. THE PART WORTH KEEPING is how long the first one hid. Its no-op test asserted the trigger COUNT was unchanged — true of a full drop-and-recreate. Told that, I asserted sqlite_master ROWIDs instead, and the mutation STILL survived: SQLite reuses rowids when the drops and creates happen in one transaction in the same order. Two observables, both about whether the state LOOKS the same afterwards, when the question was whether anything was rebuilt. So the operation reports what it did. ensureNULTriggersReporting returns whether it restored, and the test asserts on that — an observable that cannot be satisfied by doing the work and leaving things looking tidy. The query-error case is forced by renaming schema_migrations' version column, with a control on the healthy database first so the failure is the corruption and not the fixture. Both fixes are mutation-verified. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
5872647389 |
fix(store): restore triggers after every migration, gated on its own
Codex round 4's one correctness finding: the restoration ran once, AFTER the whole migration chain and the FTS validation. A table-rebuild migration drops that table's triggers, so the invariant was unenforced for it across every remaining migration — and a concurrent raw writer (an old binary on the same file, the population Layer B exists for) could commit a violating row in that gap. It runs after EVERY migration now, narrowing the window to the moment between one migration's commit and the next statement. That fix immediately broke a fresh install, which the suite caught at once: on the early migrations the tables do not exist yet, so the trigger SQL failed with "no such table". The restoration is gated on its own migration having been applied — before that there is nothing to restore, and absence is the answer rather than an error. THE RESIDUAL IS STATED, not papered over. The DROP lives in a rebuild migration's own transaction and the recreate in another, so the window cannot be closed from here — only narrowed. Making an EXISTING violating row go away is S3's repair sweep, which is needed regardless: rows written before S2 shipped are the same problem arriving by a different route. The code says so at the point where a reader would otherwise assume the restoration is total. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
12ea3f4d9a |
fix(store): compare trigger DEFINITIONS, and inspect inside the transaction
[High] The restoration had a window BEFORE the one round 2 closed. Round 2 put the CREATE statements in a transaction; round 3 found that the "are any missing?" check still ran outside it, so a raw writer could commit an invalid row between the check and the lock. The transaction is opened first now — the DSN carries _txlock=immediate, so Begin takes the write lock — and the inspection runs through it. [High] The check compared trigger NAMES, which can never repair a stale one: CREATE TRIGGER IF NOT EXISTS sees a same-name no-op body and does nothing, forever. It compares DEFINITIONS now, against the same rendered text the migration is generated from, and drops before recreating. A mutation reverting it survived until the stale-trigger test existed — the test installs a no-op trigger with the right name, proves the COUNT is correct and protection is gone, then measures the repair. [High] Two more caller-controlled columns. views.view_type is `viewType := input.ViewType` with "list" only as a fallback, on create and update. item_links.link_type is normalized on the ordinary create path and written VERBATIM by ImportWorkspace — the second-write-path shape for the third time in this unit, and the second time it was the import that had it. [Medium] The classifier test grepped nulguard.go for the classifier's name, so a dead or commented call would satisfy it. There is a fake driver now that returns a marker-bearing error from each of the four entry points, and the test asserts the caller receives the TYPED error — routing exercised rather than read. Two mutations, one per entry-point shape, confirm it discriminates. [Low] Two refusal assertions accepted ANY error. items.fields also carries migration 056's JSON constraint, so "some error" was satisfiable by a value that never reached a trigger; both assert the marker now. The disjointness assertion added in round 2 earned itself immediately: it caught both newly-protected columns still sitting in the baseline. 226 triggers over 113 columns. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
f789eaac21 |
fix(store): protect items.slug against the import path, and make restoration atomic
[P1] items.slug was unprotected, and the exclusion was true of one write path and false of another — the lesson this cluster keeps re-teaching. The API path derives the slug through slugify, whose [a-z0-9-] output cannot carry a NUL. ImportWorkspace has its OWN INSERT and writes the BUNDLE's slug verbatim: importCoercedSlug returns it unchanged whenever it is inside the length bound, so a crafted bundle puts any bytes it likes there. That is code I wrote in S1. [P1] Six more caller-controlled columns were unprotected: items.created_by, items.last_modified_by, items.source, item_versions.created_by, item_versions.source, item_links.created_by — the handlers let a request body's value win over the server's own. Plus custom_templates.icon. [P1] Trigger restoration was not atomic: 206 CREATE statements outside a transaction leave a window where some tables are protected and others are not, and a concurrent writer can commit an invalid row inside it. It runs in one transaction now. [P1] The restoration check compared a COUNT and matched with LIKE. A database with the right number of triggers but one missing and one extra read as healthy, and IF NOT EXISTS would then never repair the missing one. It compares the SET now, and matches with GLOB — LIKE's `_` is a single-character wildcard, so the old pattern also matched names this code never generates. [P2] The census baseline still listed columns that had become protected, and the test never asserted the two sets are disjoint — so losing a column's protection would have put it back in `unaccounted`, found it still listed, and passed. Disjointness is asserted and the baseline regenerated. [P2] And the finding I want on the record, because my first fix for it was worse than the gap. Codex was right that testing classifyTriggerRefusal with a synthetic error would pass even if the wrapper stopped calling it. I added an "integration" leg that wrote through a guarded connection and asserted the typed error came back. It PASSED — and the refusal came from LAYER A, whose error is the same TYPE, so errors.As succeeded while the trigger was never involved. There is no value that Layer A accepts and Layer B refuses: both implement the same predicate, and the four-way differential test asserts they agree on the whole corpus. The unreachability IS the property, so a reachable case would be testing a disagreement we work to prevent. The leg is replaced by a structural one asserting every wrapper error path routes through the classifier, and the comment says why there is no end-to-end alternative rather than implying the gap was closed. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
46c55e32f3 |
fix(store): re-assert triggers after a table rebuild, and correct two inherited classifications
[P1] A future table rebuild would have silently removed the protection. SQLite
drops a table's triggers with the table, this codebase rebuilds tables to change
constraints (migrations 025, 055, 056, 057, 068, 072), and migration 084 would
never run again because it is recorded as applied. The FTS equivalent warns and
moves on, which is the right cost for a derived index and the wrong one for a
data invariant: a missing FTS trigger breaks search visibly, a missing NUL
trigger is silently no protection against the exact writer Layer B exists for.
The triggers are now RE-ASSERTED after every migration pass, and the test
proves the loss is real before measuring the restoration.
[P1] Six caller-influenced columns were unprotected. workspaces.slug,
collections.slug, views.slug and agent_roles.slug are all `slug := input.Slug`
with slugify only as a FALLBACK — the census's exclusion note was about
items.slug, which is genuinely derived (ItemCreate has no Slug field), and I
read it as covering slugs generally. Plus comment_reactions.emoji and
oauth_clients.logo_url.
[P1/P2] Two classifications were wrong in the census and I inherited both.
agent_roles.tools is FREE TEXT — migration 019 says so in as many words
("free-text notes about preferred tools/models") — and classing it JSON would
refuse a user's note that happens to be valid JSON carrying an escape. And of
the six oauth request columns, only session_data is JSON: RequestForm is
`.Encode()`, and scopes/granted_scopes/audience/granted_audience are
`strings.Join(..., " ")`. The census extended oauth_clients' jsonStringList
classing across tables that do not use it. 20 columns reclassified.
[P2] The migration's header told readers to run a generator I had deleted. It
is a real test now, skipped unless GEN_NUL_TRIGGERS is set — an artifact that
instructs you to run something nonexistent is worse than one with no
instructions.
[P2] The pin compared trigger NAMES and counts, which would pass a wrong
BEFORE UPDATE OF clause, a wrong predicate, or a changed marker — the parts
that do the work. It compares the whole rendered text byte for byte now, and
reports the first differing line.
[P2] The census matched TEXT/CHAR/CLOB only, so a column declared JSON or with
no declared type — both BLOB affinity, both holding text fine — would have
slipped past. Widened to match what CANNOT hold text. It immediately surfaced
item_yjs_updates.update_data, which is correct: that is the binary column Layer
A exempts, and it is now recorded as an explicit exclusion rather than hidden
by a filter. The table filter also matched "_fts" anywhere in a name, which
would skip a real table called something like user_fts_settings.
[P2] The differential leg claimed persistence and only checked that no error
came back. It reads the value back and compares it now.
[P2] Trigger refusals set Ordinal 0 against a documented 1-based field,
rendering as "parameter 0". A database refusal knows the COLUMN, not the
parameter position, so it sets no ordinal and Error() says so.
Full Go suite green on SQLite and Postgres 17; lint 0 issues.
Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm
|
||
|
xarmian
|
054b192e5d |
feat(store): make the NUL invariant a property of the DATABASE (DOC-2823 S2)
Closes BUG-2813's old-binary half. S1 put the rule in the binary, which cannot help in the window this bug is about: an older binary serving the same SQLite file has no guard, so a rollback, a staged rollout or a second instance writes rows the invariant forbids. A trigger is enforced by the FILE. 194 BEFORE INSERT/UPDATE triggers over 97 columns — TASK-2825's 86-column census plus the second ring Ruling 2 admitted wholesale. ONE LIST, THREE CONSUMERS, which is the shape TASK-2825 asked for. The migration is GENERATED from internal/store/nulcolumns.go, a guard test compares that list against the LIVE migrated schema, and a second test pins the generated SQL against the list so neither can drift from the other by hand. The census guard is a BASELINE rather than a demand that all 405 text columns be classified. The census classified the 86 that can carry caller text and left ids, timestamps and hashes unenumerated; requiring an entry for each would be 300 lines nobody reads. The 301 known-outside columns are recorded, and ANY change to that set — added, renamed, removed — fails and asks for a decision. The predicate is TASK-2824's, measured on the driver Pad actually embeds: instr for the raw byte (length() C-truncates and cannot be used), and json_tree for the decoded escape, in values AND keys, with a json_valid guard because json_tree raises on a non-document. The doubled-backslash case stays literal, which is the false positive this predicate family exists to avoid. Trigger aborts are classified into the SAME typed error Layer A produces, so a caller cannot tell which layer refused and the handler's existing 400 mapping covers both. That discharges Ruling 2's condition for admitting the second ring: a header-derived user agent or IP hitting a trigger must not surface as a 500 or a broken login. THE FOURTH LEG IS LIT. S1 built the differential harness with three legs and left this one dark; all 17 corpus cases now agree across the HTTP gate, Layer A at the driver, Layer B via a real UNGUARDED SQLite write, and native Postgres. Four independent enforcers, one corpus — the property DOC-2823 named as the actual deliverable. Two consumers needed changes, both legitimate. The FTS trigger census excludes the new triggers by prefix, so adding a protected column does not require editing that test. And a timeline test that INJECTS a NUL to build its fixture now brackets the injection by dropping and restoring the two triggers — it needs a row that violates the invariant, because what it tests is the handling of legacy data, and Layer B exists to stop such a row being written. SQLite only. Postgres refuses a NUL in text natively and an escape decoding to one in jsonb, so it already owns the rule there. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
f83de5b904 |
fix(store,textguard): close codex round 4 — stop chasing shapes, and correct a claim
[P1] A valuer returning a valuer passed the guard: resolution happened once, so an outer valuer could hand the guard a clean inner value while pgx went on to evaluate it and send a NUL. Resolution is to a FIXED POINT now, bounded so a cyclic valuer is refused rather than looped on. [P1] The text detector still missed shapes pgx binds unconverted, and the next round would have named five more. Two rounds were already spent widening it one shape at a time — exact string, then *string and named types, then json.RawMessage — which is a losing game against a permissive driver. So it is an ALLOW-LIST now: the guard enumerates what this store actually binds (strings, binary blobs, numbers, booleans, times, NULL) and REFUSES anything else rather than passing an unclassifiable value through unchecked. That is the default every previous version had backwards. The full suite on both dialects passes, which is the evidence that the vocabulary is complete. [P2] The typed-nil rule was broader than database/sql's: it nil'd every nil pointer valuer, where the library only does so when the pointer's ELEMENT type implements Valuer and otherwise calls the nil-safe method. Copied exactly now. [P1 residual] And a rationale of mine was FALSE. The duplicate-key gap entry said Postgres "keeps the last too, so it accepts this as well - the two agree today, which is why it is a recorded gap rather than a dialect split." Measured on Postgres 17, a duplicate-key document whose SHADOWED value carries the escape is REJECTED with "unsupported Unicode escape sequence", while the all-clean control deduplicates fine: the parser processes the scalar BEFORE duplicate elimination. So it IS a dialect split — Postgres refuses, this guard accepts, SQLite stores — and I had asserted the opposite without checking. The entry stays in KnownGaps because closing it means replacing the shared predicate's decode with a token walk, which is BUG-2812/S4 and moves both layers together; fixing Layer A alone is the divergence DOC-2823 forbids. The native-Postgres leg ran Corpus only, so the recorded gaps were never measured against the database at all — which is how that wrong rationale survived. It covers KnownGaps now and reports the split explicitly. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
cc8bf888d2 |
fix(store,server): close codex round 3 — inspect what is actually bound
Eight findings, four P1. The three parameter ones share a cause: the guard inspected one value and forwarded another. [P1] Value() was called for the CHECK and the original driver.Valuer was forwarded, so pgx called it again — a stateful valuer could show the guard clean text and the database a NUL. Resolution now happens once and the resolved value is written back into the argument. [P1] A typed-nil valuer, (*sql.NullString)(nil), was called directly and panicked. database/sql special-cases it as SQL NULL; so does the guard now. [P1] Only an exact `string` was recognised. pgx implements NamedValueChecker and binds *string, named string types and json.RawMessage unconverted, so each carried text the guard never saw. Resolution is by reflected KIND now, not by a list of types. [P1] Registration opened a probe connection for BOTH drivers, so creating a SQLite store attempted a live PostgreSQL connection against whatever host the environment happened to name — network access as a side effect of opening a local file — and swallowed the error, skipping the guarantees in exactly the case the check could not run. The assertions moved to connect and prepare time, where the object being asserted about is the one in hand. [P2] The statement-interface assertions were claimed in a COMMENT and existed nowhere. guardStmt forwards neither NamedValueChecker nor ColumnConverter, so a driver gaining one would silently lose its own argument conversion. Asserted now, and the parity test covers the statement level too. [P2] The cross-workspace copy gave a guard refusal its ambiguous "may or may not have landed" message. That refusal fires at parameter binding, before any statement executes — nothing landed, and telling the caller to reconcile invites exactly the manual work DR-13's wording exists to prevent. [P2] The Valuer integration test passed for the wrong reason by default: SQLite's driver lacks NamedValueChecker, so database/sql unwrapped the NullString before the guard ran. The three properties are unit-tested now, independent of which driver is present. And the NULL control asserted only that the write succeeded — it reads the row back and asserts SQL NULL, which an incorrect conversion to "" would otherwise have passed. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
74c786d590 |
fix(store,server): close codex round 2 — the wrapper must mirror its base
Seven findings; two were P1 and one of them was a real hole on Postgres only. [P1] driver.Valuer bypassed the guard. checkParams type-asserted `string`, and pgx implements NamedValueChecker — so it ACCEPTS a sql.NullString unchanged rather than letting database/sql's converter unwrap it, and the guard never saw the text. Measured before the fix: a NUL-bearing sql.NullString on Postgres passed Layer A entirely and was refused by the server as SQLSTATE 22021, i.e. a 500, while the identical value on SQLite got the typed 400 — the dialect split reappearing in the response shape. wiki_links.go binds sql.NullString today. [P1] driver.DriverContext was dropped, so sql.Open used a legacy connector that ignores the context and a cancelled request could leave a pgx dial running to its 60-second timeout. The rest share one cause, and it is the thing to remember: database/sql BRANCHES on whether an optional interface is present, so a wrapper advertising one the base lacks CHANGES behaviour rather than adding a no-op. Measured, the two drivers differ — pgx has no Validator, sqlite has no conn NamedValueChecker and no DriverContext — and the single wrapper type claimed all of them. So the wrapper now MIRRORS its base: four conn variants over the two interfaces that vary, a separate driver type for DriverContext, and the non-varying ones asserted at registration so a driver bump fails loudly instead of degrading. I made the same mistake inside the fix — implementing OpenConnector unconditionally, which broke every SQLite open — and then a third time, where guardConnector.Driver() returned the inner wrapper and a pgx pool reported no DriverContext. The third was caught by a new parity test on its first run, not by review: it asserts wrapped and base advertise EXACTLY the same interfaces. Also: both gap guardrails SKIPPED when their slice was empty, so deleting an entry made the suite green — the opposite of their purpose. They assert counts now. And the oracle test compared the two walkers without pinning any answer, which is how round 27 left both wrong about scalars; known answers are pinned against Postgres, and doing that caught me re-pinning one from a stale comment. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
ef792bdf35 |
fix(store,server,textguard): close codex round 1 on S1 — seven findings
All seven verified before acting; two were production regressions I had
introduced and one was a defect older than this branch.
[High] The wrapper HID the optional connection interfaces. Measured: the raw
modernc conn implements Pinger, SessionResetter and Validator; the wrapped one
implemented none, so Ping succeeded without pinging, pooled connections stopped
being reset, and dead connections stayed in the pool. Each is now forwarded,
falling back to exactly what database/sql does for a conn lacking it.
NamedValueChecker is forwarded too, or the wrapper silently narrowed the
argument types pgx accepts.
[High] Statement-level driver.ErrSkip was wrong. At the CONN level ErrSkip is
the documented fallback signal; at the STATEMENT level database/sql propagates
it, so a base statement without the context interface would have failed rather
than degraded. Both now fall back to the positional form, refusing a NAMED
argument rather than binding it by position.
[High] Scalar JSON documents bypassed the check — and the database settles it:
SELECT ('"a<escape>b"')::jsonb;
ERROR: unsupported Unicode escape sequence
A bare JSON string is a complete jsonb document. The object/array-only shape
test is right for the HTTP gate's "is this a nested document" question and
wrong for the store's "will a jsonb parser read this". Both now use a widened
form, so the gate refuses it at the door instead of leaving it to the store.
That widening broke the gate's independent oracle, which had been NARROWED in
BUG-2803 round 27 to make the two walkers agree. The disagreement was real; the
direction of the fix was not — nobody measured which one matched Postgres, and
it was the oracle. Restored, with the measurement in the comment. Two
implementations made to agree are not thereby correct.
[High] The 400 mapping was not the single funnel I claimed. createItemChecked,
bulk ops and the cross-workspace copy carry their own error envelopes by
design. One shared CLASSIFIER now serves all of them; the envelopes stay
different, the classification and wording do not. Copy keeps its
retry-discouraging message deliberately (PLAN-2357 DR-13).
[Medium] The over-refusal justification was FALSE as written. It claimed a
JSON-shaped text value carrying a live escape "is a value Postgres would refuse
the moment anything parsed it". Nothing parses a text column; Postgres stores
it fine. The trade is now stated honestly, and pinned: textguard.
StoreOverRefusals records the case, and a test fails when it is paid down.
[Medium] Duplicate JSON keys are a real under-refusal, inherited from the
shared predicate's map model. DOC-2823 requires Layer A NOT to fix it alone, so
it is recorded in textguard.KnownGaps with a test that fails when it CLOSES —
the signal that BUG-2812's token-walk landed.
[Medium] Test weaknesses, and the fixes caught a real one. The census now sees
ALTER TABLE ... ADD COLUMN. The store corpus leg now requires accepted cases to
SUCCEED, which immediately exposed a case passing for the wrong reason: a
non-JSON value written to the fields column failed on SQLite's own JSON parser,
never reaching the guard.
Full Go suite green on SQLite and Postgres 17; lint 0 issues.
Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm
|
||
|
xarmian
|
3baf984b41 |
feat(server,store): the differential test's three live legs + the 400 mapping
Completes S1's deliverable except Layer B, which stays dark until S2. THE THREE LEGS, all driving the SAME corpus: - HTTP gate (internal/server): each case enters as a request body, classed the way a request is — by KEY NAME, "fields" for JSON, "content" for text. - Layer A at the driver (internal/store): each case is a real write, classed by column. - Native Postgres (internal/store): each case is a real INSERT through the RAW pgx driver, no guard in the path, so what is measured is Postgres's own verdict rather than ours reflected back. All 15 cases agree across all three. That is the calibration result the design needed: the guard neither over-refuses (breaking writes Postgres accepts) nor under-refuses (leaving the dialect split BUG-2831 was about). The Postgres leg additionally asserts the SQLSTATE, because "some error occurred" would have been satisfied by a typo in the test's own SQL. THE 400 MAPPING is in writeInternalError — the single funnel every 500 already passes through, so one insertion covers every handler present and future. The item-title unit had to find the same error block three times in one function before a structural test caught the third; this avoids the enumeration entirely. 400 rather than 500 because the request is understood and will be refused identically on retry, and 400 specifically because it is what the gate answers for the SAME value refused at the door — two statuses for one rule would be the layers disagreeing in the response instead of the predicate. The residual is stated rather than hidden: a value can reach the store from something the SERVER composed (BUG-2814's re-emit population), and for those a 400 misattributes the fault. It is still better than 500, and if that case ever needs its own status it needs its own error type first. Mutation matrix extended: 3 more mutants, all killed — mapping removed (500 returns), mapping swallowing every error (the control leg), and the gate ceasing to delegate to the shared core, which is the layers-diverge scenario the corpus exists to catch. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
976fff4060 |
test(store): measure the write guard's cost, and report it at the width it holds
S1 named this measurement as owed: the guard touches EVERY statement, reads included, so a byte scan being "obviously cheap" is an assumption, not a result. Per-call, against a typical read's parameters: 33.5 ns, ZERO allocations. The []byte exemption costs 20 ns and never touches the bytes. A realistic JSON fields blob with no escape costs 32.5 ns and zero allocations, which is the important one — the expensive arm is gated behind the escape pre-filter, so it runs only for a value that actually contains the escape, and never on the read flood. The one expensive case (2167 ns, 16 allocs) is a value carrying an escape, and 256 KiB of prose scans in 7.4 us with no allocation. The end-to-end pair — the same GetItem guarded and unguarded — is reported but deliberately NOT quoted as a percentage. Run-to-run spread inside each group is ~25 us against an expected difference of ~34 ns; three orders of magnitude apart. Six samples cannot resolve that, and "3.6% slower" would be noise read as signal. What it DOES resolve is the allocation profile: 7034 B and 158 allocs on both sides, identical, which is a real invariant. One benchmark silently measured nothing at first: its "harmless escape" was textguard.EscNUL, which decodes to a NUL, so every iteration was refused and the case produced no line at all. It builds a A now, and the comment says why — the same escape-literal trap this unit has hit five times. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
52c480c6d2 |
feat(store): enforce the NUL invariant at the driver (DOC-2823 S1 Layer A)
Closes BUG-2814 and the current-binary half of BUG-2813: the fixed binary can no longer WRITE a decoded NUL from any path, request or re-emit. The guard is a database/sql driver wrapper, not a seam in this package, and the two measurements that decided it are in nulguard.go's header. The design named the Queryer seam; Queryer is Query + QueryRow, its own doc says "the read-only subset", and no store write passes through it. Writes reach the driver by four receivers — db.Exec (139 sites), tx.Exec (99), stmt.Exec (2 prepared), and a passed executor — and a wrapper at the *sql.DB level cannot see the prepared ones at all. The deeper reason is this cluster's own lesson: a seam every write site must be EDITED to route through is an enumeration wearing a seam's costume, and nothing stops the next site taking the raw handle. The wrapper never reads SQL text, only bound parameters, so Sprintf-built statements are covered without being understood and a new call site is covered before it is written. It covers Query as well as Exec, which is not symmetry: three writes ride the Query path today (UPDATE ... RETURNING in password_resets and email_verification, INSERT ... RETURNING in yjs_updates). A single-line regex for that shape found ZERO; a structural scan found all three, each a multi-line raw-string literal. CLASSING: string parameters are checked, []byte is exempt. The invariant is about text and JSON columns, and in this store []byte binds BINARY — item_yjs_updates.update_data, the only BLOB/BYTEA column in either schema. The first version checked []byte and refused every Yjs op-log append; the existing collab suite caught it immediately. The exemption is pinned by a census test that fails when a new binary column appears, because the supporting sweep was source-level and this unit is a catalogue of source-level sweeps missing things. Mutation matrix: 13 mutants, 12 killed by named tests. Four survived first and each taught something rather than needing a weaker claim — the Query path, both prepared routes, and a Prepare branch that turned out to be DEAD for both drivers (database/sql routes to PrepareContext when the conn implements it), so that duplicate was removed rather than tested. Full Go suite green on SQLite and Postgres 17. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
0d0f1c9125 |
fix(items): require and bound item titles at every write door (BUG-2833, BUG-2831)
`PATCH {"title": ""}` was accepted and applied while `POST` refused the same input
with 400 "Title is required": the guard was an inline literal inside
handleCreateItem, so the sibling handler on the same field never had it. Item
titles were also unbounded, and the slug derives from the title with no
truncation, so the same input was accepted on SQLite and refused by Postgres at
the UNIQUE(workspace_id, slug) btree with an unmapped SQLSTATE 54000 — a latent
`pad db migrate` failure as well as a create-path one.
One models.NormalizeItemTitle / models.ValidateItemTitle pair now backs every
door, enforced authoritatively in store.CreateItem and store.UpdateItem so a
future door inherits the rule rather than having to repeat it. The handlers keep
a pre-lock copy that REFUSES ONLY: it may answer 400 early and must not alter
the input, because its view of the row predates the write lock.
- trim: whitespace-only titles are refused, widening the create door. Artifact
import already trimmed while create tested == "" exactly, and its comment
claimed to mirror the gate it was stricter than.
- bound: 255 runes, matching MaxDocumentTitleRunes but justified for items —
slugify emits only [a-z0-9-] at one byte per rune and truncates nothing, so
255 runes bounds the slug well under the btree index-tuple cap. That cap is
2704 bytes in practice, not the 8191 the filing quoted; both figures and the
readings behind them are in the constant's comment.
- non-retroactive: a title identical to the stored one is not a rename, is not
validated, and is dropped rather than re-applied — so rows predating the bound
stay editable and a no-op echo cannot move an item's slug.
- import coerces rather than refuses (empty -> "Untitled", over-long ->
truncated, both logged, colliding truncations resolved), matching
coerceJSONForImport's recorded disposition three lines away. Refusing would
break restoring archives of data this product already accepted.
- cross-workspace copy propagates a legacy source title, by ruling. It takes no
title from the caller, so it cannot mint one.
The guarantee that holds across every path is narrower than "every stored title
satisfies the bound", and the comments say so: no CALLER-SUPPLIED title is
stored without being validated.
Seven codex rounds, 23 findings, ending CLEAN. Two of the findings were defects
introduced by earlier fixes in this same unit — an empty-title hole opened
through the legacy-protection clause, and a handler-side decision that dropped a
concurrent rename — both recorded on BUG-2833's trail. 38 mutants; every
behavioural fix has a mutant that is the defect at its site, killed by a named
test.
Prose sweep per CONVE-23: three comments asserting item titles are unbounded,
and a cost model resting on a ~2 MiB single-request title, corrected in place —
the guards they document still hold, because the bound is non-retroactive and
the cascade charges STORED titles.
Filed rather than bundled: BUG-2836, BUG-2839, BUG-2840, BUG-2842.
Closes BUG-2833, BUG-2831.
Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm
|
||
|
xarmian
|
b2c303c4bb |
docs(links,store,models): cite markdown.ts by symbol, and check it (BUG-2832)
Go comments describe the web renderer constantly and cite it by LINE NUMBER. Nothing verifies those citations — they cross a language boundary, so no compiler, test or linter has ever checked one — and they had drifted onto unrelated code. This converts all 32 to `markdown.ts::symbolName` form and adds the check that makes the conversion worth something. Scope note, because this is wider than the rider it was dispatched as. The BUG-2834 commit added the pattern constant near the top of markdown.ts, shifting the file by +45 lines and invalidating EVERY line citation into it — including the three BUG-2832 had confirmed were still accurate. Leaving 13 knowingly-wrong citations because they sit outside the files this unit otherwise touched is not the neutral option when this branch is what broke them. Happy to split this commit back out if the lead would rather hold the rider to its stated bound. While converting, five of the filing's six "suspect, not established" citations were settled by reading the shifted positions: :307, :478-481, :485, :513 and :516 point at a @param doc line, unescapeDocLinks, REF_PATTERN, the tail of parseCrossWorkspaceBody, and findItemByRef respectively. All substantively stale, not merely off-by-lines. That answers the filing's open question. Two guard tests, per the filing's own proposed fix shape: TestMarkdownCitationsNameLiveSymbols verifies every cited symbol is really declared in markdown.ts. This is the check a line number could never have. TestMarkdownCitationsAreNotLineNumbers bans the line-number form, so the fix cannot erode the next time someone reads a number off their editor gutter. The first version of the symbol check FAILED its negative control and that is the part worth reading. It asked strings.Contains(ts, "function "+sym) — a PREFIX match. Renaming resolveWikiBody to resolveWikiBodyRENAMED leaves "function resolveWikiBody" a substring of the renamed declaration, so the guard stayed green through precisely the rename it exists to catch. It passed its first real run and would have shipped as coverage. Fixed by requiring the following character to be one that cannot continue a JS identifier; the control now fires and names the symbol. Both guards are non-vacuity-asserted: the sweep fails if it finds fewer than 50 Go files, and the symbol check fails if it finds no citations at all. Currently verifying 7 distinct symbols across 29 citation sites. The line-number guard earned its keep before being committed — it caught three citations silently reverted when a file was restored from a snapshot taken before the conversion. |
||
|
xarmian
|
6114954236 |
fix(links,store): decide bracket qualification by the old title (BUG-2830)
An item whose LITERAL title starts with its own collection's slug plus a
slash — say "tasks/Setup" in collection "tasks" — stores an index row
byte-identical to a genuinely collection-qualified reference to an item
titled "Setup". Both read target_title = "tasks/Setup", collSlug =
"tasks". Renaming to "Renamed", the first must become [[Renamed]] and the
second [[tasks/Renamed]].
bracketRewriteAt inferred "this was qualified" from targetTitle merely
STARTING WITH collSlug + "/", which answers the second case for both. So
renaming the literal-titled item emitted [[tasks/Renamed]] — converting a
literal-title reference into a qualified one, resolved by a different
rule.
What that costs depends on what else the workspace holds, and both cases
are measured:
- With an item LITERALLY titled `tasks/Renamed` (slash included), the
link is STOLEN outright. resolveTitleTx tries an exact full-title
match before the qualified fallback, so that item wins and the
renamed item loses its backlink. Under the unfixed code the decoy
gains 1 backlink. Silent retarget, exactly as filed.
- With no such item, the emitted bracket still finds the renamed item,
because collSlug is that item's own collection and it now carries the
new title. The cost there is ambiguity wherever a same-titled sibling
exists, plus a later collection move breaking `[[tasks/X]]` where
`[[X]]` would have followed.
Repro delivered before the fix, as the filing required:
RewriteBracketAt("see [[tasks/Setup]] here", 4,
"tasks/Setup", "Renamed", "tasks")
=> "see [[tasks/Renamed]] here"
with the correct answer depending on information the function did not
have.
The discriminator is the renamed item's OLD title, and the cascade has
had it all along — cascadeTitleRename takes oldTitle and simply never
passed it down. It now rides on TitleEscaper (per-cascade, like
everything else there), and qualifiedFor decides by COMPARISON:
targetTitle == oldTitle -> literal
targetTitle == collSlug + "/" + oldTitle -> qualified
neither -> index drift, refuse
Literal wins when both could apply, and that is the correct precedence
rather than a convenient tiebreak: the renderer's stage 1 beats stage 2,
so a row pointing at this item resolved literally. resolveBrokenTitleLinks
already makes the same stage-1-over-stage-2 ruling for the same reason —
the discriminator existed in the codebase and was thrown away before
reaching the rewriter.
oldTitle is a required parameter of NewTitleEscaper rather than an
optional setter, so a caller that forgets it fails to compile instead of
silently getting the old behaviour back.
NO byte-length precondition guards the fold comparisons. strings.EqualFold
is Unicode simple case folding and case-equivalent strings can differ in
byte length — EqualFold("K", "K") (KELVIN SIGN) is true at 1 byte vs 3 —
so a length check is not a cheap pre-filter but a strictly narrower
predicate, and it made a qualified bracket whose title folds across
lengths read as index drift, leaving the link stale. The slug boundary is
still located by byte offset, which IS sound: collection slugs are
ASCII-lowercase by construction (store.slugify).
The frozen pre-refactor oracle is deliberately NOT updated — it is an
oracle, not live code. BUG-2830 is added to the named list of intentional
divergences from it, and the guarded corpus reaches the new function
through v0OldTitle, which states what the old implementation implicitly
assumed. Inputs where that assumption was WRONG cannot be produced by the
derivation and are pinned by name instead.
TestProjectRewrittenLen_IsLockstepWithTheRealPass grew a totalApplied
assertion: lockstep is trivially true when both sides refuse everything,
and this change makes the rewriter refuse more. It applies 5499 rewrites,
so it is measuring something. The codex-R2 overlap fixture was respelled
for the same reason — its two unrelated target titles are a shape a real
cascade cannot produce, so it would have decayed into two no-ops and lost
the regression; `[[A[[A]]]]` reproduces the no-op-then-overlapping-change
shape with reachable inputs and asserts it applies exactly one.
Negative-controlled four ways: reverting qualifiedFor to the prefix rule
kills the case-A regression in both its homes while the twin correctly
survives; making it accept drift kills the drift test (added because the
first mutation run showed that branch was unreachable by the whole
suite); the fold-length regression fails without the EqualFold fix; and
passing the WRONG oldTitle at the store call site kills five tests
including three pre-existing ones — the binding control, since the unit
tests pin qualifiedFor and only that shows the cascade hands it the right
value, which was the entire bug.
The severity above took two wrong turns before it was measured, and both
are recorded in the tests rather than quietly corrected. I first asserted
the retarget with a decoy that could not be stolen; then, finding that
decoy inert, concluded retargeting was impossible and wrote that into a
production doc comment. Neither conclusion came from reading
resolveTitleTx — both generalised one fixture's result. The two store
fixtures now split along exactly that line and each says which case it
pins.
|