mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-20 17:43:26 +00:00
c4d429d14c94de941ac761333cdac0a33895a05f
506 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xarmian
|
c4d429d14c |
fix: three codex round-6 findings, one premise corrected (TASK-2878)
Round 6 confirmed round 5 and found three. All three are real; one arrived
with an account of its own cause that the test refuted, and the corrected
route is narrower than the report.
## A NON-STRING RELATION DEFAULT, AND WHERE IT ACTUALLY GETS IN
The finding: injected defaults are never type-checked, so `42` or `[]` can
persist in a relation field. True, but not by the route described.
`MigrateFields` injects destination defaults ITSELF, so in the ordinary case
the key is present when `ValidateFieldsDetailed` runs and its type IS
checked — a numeric default lands in needs_value with "must be a string",
which is correct behaviour. My first test asserted the wrong thing and
FAILED against the fixed build, which is how I found this out.
The unchecked route is narrower: a NULL OVERRIDE deletes the key after
MigrateFields filled it, so validation injects the default itself — and its
own injection branch `continue`s PAST the type check. That is the one way a
non-string reaches a relation field unchallenged. The late-default pass owns
values that arrive from defaults, so it reports this one:
`invalid_shape`, a new reason, because every existing reason describes a
lookup that never happened.
Retargeting the test then exposed a second defect IN MY OWN FIX: the late
pass's `if len(late) == 0 { return nil, nil }` discarded the non-string drops
it had just recorded. The value vanished from all three buckets. Green on the
first route, silent on the second.
THE PARITY GATE FROM AN EARLIER COMMIT CAUGHT THE NEW REASON: adding
`invalid_shape` failed TestCopyPreflightDropReasonsAreRenderedByTheDialog
until the TypeScript union and CopyItemDialog learned it. That gate exists
because `referent_not_portable` shipped unrendered in BUG-2674, and it just
did its job on its author.
## A WHITESPACE-ONLY VALUE IS "NO REFERENCE", NOT A BAD ONE
The store resolver trims and ignores `" "`. The server wrapper checked the
UNTRIMMED string, so the value fell through to the visibility loop — and
since round 1's vanished-target arm turns a missing lookup into a refusal,
`" "` came back as not_found instead of an empty field. A defect my own
round-1 fix introduced: before it, that path did `continue`.
## A MALFORMED CARRIED VALUE IS NOT "NOT PORTABLE"
The cross-workspace branch dropped every carried value without looking,
including non-strings, and labelled them `referent_not_portable` — a false
account of why the value is going. It is not a reference at all. Left in
place now for ValidateFields to reject on shape, which is what the
SAME-workspace branch already did with it: the two modes disagreeing about
one malformed value was the defect.
## Counterfactuals
Skip non-string defaults again -> TestCopyEndpoint_NonStringRelationDefault
DETECTED. Restore the untrimmed skip -> TestRelationDoors_WhitespaceOnly
DETECTED. Both build-checked, and the first form of the second mutant did NOT
build (unused import) — reported as such rather than scored, since a
non-compiling mutant produces no failures and reads as survived.
Gates: internal/server ok 324.8s · internal/store ok 344.5s · internal/mcp ok
16.9s · go vet clean · gofmt clean · make lint 0 issues · npm run check 0
errors. Postgres green on the parent commit (store 596.1s, server 299.4s);
re-running on this tree.
|
||
|
xarmian
|
ae8551274b |
fix: codex round-5 finding — the other direction of the origin guard (TASK-2878)
One finding, P2, and it is the tail of round 4's fix. Round 4 stopped labelling a null-source key `migrated`, because validation treats null as missing and fills the DESTINATION's default in its place — so `migrated` named the source for a value the destination chose. With NO destination default there is nothing to fill it: the null is what carries, and `default` names a value the schema never declared. One guard, wrong in both directions, and the reason is the same in both: "was the key present in the source" is not "where did the final value come from". The label now follows the value: - source value, non-nil -> migrated - source null, destination default -> default - source null, NO default -> migrated `from` is what a dialog uses to say "this came across" versus "this is the destination's default", so either error is the preview misreporting the thing it exists to report. ## Counterfactual Remove the round-5 branch so a null source always reads as `default` -> TestCopyEndpoint_NullSourceWithoutDefaultIsNotReportedAsDefault DETECTED, build-checked first. The mutant also confirms the key really is in `carried` in that scenario — the test returns without asserting if it is not, so without the mutant its green would have been consistent with a vacuous pass. Gates: internal/server ok 288.8s · internal/store ok 302.4s · internal/mcp ok 13.7s · go vet clean · gofmt clean · make lint 0 issues. Postgres green on the parent commit (store 626.4s, server 311.9s, private container port 5481); re-running on this tree. |
||
|
xarmian
|
f4d733e57e |
fix: two codex round-4 findings — oracle race, null-source provenance (TASK-2878)
Round 4 confirmed round 3 and dropped from P1 to P2, which is what
convergence looks like. Both still real.
## THE ORACLE COLLAPSE HAD A HOLE, AND MY OWN COMMENT ASSERTED IT DID NOT
Round 3's fix collapses `wrong_collection` to `not_found` when the requester
cannot see the target. It needs the item to ask that, and when the second
lookup came back nil — deleted between the resolver's read and this one — it
did `continue` under a comment reading "vanished since; already the safe
answer".
That comment was wrong. Nothing had rewritten the reason, so the issue still
said `wrong_collection`, whose message tells the caller the value named
something a moment ago. The disclosure round 3 closed, reachable by a race.
Collapsed now.
Worth stating plainly because it is the same defect twice: I wrote a comment
asserting a state ("already safe") that the code did not establish, and I
wrote it in the fix for exactly that disclosure.
## A NULL SOURCE VALUE IS NOT A CARRIED ONE
The preflight marked any key PRESENT in the source's field map as
`from: "migrated"`. A source key holding `null` is present and carries
nothing: MigrateFields keeps the key, the relation pass skips a nil value,
and validation then treats it as missing and fills the DESTINATION's default
in its place. So the response reported the source as the origin of a value
the destination chose.
`from` is what a dialog uses to say "this came across" versus "this is the
destination's default", so naming the wrong one is the preview lying about
the thing it exists to report. Presence with a VALUE is the test now.
Not relation-specific — any nulled source field with a destination default
had it — so the guard is on the shared origin loop rather than beside the
relation pass.
## Counterfactuals
Remove the null guard from the origin loop ->
TestCopyEndpoint_NullSourceRelationIsNotReportedAsMigrated DETECTED
(build-checked first).
The oracle-race collapse has NO test, for the reason the vanished-target
refusal has none: reproducing it means deleting a row between two lookups
inside one request, and a test that faked the interleaving would pin the
fake. Both are listed in the PR body under "Not tested, and why".
Gates: internal/server ok 343.8s · internal/store ok 349.9s · internal/mcp ok
22.1s · go vet clean · gofmt clean · make lint 0 issues. Postgres green on
the parent commit (store 553.9s, server 297.1s, private container port 5481);
re-running on this tree.
|
||
|
xarmian
|
8fedc5487c |
fix: three codex round-3 findings — oracle, bulk defaults, required (TASK-2878)
Round 3 confirmed round 2's fixes and found three more. Codex still could not execute anything, so all three were static reads I verified myself. ## `wrong_collection` DISCLOSED EXISTENCE The store emits `wrong_collection` when a value names a LIVE item outside the field's declared target, and the server's visibility layer skipped any key that already carried an issue. So the message — "is not an item in collection X" — told a caller the value EXISTS, distinguishably from the `not_found` a nonexistent value gets. An existence oracle for anyone who cannot see that item, and the exact shape the `not_found` collapse exists to prevent. Collapsed to `not_found` when the requester cannot see the target, and KEPT otherwise: "you linked a task where a person belongs" is the useful half of this reason, and blanket-collapsing would have passed the security leg while destroying it. Both legs are in the test for that reason. Needed one new store export, `ResolveRelationTarget`: deciding what to disclose about a live item requires the item, and the resolver is the only thing that maps a value to one under the no-slug rule. ## BULK STATUS / PRIORITY BYPASSED RELATION DEFAULTS `bulkFieldUpdate` resolves only the keys `changes` names — correctly, since re-litigating stored values would freeze legacy items — but `ValidateFields` runs BEFORE it and INJECTS schema defaults. A defaulted relation was persisted raw: never canonicalised, never checked against its collection. The same late-arrival the migrate doors hit, reached by a different route, and the narrow pass built for them covers it unchanged. `dropped_fields` now rides every bulk op's activity row, not just the move branch: a status or priority change can discard a relation default too, and a drop nobody records is the defect BUG-2674 closed. ## A REQUIRED RELATION COULD END UP ABSENT AND REPORTED VALID The late pass deletes a key AFTER validation passed, so nothing re-checked required-ness: a required relation whose default did not resolve left the item with the field absent, and the preflight reported `valid: true`. Re-running validation is not the fix — it would re-inject the same broken default. There is no valid value for that field, so the doors refuse: `missing_required_fields` on move and bulk move, a FieldValidationError on the copy, and on the preflight a `needs_value` row with `valid: false`. That split is the one this pair has everywhere else — the preview says what is wrong, the copy refuses. ## Counterfactuals Remove the oracle collapse -> the visibility leg DETECTED. Remove the bulk-update late pass -> DETECTED. Remove the required-relation refusal at the copy door -> DETECTED. All build-checked first. Gates: internal/server ok 352.1s · internal/store ok 358.7s · internal/items ok · internal/mcp ok 21.0s · go vet clean · gofmt clean · make lint 0 issues. Postgres green on the parent commit (store 596.7s, server 294.7s, private container port 5481); re-running on this tree. |
||
|
xarmian
|
2e9df62c80 |
fix: two codex round-2 findings — late defaults, UUID disclosure (TASK-2878)
Round 2 confirmed the three round-1 fixes and found two more. Both real,
both verified in the code before acceptance; codex still could not run
anything ("the read-only filesystem prevented Go cache/temp creation").
## THE DESTINATION-DEFAULT FIX ONLY COVERED HALF THE WAYS IN
The migrate doors resolve BEFORE they validate, and that order is
load-bearing: the required-field check has to see a value referent
resolution dropped, or a dropped value in a REQUIRED relation field would
store the item with the field absent instead of refusing. But
`ValidateFields` INJECTS schema defaults, so a default can land after the
resolver has finished. Two ways in, both now closed:
- a NULL OVERRIDE deletes the key and the default fills the hole, arriving
uncanonicalised — a `PEOP-1` default reached the row as the literal
string;
- a default the resolver DELETED as unresolvable is put straight back by
validation, and `StillDropped` then suppresses the warning about it.
Dropped, reported, restored, and reported as not-dropped.
`ResolveLateRelationDefaults` is a narrow second pass over exactly the keys
validation added. Reordering the two would have traded this defect for the
required-field one; a second pass costs one lookup in the rare case a
relation field declares a default and nothing otherwise. A default is
asserted by nobody, so an unresolvable one is DROPPED and reported, never
refused — the same disposition the main pass gives
RelationOriginDestinationDefault, which is what this is: the same origin,
arriving late.
THE SNAPSHOT IS TAKEN AFTER THE MAIN PASS, NOT BEFORE, and getting that
wrong is what the second sub-case caught. Snapshotting before the pass
treats a key the pass DELETED as already examined, so the late pass skips
the very value validation just put back — the arrangement that hid it. My
first version did exactly this and the second sub-case failed on it.
## A REFUSAL FOR AN INVISIBLE ITEM DISCLOSED ITS UUID
The store resolver rewrites a supplied ref into its target's id before the
visibility check runs, so a message built from the resolved value handed
back the canonical UUID of an item the requester may not see — confirming
both its existence and its identity. That is the existence oracle the
`not_found` collapse exists to prevent, reopened by the message. Every issue
this function raises now quotes what the CALLER sent.
## Prior findings, per codex
Override visibility: fixed. Bulk move dropped list: fixed. Nil-target race:
no dereference. Destination defaults: this commit.
## Counterfactuals
Neuter the late-default pass -> TestCopyEndpoint_LateInjectedRelationDefault
DETECTED on both sub-cases. Restore the canonical UUID in the message ->
TestCopyEndpoint_InvisibleRelationOverrideIsRefused DETECTED. Both mutants
build-checked first.
Gates: internal/server ok 346.3s · internal/store ok 351.3s · internal/items
ok · internal/mcp ok 22.9s · go vet clean · gofmt clean · make lint 0 issues.
Postgres green on the parent commit (store 534.5s, server 268.5s, private
container port 5481); re-running on this tree.
|
||
|
xarmian
|
ed03d488da |
fix: four codex round-1 findings — origin, visibility, bulk reporting (TASK-2878)
Codex round 1 named four; three were real P1s and I verified each in the
code before accepting it. Codex could not run anything ("Go could not
create its build cache because the workspace is read-only"), so every
finding here is a static read that I confirmed and pinned.
## A THIRD ORIGIN, not two
`items.MigrateFields` injects the DESTINATION schema's defaults for keys the
source item has nothing for. My classifier split on `supplied` versus
everything-else, so a destination default was filed as CARRIED — and on a
cross-workspace copy every carried relation drops without a lookup. The
destination's own default was discarded and reported `referent_not_portable`,
which is flatly false about a value the destination chose.
There are three origins: SUPPLIED (refuse on failure), CARRIED from the
source item (cross-workspace: drop as not-portable), and DESTINATION DEFAULT
(resolve against the destination in BOTH modes; drop with the resolver's own
reason on failure, because nobody in this request typed it). Telling the last
two apart needs the source field map, which all four migrate doors have as
`currentFields`, so it is now a parameter.
Empty values are skipped at every origin. An empty relation is a cleared
field, not a referent, and reporting it as dropped tells a user they lost
something they never had.
WHAT THE FIRST VERSION OF THIS TEST PROVED: nothing. The mutant that reverts
the classifier SURVIVED it. `ValidateFields` re-injects the default after my
resolver deleted the key, so the value comes back either way and
`StillDropped` filters the false report out — the end state is identical
unless the default is a REF. A UUID default is already its own canonical
form, so "resolved" and "dropped then re-injected raw" produce the same
bytes. With `PEOP-1` as the default the mutant is DETECTED, because only a
resolved default lands as the id.
## SUPPLIED OVERRIDES AT THE MIGRATE DOORS SKIPPED THE VISIBILITY CHECK
The four write doors go through `s.resolveRelationReferents`, which adds
`checkItemVisible` on top of the store resolver. The migrate doors called
`store.MigrateRelationReferents` directly — it is a store function and cannot
answer a request-scoped question — so their SUPPLIED half, which this unit's
own rule calls an ordinary write, resolved against the database alone. A
caller able to edit both collections could point a relation at an item they
cannot see.
The ROLE is the part worth getting right. For `move` it is `workspaceRole(r)`.
For copy and preflight it is the caller's role in the DESTINATION, and
`CrossWorkspaceAccess.Role` is exactly that — its own doc says never to
substitute `workspaceRole(r)`. So `resolveRelationReferents` now takes the
role explicitly (`resolveRelationReferentsAs`), and the new
`refuseInvisibleRelationOverrides` runs at all three doors with the right one.
At the copy it runs in the HANDLER, before the store call: a pre-write
refusal must not open a transaction to roll it back, and the preflight runs
the identical check — DR-6's "the preview IS the copy" only holds if both
doors refuse the same request.
## BULK COLLECTION MOVE DISCARDED FIELDS SILENTLY
`bulkMoveCollection` has populated `result.Dropped` since MigrateFields
existed and NOTHING read it — the only reference in the file was my own
append. BUG-2674 fixed the single-item door and left this one, so a bulk move
discarded values with no record anywhere. Pre-existing, and routing relation
drops into the same dead list is what made it mine to fix.
Reported on the activity row, same key, same joined-string shape and the same
BUG-2628 reason as `handleMoveItem`, filtered against the final map so the
report is true when written. Threaded as an out-parameter, deliberately: only
this branch produces drops, the caller needs them for ONE activity row per
item, and a third return value would put `nil` in fourteen unrelated returns.
## THE P2, AND WHAT IT IS NOT PINNED BY
`resolveRelationReferents` did `if item == nil { continue }` after the
visibility read — "treat a race as someone else's 404". It now refuses with
the same `not_found` the resolver would have given moments later. This whole
unit exists to keep a dangling referent out of the blob, and a target that
vanished mid-request is the one case where waving it through would have been
deliberate.
NO TEST. Reproducing it means deleting a row between two reads inside one
request, and a test that faked that would pin the fake. Stated here rather
than left to look covered.
## Counterfactuals
Every fix has a mutant that its own test detects, each build-checked first:
classifier reverted -> DETECTED (destination default); empty-skip removed ->
DETECTED; visibility helper neutered -> DETECTED at both the move door and
the copy/preflight pair; bulk-move report removed -> DETECTED.
Gates: internal/server ok 224.6s · internal/store ok 261.5s · internal/items
ok · internal/mcp ok 15.1s · go vet clean · gofmt clean · make lint 0 issues.
|
||
|
xarmian
|
34861f8658 |
feat(mcp): ToolSurfaceVersion 0.29, and the drop-reason renderer it exposed (TASK-2878)
PLAN-2857 U1. The bump, its documentation sweep, and the consumer this
change turned from a rare wart into a routine one.
THE BUMP, at 0.29 rather than 0.28. Rebasing onto main found
|
||
|
xarmian
|
94eddb9b8a |
test(server): the per-door x per-provenance table, and the defect it found (TASK-2878)
PLAN-2857 U1. `internal/store` already tests the resolver exhaustively, but those tests call it DIRECTLY: they vouch for the component and say nothing about whether any door is bound to it. A door that never calls the resolver passes every one of them. This table is the binding claim — one leg per (door, provenance) pair, driven through the handler a client reaches. PROVENANCE IS THE SECOND AXIS BECAUSE THE ANSWER DEPENDS ON IT, not for symmetry. A SUPPLIED value is the caller's assertion and an unresolvable one is refused; a CARRIED value was asserted by nobody, and refusing it makes legacy items un-updatable, un-movable and un-copyable — the failure this unit would otherwise CAUSE while fixing another. Which provenance a door sees is a property OF THE DOOR, and getting it wrong is invisible until a legacy item meets it. THE TABLE FOUND ONE, ON ITS FIRST RUN. `bulkFieldUpdate` merges the item's STORED fields blob with the caller's `changes` before validating, and the resolver was pointed at the MERGED map — so a bulk status move or set-priority re-litigated every stored relation value and REFUSED the item. An item carrying a legacy relation value had its status and priority frozen by a field the operation never mentioned. Fixed the way the fields_patch door already handles it: resolve only the keys the operation CHANGES, read out of the coerced map so the value is final, written back so a supplied ref is still canonicalised. Verified by prediction before the run and by the leg failing against the unfixed code. THE DISPATCH MUTANT, which is what makes the table's coverage a measurement rather than a hope. Wire ONLY the two `extractParentLink` doors (update fields, update fields_patch) and neuter the other six by swapping their field-map argument for an empty one — types unchanged, so the mutant compiles and its verdict means something: build OK · failing legs: create (both), move (both), bulk move, bulk update supplied, copy, preflight. Update and update-patch pass. Exactly the six unwired doors, and only those. Then each door alone, eight runs: every one detected by its own legs and no other door's. That is the claim the table exists to make — each leg reaches its OWN door rather than being satisfied by a neighbour's check. ONE DOOR NEEDS A WEAKER INSTRUMENT, AND THE TEST SAYS SO. No bulk op puts a relation key into `changes` — `op` is a closed list and the only field values any of them set are `status` and `priority` — so that door's SUPPLIED branch is unreachable from outside. The first mutant run proved it: unwiring door 4 alone left every black-box leg passing. It gets a direct-call leg, labelled as vouching for the FUNCTION and not for a binding that does not exist yet, and kept for the same reason `bulkMoveCollection`'s refusal branch is kept: the day the bulk path grows per-field overrides, the branch must already refuse rather than be a silent no-op nobody notices is missing. Every refusing leg has a resolvable counterpart. Without them the table is equally consistent with a build that refuses every relation value. Gates: internal/server ok 156.1s · go vet clean · gofmt clean · make lint 0 issues. |
||
|
xarmian
|
ee607047dd |
feat(copy): the two cross-workspace doors resolve referents, with their pin (TASK-2878)
PLAN-2857 U1, doors seven and eight. `migrateCopyFields` and `handleCopyItemPreflight` now take their relation decision from `store.MigrateRelationReferents` — the same function the four write doors and the two move doors already call, which is the entire reason it exists. The defect this closes: MigrateFields matches on key and TYPE, so a same-named `relation` field carried a SOURCE-workspace item id across the boundary and the preflight reported it as a clean carry. What landed in workspace B was a value naming a row in workspace A — unrenderable, and indistinguishable on read from a legitimate reference. Provenance decides, as at the move doors. A CARRIED value on a cross-workspace copy is dropped without a lookup (no id from A can mean anything in B) and reported through the `dropped_fields` channel BUG-2674 established; a SUPPLIED override is an ordinary write and an unresolvable one is refused, 400 validation_error on both doors, rendered by the same `store.RelationIssuesMessage` so one refusal cannot acquire two phrasings. `internal/server`'s `relationIssuesMessage` now delegates to it: the eighth door refuses from inside `store`, so the sentence had to be reachable there. Two things are threaded rather than re-derived, and both are load-bearing: - The TRANSACTION, not the pool. `migrateCopyFields` becomes a method taking a Queryer, and `copyItemAcrossWorkspacesTx` passes its `tx`. That function has held a transaction since its second statement, so a pool read from inside it can wait for a free connection while every pooled connection is blocked on this transaction's locks — the starvation shape BUG-2409 fixed for the attachment planner and this repo keeps a deterministic test for. This is what the day-70 handoff named as the reason these two doors were not wired with the other six. - The MODE comes from the `scope` MigrateFields was already given, not from a second boundary test. Two independent answers to "is this crossing a workspace" is how one request gets migrated one way and validated the other, and this path also serves a copy whose target IS the source workspace, where relations resolve and survive exactly as on a move. The destination workspace id is the resolution scope: a supplied override is a write into B and must name something that exists there. THE PIN, and why it is not a per-door table. These two doors sit in different PACKAGES and the code at both sites says so is how they drift unnoticed. A table with a row per door can be fully green while the two disagree about one request, which is the defect rather than a gap in coverage of it. So every case sends ONE body to BOTH endpoints: - carried relation — must drop on both, and the preflight must say referent_not_portable rather than the generic no_target_field, which is false here (the destination DOES declare the key, so that answer sends the reader to fix a schema that is fine); - supplied + unresolvable — both refuse, same status, same code, both name the offending field and value, and nothing is written; - supplied + resolvable — the positive control, supplied as a REF so resolution is visible in the result. Without it the first two legs are equally consistent with "relations always fail". Negative controls run, all three mutants BUILD-CHECKED first (a non-compiling mutant produces no `--- FAIL` lines and reads as survived): both doors unwired = DETECTED; preflight unwired alone = DETECTED; store unwired alone = DETECTED. Each single-door mutant failing is the pin's whole claim — neither door can be wired without the other. CONVE-23 sweep: the preflight's LIMITATION comment said this gap belonged in MigrateFields "for both callers at once". That is now false in its prescription as well as its premise — `internal/items` is DB-free by construction and cannot ask whether a string names a live item — so the comment records where the fix actually went and what of it remains open (`computed`, `terminal_options`, `unique_scope`). Gates: internal/server ok 170.0s · internal/store ok 296.1s · internal/items ok · go vet clean · gofmt clean · make lint 0 issues. |
||
|
xarmian
|
f6b1dd96cc |
feat(server): the two same-workspace migrate doors resolve and report (TASK-2878)
PLAN-2857 U1: `handleMoveItem` and `bulkMoveCollection` now take their relation decision from `store.MigrateRelationReferents` — the same function the two copy doors will call, which is the point of it existing. Within a workspace the targets are still present, so a correctly-related item KEEPS its relation across a move; only an unresolvable value is dropped, and it joins the `dropped_fields` report BUG-2674 established rather than failing the move. Refusing carried values here would make every legacy item permanently unmovable, and `internal/items` has accepted any string for a relation all along, so "legacy" is most of them. The bulk path carries no per-field overrides — only `status` — so every relation value reaching it is CARRIED and nothing there can refuse. It passes nil for `supplied` to say so, and keeps the refusal branch: it is unreachable today and stops being a silent no-op the day that path grows overrides. internal/server ok (270.8s), internal/store ok. KNOWN GAP, recorded rather than half-built: the two CROSS-workspace doors are not wired yet, and the reason is a real constraint rather than running out of road. `migrateCopyFields` is called from `copyItemAcrossWorkspacesTx` with a transaction already open (`s.db.Begin()` at the top of that function), so resolving a SUPPLIED override there would issue POOL reads while holding a tx — the deadlock shape this repo keeps a deterministic test for. The carried half needs no lookup at all and is safe; the supplied half needs `GetCollectionBySlugQ` / `GetItemByRefQ` so the resolver can run on the tx's connection, which is exactly the `...Q` convention the store already uses (`GetItemQ`, `getCollectionInWorkspaceTx`, `uniqueSlugQ`). Adding those two is the remaining work, and it is what makes one function genuinely serve all four doors. |
||
|
xarmian
|
987fc79fde |
feat(server): refuse unresolvable relation values at the four write doors (TASK-2878)
PLAN-2857 U1, second slice: the doors that take CALLER-SUPPLIED field values now refuse a relation value that does not name a live item in the declared target collection — create, update (full fields), update (fields_patch), and bulk update. The server half adds the one thing the store resolver deliberately does not: visibility. It folds into the SAME `not_found` reason rather than getting its own, because "that item exists but you may not see it" is an existence oracle, and this codebase has a standing rule against handing one out. Ordering at every door is after the shape check and after coercion, so one bad value produces one error rather than two describing it differently, and so the value is in its final form when it is resolved. `fields_patch` examines only the keys the patch carries — the resolver skips absent keys — so an unresolvable value already stored on an item is not re-litigated by an update that does not touch it. That mirrors the undeclared-key rule immediately above it, and it is what stops this turning every edit of a legacy item into a failure. Refusals use the ORDINARY `validation_error` shape with no new details key. The MCP stdio transport classifies errors by matching CLI stderr prose, so a structured field it cannot see would help nobody there, and a new error shape is a contract change for every client. Existing suites unchanged: internal/server ok (224.5s), internal/store ok (258.0s), internal/items ok. Nothing in the tree was writing a bogus relation value through these doors, which is what made this slice safe to land before the per-door pins. |
||
|
xarmian
|
b437cc582d |
feat: item reminders — the fire-at-an-instant primitive, and one overdue rule for all four surfaces (IDEA-2641, closes #1010) (#1244)
* feat(store): item reminders — the fire-at-an-instant primitive (IDEA-2641) Adds the storage, the scheduler tick, and the canonical event for one-shot item reminders (GitHub #1010). Nothing in Pad acted at a target time before this: a due_date makes an item show up as overdue once somebody asks the dashboard, so "revisit TASK-X on the 1st" had to live in an external cron. A TABLE, NOT A SCHEMA-FIELD ANNOTATION. The design sketch proposed marking schema date fields with a `reminds: true` key on models.FieldDef; recon overturned it. Such a key does not survive an ordinary collection edit, two independent ways: the web editor destructures each field into an EditableField and rebuilds a fresh definition key-by-key on save, so unknown keys are dropped (`pattern` and `unique_scope` survive only because two lines were hand-added for them), and models.CollectionSchema has fixed fields with no catch-all, so any Go unmarshal+marshal round-trip strips unknown properties — the hazard retargetRelationFieldsTx mutates raw JSON to avoid. Both failures are silent and both disarm a whole collection's reminders at once. It is the same defect class that moved traits out of the schema column in TASK-2657. The table also gives the lifecycle a home. A reminder is armed, then fired, then acknowledged, and a re-arm returns it to armed — per-reminder state a field definition has nowhere to keep. remind_at is an RFC3339 UTC instant, deliberately not a `date` schema value: those admit both YYYY-MM-DD and full RFC3339 and are compared against the SERVER'S LOCAL calendar day. A fire-at time cannot carry that ambiguity. The remaining timezone question for due_date is filed separately. Firing is one transaction per reminder carrying BOTH the fired_at write and the outbox insert. That pairing is the point: a fired_at committed without its event is a reminder that silently notifies nobody and can never be retried, because the row has left the armed set; an event without fired_at fires every tick forever. The UPDATE's own `fired_at IS NULL` predicate is the arbiter, so two instances ticking at once produce exactly one winner. item.reminder_due is admitted to the closed events/1 set as v1.2, with a new PayloadReminder family and no SSE name. The subject is the REMINDER, not the item: two reminders can be armed on one item, so an item-subject event could not say which fired, and the reminder id is what an acknowledgement addresses. A new payload family rather than reusing the item snapshot for the same reason — a snapshot would validate and still not answer the only question the event exists to answer. No SSE name in v1 because the poll surface is the contract; adding one later is additive, removing one is not. Ack is explicit and nothing else acks. An item reaching a terminal status deliberately does NOT ack: that would make every status write a reminder mutation, and it would silently consume a reminder set to fire after the work was done. * feat(server): reminder surfaces, and one shared overdue rule for all four Second half of IDEA-2641: the HTTP surface, the scheduler tick's wiring, and the fix for the finding that justified the unit — `ready` / `next` did no date handling at all. OVERDUE NOW HAS ONE IMPLEMENTATION. It used to live inline in the dashboard's attention loop, which meant `pad project stale` inherited it (it filters that very list) and the recommendation surface never saw it. So a deadline reached the two surfaces that REPORT on work and never the one an agent PULLS from. overdue.go is now the only place that decides, and all four call it. Two behaviour changes fall out, both deliberate: - An overdue item bypasses the orphan branch's high/critical priority gate. That gate was where a deadline quietly stopped: a low-priority item three weeks late was reported by `stale` and never suggested by `next`. - Overdue sorts above in-progress. The list is capped at three, so a rank below in-progress would not merely order the deadline lower — on any workspace with three things in flight it would keep an overdue item off the surface entirely, which is indistinguishable from not shipping this. The server-local-today comparison is UNCHANGED and known to be wrong for multi-timezone deployments; it is filed as its own item with the cloud case stated. Changing what "overdue" means on every existing instance inside a change about where the rule LIVES is the kind of behaviour change nobody reviews. Fired reminders reach `next` / `ready` two ways, from one filtered list: PendingReminders is the addressable form (it carries the id an ack needs), and a prepended suggestion is the rendered form. They are prepended AFTER the cap rather than entered as ranking candidates — a reminder is not a task competing on priority, and whether it appeared should not depend on how busy the workspace is. Terminal-item reminders are FILTERED from the surface, never acked. Acking on terminal status would couple every status write to reminder state and would consume a reminder armed to fire after the work was done. The row stays exactly as the user left it; the distinction is observable, and asserted. Three guard tests caught this change and each was answered rather than silenced: - The request-body reader guard was right: the handlers now go through decodeJSON, inheriting the NUL refusal and the size cap. - The canonical-events guard was right: item.reminder_due is admitted to the duplicated contract table as SPEC-3 v1.7, with the reminder subject kind and the new payload family. SPEC-3's own text owes the same amendment. - The NUL census asked for a decision on eight new columns. None carries caller text: ids and FKs are server-generated, four are the server clock, and remind_at is now re-parsed and re-formatted in the STORE as well as at the edge — so the stored value is always machine-produced from a parsed time and no caller bytes reach the column. The doc comment that used to say "the caller normalizes" protected nothing. Regenerating the baseline also found that GEN_NUL_BASELINE=1, which the test's own instructions name, was never implemented — the flag did nothing, so the documented path was hand-editing the file. Implemented, so the next reader gets the mechanism the instructions promise. * test(reminders): the lifecycle, the four surfaces, and 22 killed mutants Every test here was designed against a specific mutation and the mutation was RUN. A green suite proves nothing about a suite nobody tried to break, and three of the mutants I first wrote were not experiments at all. Store (10 mutants, all killed): candidate predicate <= flipped to >=; the event emission lifted out of the fire transaction; the fire UPDATE's `fired_at IS NULL` arbiter removed; the RowsAffected check ignored; re-arm clearing fired_at but not acked_at; ack losing `fired_at IS NOT NULL`; the poll surface losing `acked_at IS NULL`; normalizeRemindAt no longer refusing; it dropping .UTC(); GetReminder losing its workspace scope. Surfaces (12, all killed): the priority gate no longer bypassing on overdue; the sort no longer ranking overdue first; attention leaving the shared helper; the reason losing its OVERDUE prefix; the comparison flipped to >; terminal items no longer skipped; terminal reminders no longer filtered; the filter ACKING instead of hiding; reminders appended instead of prepended; the tick running on a far-future clock; ack answering 200 for an unfired reminder; parseRemindAt accepting a bare date. THREE MUTANTS DID NOT COUNT ON THE FIRST PASS and were rewritten. Two failed to compile (`if false` orphaned a variable; deleting a parse orphaned an import) and one had an anchor matching two call sites. A non-compiling mutant emits zero FAIL lines and reads exactly like a surviving one — it invents a hole that is not there — so the harness reports BUILD-FAIL and ANCHOR-BAD as outcomes distinct from SURVIVED. It also restores files from an in-memory copy rather than `git checkout`, which would delete uncommitted work in the tree. ONE MUTANT GENUINELY SURVIVED and the test was at fault, not the mutant: appending rather than prepending reminder suggestions was undetectable because the fixture had a single item, so the reminder sat at index 0 either way. The fixture now fills the three-item cap with in-progress work, where an appended reminder lands fourth and vanishes. Faithful mutant, weak test — checked in that order. The same lesson shapes the four-surface fixture: it is a LOW-priority open orphan, because that is the case the old code handled worst. A high-priority task would have made the ready/next leg pass against the unfixed tree, which is a green that measures nothing. Negative controls throughout: a future deadline is not overdue and does not reach the gate bypass; a tick with nothing due fires nothing; a completed item is neither overdue nor suggested. Without them a helper that reported every date, or a tick that fired everything, would satisfy every positive leg. The lead's pin is asserted in both directions: a fired reminder on a done item is ABSENT from the surface and PRESENT and still unacknowledged in the table. Asserting only the absence would pass against an implementation that consumed the row, which is the behaviour the pin exists to forbid. * feat(mcp): pad_item.remind + ack-reminder, ToolSurfaceVersion 0.28 An agent that can RECEIVE a reminder but not set one has half the primitive. The poll surface is pad_project.next / ready, both long exposed, so reminders already reached agents — what was missing is the other half: deferring a piece of work is exactly the moment an agent knows when it wants to be asked again, and it had no way to say so. Two additive actions, two optional params. Nothing existing moved, so a v0.27 consumer enumerating neither is unaffected — the v0.13 / v0.11 / v0.8 disposition, which likewise wired existing CLI verbs onto the catalog. remind_at REFUSES a bare date rather than reading it as midnight. Worth stating because the `date` schema type accepts YYYY-MM-DD and a caller will reasonably try it here: a bare date names a 24-hour span, and choosing an hour inside it would fire at a time nobody picked. Re-arm and disarm stay CLI-only. Both address a reminder by an id the agent would have to list first, and no listing action exists on this surface — a door with no handle. Adding them later is additive. Five guards had to be taught, and each was answered on its merits rather than excluded: the HTTP parity test (route mappers added, so the actions work on the remote transport rather than being advertised and unrouted), the read-only catalog's cmdhelp fixture and expected cmdPath map, the field- conflict classifier (remind_at / reminder_id are NOT field writers — a reminder is a row in its own table addressed by its own id, so listing them as classified sources would have pointed detectFieldConflicts at something that is not a field source), and the instructions.md / README action tables. That machinery is why the version bump is safe to make now, and it earned its keep on this change: every one of the five failed on the first build after the catalog entry landed. CONVE-23 sweep for prose this falsifies: - SPEC-3 (DOC-2653) amended to v1.7 in the room, recording item.reminder_due with its new subject kind and payload family — the first canonical event with no user mutation behind it, since a scheduler tick produces it. - CLAUDE.md gains the reminder routes, the CLI verbs, and the v0.28 entry. It was also stale at 0.26 with NO v0.27 entry at all: the 0.27 unit swept instructions.md and README.md and missed this file. Both added. - skills/pad/SKILL.md gains the verbs and a routing entry, including the two things an agent will get wrong — the time is an instant, so ask for a time of day rather than picking one, and finishing the item does not acknowledge the reminder. * fix(reminders): codex round 1 — four findings, all real, all with a pin Round 1 found four defects and refuted none of them. Each fix carries a test that fails against the code as it was, and each of those was mutation-checked. **P1 — pending reminders bypassed item-level visibility.** Every other dashboard section reads `allItems`, which the store already scoped to the caller's collections AND their granted item ids. The pending-reminder list is a direct workspace-wide query and inherited none of that, so a guest holding a grant on ONE item could read the refs and titles of every other item in the collection through its reminders — an item-level leak wearing a notification's clothes. Now filtered with the same `isItemVisibleToGuest` call the sibling sections use. The test's two items share a COLLECTION on purpose: a collection-level filter was already applied, so separate collections would have made it pass against the unfixed code. **P1 — soft-deleted items could starve the queue permanently.** Candidate selection ignored `deleted_at`, and `fireOneReminder` rolls back when it finds the item gone — which leaves the reminder ARMED and therefore a candidate again on the next pass. Candidates are ordered oldest-first and bounded by a limit, so enough archived reminders fill every batch and no live reminder ever fires. Silent, too: the tick reports zero fired and looks idle. Excluded in the candidate query rather than skipped downstream, so those rows never occupy a slot; the reminders themselves are kept, so restoring an item restores its reminder with it — asserted, because a fix that reaped them would pass the starvation test alone. **P2 — the pass stopped at the first failing reminder.** The per-reminder transaction exists precisely so one unfireable row cannot hold back the rest, and `return fired, err` made that comment false — with candidates oldest-first, one persistently broken old reminder blocks every newer one forever. Now continues and joins the errors, so a pass that fired seven and failed three reports both halves rather than reading as clean. The loop is split behind an injected seam because a real mid-transaction failure is not reachable from outside: the database refuses the corrupt rows that would cause one (verified — invalid JSON in items.fields is rejected by the schema). **P2 — suggestions dropped the reminder id.** The docs tell an agent to acknowledge what it sees in next/ready, and the payload carried no handle: a stateless poller could read the reminder and had no way to retire it, so it would be shown the same item forever. `DashboardSuggestion` now carries `reminder_id` (omitempty), `pad project next` prints the exact ack command, and the test acks with the id the surface handed out rather than merely checking the field is populated — a wrong-but-present id satisfies equality with itself. Four mutants, four killed; one was rewritten first because its anchor matched two call sites and was therefore not an experiment. * fix(reminders): codex round 2 — four findings, all real **`--rearm` was unusable.** `ExactArgs(1)` forced an item ref that the rearm branch then ignored, so the flag could not be reached without supplying a ref that was silently discarded. Now `MaximumNArgs(1)`, with each mode checked explicitly: a ref is required to arm, and a ref supplied ALONGSIDE `--rearm` is refused rather than ignored — it names an item the reminder may not even belong to, and quietly dropping it is how a user learns nothing about the reminder they just moved. **`unremind --format json` emitted plain text**, breaking the parseable-output contract every sibling command honours. **The MCP `ref` param did not list `remind`.** Agents read that flat description to decide what to send, so an action missing from it is an invalid call waiting to happen. It now also says what `ack-reminder` takes instead, and why: a reminder is addressed by its own id because an item can carry several. **Fractional seconds fired early.** `time.Parse` accepts `09:00:00.900Z` and `Format(RFC3339)` drops the fraction, so it was stored as `09:00:00Z` and fired 900ms BEFORE the moment the caller named — silently, having rewritten their value on the way in. Seconds are genuinely the stored resolution (the column is compared as a string against a whole-second clock, and the tick runs every 30s), so the only question was which way to resolve it, and truncation resolved it the wrong way. `NormalizeInstant` now rounds UP: at most a second of lateness, in exchange for a guarantee that can be stated — a reminder never fires before the instant it was set for. Late is a reminder; early is a wrong answer. Whole seconds round-trip exactly, which is asserted, because an implementation that added a second unconditionally would otherwise pass. Three mutants for this round, three killed (round-up→truncate, round-up→unconditional-add, MaximumNArgs→ExactArgs). Thirty across the unit. Two fixes carry no dedicated test and it is worth being explicit rather than implying coverage: the `--format json` branch on `unremind` is a one-line output change with no server-free way to drive it, and the MCP `ref` description is prose the drift tests do not read — they assert an action is DOCUMENTED, not that a param's sentence lists it. * docs(reminders): the ack id is on the surface an agent polls, not only on the arm response CONVE-23 follow-through on the round-1 fix. Both agent-facing docs told a caller to acknowledge a reminder with the id "returned when you armed it" — true, and useless to the caller that matters: a poller reading next/ready never armed anything. The suggestion now carries reminder_id and `pad project next` prints the exact ack command, so the docs say that instead. The prose was written before the fix existed, which is exactly the case CONVE-23 is about: a change that makes an instruction stale without touching the file the instruction lives in. * test(reminders): bind the tick LOOP to the work, not just the pass (CONVE-19) Every other test in this file calls runReminderTick directly. That vouches for the component and says nothing about whether anything ever calls it — a tick that is never started is indistinguishable, from those tests, from one that is. It is the convention's exact case, and the failure I recorded on my own identity doc three times in one unit: I test the component and not the binding. Driven through the injectable tick channel so the assertion pins a SPECIFIC pass instead of racing a 30-second ticker, and polled to a bounded deadline so a loop that never runs FAILS rather than hanging the suite. Mutant: drop `s.runReminderTick()` from the select and this goes red while every direct-call test stays green. Killed. The idempotence leg exists because a second Start spawning a second loop would leave one running after Stop, making the BUG-842 drain invariant false for this sweeper specifically — the one property a copied lifecycle is most likely to get right by accident and least likely to be checked. The cmd/pad call site (cmd_server.go, alongside StartTokenReaper) stays verified by inspection: a source-scanning guard for it would be an instrument asserting facts about source, which is code with an adversary and not worth it for one line that sits in the middle of five identical neighbours. * fix(reminders): codex round 3 — a deferred reminder fired anyway, and the poll surface was unbounded **A re-arm mid-pass did not stop the fire.** The candidate scan selects an id; before the UPDATE runs, a `--rearm` can move that reminder into the future. Re-arm clears `fired_at`, so a predicate checking only `fired_at IS NULL` still matched — the pass fired a reminder the user had just deferred and emitted its event. The re-arm cannot undo that: it can clear the mark, but the event is already on the outbox and at-least-once means a consumer has seen it. The fire UPDATE now revalidates `remind_at <= nowTS` against the SAME nowTS the candidate scan used. Same-value deliberately: the arbiter and the scan must agree about when this pass is, or a reminder could pass one and fail the other for no reason but clock drift inside a single pass. **The poll surface was unbounded.** Every fired-and-unacknowledged reminder was loaded and turned into a suggestion prepended to a list that is otherwise capped at three, so a workspace with five hundred unacknowledged reminders returned five hundred suggestions — in the dashboard response, the hottest read in the product, growing until somebody acknowledged them. Two bounds, because they are two different guarantees: the query takes a window (default 50, oldest-fired first, so it holds what has waited longest), and the prepended suggestions are capped at 5 so `suggested_next` stays a recommendation rather than a second inbox. The full set stays addressable in `pending_reminders`. Truncation is REPORTED as a boolean, not a count. A count would have to be post-visibility-filter to be true for the caller reading it, and the store cannot compute that — the filter runs per item, above. "There are more than you can see here" is the strongest claim the data supports, so it is the one made. Four mutants; two killed outright, two survived and were run down under CONVE-28: - **Uncapped suggestions survived because the fixture had ONE reminder** — capped and uncapped are the same list at n=1. That is the SECOND time a single-item fixture hid a count-or-order property in this file. Fixture now arms eight; it also asserts all eight remain in `pending_reminders`, so the cap is pinned to the recommendation and not to the data. - **Removing the SQL LIMIT survived, correctly, and the test comment now says so.** The Go slice cap bounds the PAYLOAD; the SQL LIMIT bounds the DATABASE'S work. Only the first is observable at this level — with the LIMIT gone the response is still bounded, while the query silently goes back to materialising every pending row before discarding most of them. That is a memory and I/O property with no assertion available here, so it is stated as a coverage boundary rather than papered over with a green that would not have measured it. * docs(reminders): the fire predicate arbitrates against two actors, not one CONVE-23 inside the file the round-3 fix touched. The comment described the UPDATE as an arbiter for concurrent TICKS, which is what it was written for and is why I did not re-read it when asked whether a user edit could race the pass. It now says what it actually defends against, and names the general shape: an arbiter is only an arbiter with respect to the writers it can see. * fix(reminders): codex round 4 — the round-3 bound recreated the round-1 starvation Round 3 bounded the poll surface. Round 4 caught what that bound did: the query took the first N rows and the dashboard then discarded the ones it could not show — hidden items, unauthorised items, completed items — so N such rows hide a visible reminder behind them indefinitely, with no continuation to reach it. That is the SAME defect I had removed from the fire path one round earlier, reintroduced in the read path within the hour. The general form is worth stating because I clearly did not hold it: **a bounded window is only safe when the discarding happens BEFORE the bound.** Filtering above a limit is a starvation every time, and it does not matter what the filter is for. Two halves, because the two filters are not the same kind of thing: **Visibility is now scoped IN SQL**, using the same collection-id / item-id sets every other dashboard section gets through `allItems` — the same three-way shape as ItemListParams, where holding both collection grants and item grants is an OR. Invisible rows no longer occupy the window at all, which is strictly better than filtering them out afterwards and is what the sibling sections have always done. **Terminality is paged**, because SQL cannot evaluate it — a collection's schema defines which statuses are terminal. The collector refills from the next page when a page comes back short, bounded by a max scan so a workspace full of completed items cannot turn a dashboard read into a table scan. The bound is 10x the window: the common shape fills on the first page, and the pathological shape terminates in a fixed number of indexed reads. Stopping at the scan bound reports truncation, which is honest — there may be more, and we did not look. The empty-scope case is a THIRD state that reads like the second: nil CollectionIDs means unrestricted, a non-nil EMPTY slice means this caller sees no collections. Without an explicit guard they collapse, because the switch matches none of its cases at length zero and adds no clause at all — so "nothing visible" would return the whole workspace. Three mutants, one survived: the empty-scope guard, because no dashboard-level test produces that state (callers that would are refused earlier by workspace access). Faithful mutant, missing test — it now has a direct one, with a sanity leg so a build returning nothing cannot pass it by accident. A guard for a state nothing exercises is exactly the one that rots. * fix(reminders): codex round 5 — the MCP action I shipped did not work over stdio **P1: local stdio MCP `remind` was unusable.** cmdhelp derives positionals by regex from a command's `Use` string, and `<instant>` inside `remind <ref> --remind-at <instant>` matched — it became a second REQUIRED positional, so dispatch failed with `missing required argument "instant"`. The action was advertised on a transport where it could not run. **The MCP catalog's own tests did not catch it, and the reason is the finding.** That suite builds its cmdhelp document BY HAND: I wrote `Args: mkArgs("ref")` in it, so the fixture agreed with what I meant rather than with what the CLI says. Five parity and drift tests passed against a document I authored to match my own intention — the "a test that agrees with whatever the table says is not a test of the table" shape, which the canonical-events test warns about in its own comment two packages away. The new test reads the REAL command tree via cmdhelp.Build, which is the only thing in this repo that can disagree with me about what the CLI declares. **P2: `pad project ready` withheld the ack handle** that `next` prints. Showing a fired reminder on the surface an agent polls while withholding the id it needs to retire it means the same entry comes back on every poll, forever. **P2: suggestions asserted a collection they did not have.** The orphan branch admits ANY collection — its own comment claimed it gated on tasks "mirroring the active-plan branch", and that comment was simply false — while the output hardcoded `Collection: "tasks"` and the reason said "Open task". Pre-existing for high-priority items since BUG-1082; my overdue bypass widened it to any overdue item, which is how it surfaced. Fixed by carrying the item's REAL collection rather than by narrowing the branch: narrowing would silently drop the non-task items this has surfaced for a year, and the defect is the mislabelling, not the inclusion. The false comment is replaced with what the code actually does. The first version of that test used an overdue IDEA and SKIPPED — ideas use `new`, and the branch requires `open` or an active status, so it never became a candidate. A test that cannot fire is a failed reconstruction, not a pass; the fixture is now a bug-like collection whose vocabulary contains `open`, which is the population the defect can actually reach. Three mutants, three killed. Forty-one across the unit. * fix(reminders): codex round 6 — reminders fired from soft-deleted workspaces **P1, and the only defect in this unit whose consequence leaves the process.** Workspace soft-delete deliberately keeps items for the 30-day restore window, so the candidate query's filter on the ITEM's deleted_at found nothing wrong — and the tick kept firing, emitting outbound webhook events for a workspace whose owner had deleted it, possibly while deleting their account. Both queries now join workspaces and require `w.deleted_at IS NULL`. Nothing is destroyed: a restored workspace resumes firing, which the test asserts, because "stops firing" and "is destroyed" are very different answers to someone who restores a workspace and only one of them is right. That test first failed for the WRONG REASON and the fixture was at fault: it counted every outbox row in the workspace, and item creation writes its own, so the assertion was satisfiable by the fixture itself and discriminated nothing. Scoped to the reminder event type. **`Use: "remind <ref>"` declared a requirement the command contradicts.** cmdhelp derives the machine-readable arg spec from that string, and `--rearm` takes no ref — so the published contract said "required" for something optional. The requirement is CONDITIONAL, which cmdhelp cannot express, so the honest declaration is `[ref]` plus the explicit check that names both call shapes. The round-5 test grew a `required` column, which is what makes this observable at all: asserting only the arg NAMES would have passed. **The pad_item tool description omitted both new actions.** The params were declared and the actions dispatched, but the prose an agent reads to decide what a tool can do did not mention them — discoverable only by someone who already knew to look. It now describes both, including the two things an agent gets wrong: remind_at is an instant, and nothing but an explicit ack retires a fired reminder. Three mutants, three killed. Forty-four across the unit. * fix(reminders): codex round 7 — one predicate for the scan and the arbiter Third instance of one class, so this fixes the SHAPE rather than the instance. The class: the candidate scan filters on something the fire transaction does not revalidate, so a change committed between them fires a reminder that no longer qualifies. Round 3 was a re-armed instant. Round 1's soft-deleted item was the same thing caught from the other side. Round 7 is a workspace deleted between the scan and the fire — the round-6 fix added the condition to the SCAN only, and the arbiter went on not knowing about it. Fixing those one at a time is what let the third happen. `reminderFireable` is now a single string that both sites reference: the scan asks it and the fire UPDATE re-asks it, so they cannot disagree, and a fourth condition is one edit in one place rather than two edits someone has to remember are paired. Written as a correlated EXISTS on item_reminders.item_id rather than a JOIN precisely so the identical text is valid in both a SELECT and an UPDATE, and the scan drops its table alias so the two uses are the same characters. What deliberately stays outside it: `fired_at IS NULL` and `remind_at <= ?` live on the reminder row itself, are already spelled identically at both sites, and folding them in would need a parameter order the shared form cannot express. Said in the comment so the omission reads as a decision. Both directions are now tested at the arbiter — a workspace deleted mid-pass and an item deleted mid-pass — because the item case previously relied on the item load coming back nil, and someone simplifying the EXISTS down to the workspace check alone would otherwise still see green. Three mutants, three killed: the arbiter dropping the shared predicate, and the predicate dropping each of its two halves. Forty-seven across the unit. * fix(reminders): codex round 8 — workspace export silently dropped every reminder WorkspaceExport is a hand-maintained field list, so a new table joins it only if someone remembers. Reminders did not: a backup/restore, or a SQLite→Postgres migration via `pad db migrate-to-pg`, dropped every pending reminder with nothing in the destination to show anything had gone. The line that list has always drawn is item-scoped workspace CONTENT (comments, links, versions — exported) versus per-user state (stars, watches — not). A reminder has no user column and hangs off an item, which puts it on the exported side. Stating the rule rather than just adding the field, because the next person adding a table needs to know which side they are on. LIFECYCLE MARKS ARE CARRIED, not reset. A fired-and-unacknowledged reminder is still owed to whoever armed it, so it arrives pending; an armed one whose instant has passed fires once on the destination's first tick, which is what would have happened had the workspace never moved. Re-arming everything on import would invent a schedule the user did not set. NULL rather than empty string for the unset marks — the lifecycle is defined by NULL-ness, and "" would make a never-fired reminder read as fired at "". TestMigratedTablesCoversTheExport caught the second half, which I would have missed: `pad db migrate-to-pg`'s NUL preflight decides what to REFUSE on from MigratedTables, so a table the migration copies and the preflight does not know about is a gap in exactly the guard that exists to prevent one. Added there too, with the reason it can never actually fire — every column is machine-produced, so it is listed for coverage rather than expectation — and the "six tables" prose it falsified is now seven. Two mutants, two killed: export dropping the block, and import discarding the marks. Forty-nine across the unit. * test(reminders): state the fire-path invariant and pin it from the invariant The lead's read on why rounds 4 and 7 were the same class: the fire path had no stated invariant, so each fix defended an instance. This states it, and derives the pin from the paragraph rather than from the bug history. THE INVARIANT: the candidate scan is a hint and may be assumed to prove nothing. Every condition that made a row a candidate is re-asserted inside the transaction that marks it fired, in the same statement that does the marking, so checking and writing are one atomic act. Worded as "the scan proves nothing" rather than as a list on purpose — a list invites the next person to add a condition to the scan and stop, which is exactly what happened four times here. TestFirePathInvariant is the pin: one table, one row per scan-side condition, each invalidating that condition in the window between the scan and the fire and asserting the same three things — nothing fires, no event leaves, the reminder is not consumed. The earlier per-defect tests are folded in as rows; they said the same thing one instance at a time, which is how four of these shipped. Adding a fifth condition to the scan without a row here should feel like an omission. It carries a positive control, because four cases that all assert nothing happens would pass against a build that never fires at all. The matrix immediately falsified a claim in the paragraph I had just written. I wrote that the item load inside the transaction is "for the payload, not for the check"; removing the item half of reminderFireable alone changes no observable behaviour, because the load then returns nil and the deferred rollback undoes the write. Item liveness is defended TWICE and a single-mutant experiment cannot say which guard is carrying it — removing both is what kills the test. Both are kept, the predicate is named as primary (the row never matches, so no write happens at all), and the asymmetry is stated: workspace liveness has no second line, which is why dropping ITS half does fail the pin. Six mutants: five singles plus the pair. Five killed alone; the item single survives by design and is documented as such rather than left as an unexplained green. Fifty-five across the unit. * fix(reminders): codex round 9 — one legacy row could hide every reminder **P1: items.item_number is NULLABLE and I scanned it into an int.** Migration 006 added the column to existing rows, so a pre-numbering item still carries NULL — and scanning NULL into an int fails the Scan, which fails the QUERY, which degrades the whole pending-reminder section. One old row, and the feature is dark for everyone in that workspace. ListWatchesForUser, which this query was modelled on, uses sql.NullInt64 for exactly this column. I copied its shape and dropped the part that handles the column's actual nullability — the same way of being wrong as the round-5 cmdhelp fixture: borrowing a form without borrowing what it knows. The legacy row now carries no ref rather than a fabricated "PREFIX-0", which would name a different item. **P1: export shipped reminders that import could only discard.** The items section filters on deleted_at IS NULL, so a soft-deleted item is not in the bundle and its reminder can never be reunited with it. My comment claimed the item_links rationale — round-trip the raw graph so a restore reunites them — which is true for links and false here, because links keep soft-deleted endpoints in the bundle and items do not. A link is a row ABOUT two items; a reminder whose item is absent is a dangling schedule. **P2: import wrote remind_at raw.** Import is a writer, and a bundle is not necessarily one this server produced — hand-edited, or from another instance. A local offset or a bare date would land in the one column every comparison downstream treats as a UTC instant, firing early, late, or never. It now normalizes like every other door. An unparseable value is SKIPPED with a warning rather than failing the restore, matching the lenient import-side precedent already in this file, and the raw value's LENGTH is logged rather than its content. Three mutants, three killed; two needed rewriting because the single-line form did not compile — reverting the nullable scan also requires reverting the render, and dropping the normalization orphans a variable. PROCESS FAULT, recorded because it makes this round's findings weaker than they look: I edited the tree while this review was reading it — committed the invariant work and ran five mutation experiments, which write and restore source, over the same files. A review binds to the tree it read and I moved it underneath. Every finding above was re-verified against the current tree before being acted on, and the next round runs with no concurrent edits. * fix(reminders): codex round 10 — one orphaned item aborted a whole restore An ORPHANED item — one whose collection is missing from the bundle — still gets an itemMap entry. It has to: the entry is written before the skip because parent resolution inside the same loop reads the map for items it has not reached yet. So `itemMap[x] != ""` is satisfied by an id that names no row, and inserting a foreign key to it fails (SQLite enforces FKs here via the DSN's `_pragma=foreign_keys(on)`; Postgres always does). The pre-existing mapping is the sharp edge. The aggravating half was mine: this loop treated a failed reminder insert as FATAL, where item_links and item_versions both skip, so one orphaned item carrying a reminder rolled back an entire 900-item workspace restore. A reminder is the least critical thing in a bundle and it had the strictest failure handling in the file. Both halves fixed: the loop gates on items that actually landed, and a failed insert warns and skips like its siblings. TWO GUARDS THAT ONLY DIE TOGETHER, and this is measured rather than assumed. Reverting either alone leaves the test green — with the map gate restored the skip survives the FK failure, and with the fatal return restored the gate means the insert never fails. Removing both is what fails it. They are kept as a pair because they defend the same failure at different depths (prevent the bad write / survive a bad write arriving some other way), and the pair is recorded in the code so a future reader does not delete one as dead after watching its mutant survive. Second time this shape appeared today; the first was item liveness on the fire path. The bundle in the test is hand-built, because ExportWorkspace cannot produce an orphan — which is the reason it needed a test. That shape only arrives from a hand-edited or foreign bundle, and surviving those is what import is for. Three mutants: two singles that survive by design, plus the pair that kills. Sixty-one across the unit. * fix(reminders): codex round 11 — four contract slips, one of them another unit's **suggested_next returned up to eight entries against a cap of three.** Round 3 prepended reminders PAST the list's own cap, reasoning they should not compete for slots. Every consumer — the web dashboard, `pad project next`, `pad project ready` — is written for three. Worse, it silently falsified a decision recorded elsewhere: BootstrapDashboard deliberately has no suggested_next_overflow_count BECAUSE this list is capped at three upstream, and its comment names raising that cap as the moment to add one. My change made another unit's reasoning wrong in a file I never opened. The combined list is now trimmed back to three, reminders still leading — a reminder can push a task suggestion out, which is the right way round, and the full set stays addressable in pending_reminders. My first version of that trim used `limit`, which is REASSIGNED above to len(candidates) — so on a workspace whose only entries are reminders it would have truncated to zero, killing precisely the case the surface exists for. Caught by reading the surrounding lines before running anything; it has its own test now. **pending_reminders was uncapped in the bootstrap projection.** BootstrapDashboard embeds *DashboardResponse, so every new field joins the boot payload automatically — here, a window of up to 50, which is the budget PLAN-1410 spent a unit trimming. Capped at 5 with an overflow count, under its own constant rather than borrowing bootstrapAttentionCap: they answer different questions and a future change to one must not silently move the other. **Truncation was reported from the wrong question.** The collector used the store's `more` flag, which answers "is there another PAGE", not "did I read all of THIS one" — so a window filling part way through the final page reported that the caller had seen everything while unread rows sat behind the fill point. The paging bounds are now injectable so the case is testable at all: building it with a window of 50 needs ~75 rows in a specific pattern, with a window of 3 it is four. **Import accepted acked-without-fired**, which is not one of the lifecycle's three states. Such a row fires, is excluded from the pending surface because it is already acked, and can never be acknowledged because AckReminder requires acked_at IS NULL — an event emitted into permanent invisibility. The acknowledgement is dropped and the schedule kept, since an ack of something that never fired means nothing. Five mutants, five killed (one rewritten — removing the flag orphans a variable). Sixty-six across the unit. * fix(reminders): codex round 12 — a read is not a hold; scope the arm; ack from the ack Four P2s from round 12 (two independent runs, both landing on the same line of the fire path), each closed at the layer where it lives: - fireOneReminder pins the item and workspace rows FOR NO KEY UPDATE on Postgres before the arbiter UPDATE. reminderFireable re-asserted liveness at the predicate's instant and nothing held it to the commit instant; under READ COMMITTED an archival could commit in between and the event left the process about a deleted resource. Same idiom and same lock strength as CreateAttachmentForLiveItem; SQLite is excluded by its BEGIN IMMEDIATE, not skipped for convenience. Two PG-only pins verify "blocked" in pg_stat_activity, not by elapsed time; the pin-removed mutant fails both. - CreateReminder asserts "live item of THIS workspace" in the INSERT's own SELECT and returns ErrReminderItemGone otherwise. The table had an FK and no same-workspace constraint; a mismatched pair fed another workspace's title to this one's dashboard and webhooks. Handler maps it to 404. - AckReminder matches every fired row (COALESCE keeps the first ack, updated_at moves only when acked_at does), so a no-match means exactly "not fired at the instant of the ack". The handler no longer decides 409-vs-200 from the row it read before the UPDATE. - The invariant paragraph gains its missing sentence: "at that instant" means the commit instant, and the pin is what makes the predicate's instant and the commit instant the same one. Round-12 caveat carried: both runs were static reads (sandbox blocked Go's build cache), so "four" is a floor, not a measurement. Refs IDEA-2641 * fix(reminders): codex round 13 — a reminder's workspace must agree with its item's, at every read Every reader scoped by r.workspace_id and then joined the item without asserting the two agree. No door writes a disagreeing row today (CreateReminder derives the pair from the item; import maps within the workspace), and the table has nothing that forbids one — so a hand-edited bundle, a future move door, or a direct write would carry one workspace's item into another's dashboard, export, and webhooks. The identity goes into reminderFireable (scan + arbiter), the Postgres row pin, ListPendingReminders and the export query. One test writes the row raw — the only way one can exist — and asserts it is inert at each site; the predicate-removed mutant scans and fires it. Refs IDEA-2641 * fix(reminders): codex round 14 — the by-id and by-item reads assert the same identity as every other read GetReminder scoped by the row's own workspace_id and ListRemindersForItem by item_id alone, so a row whose two columns disagree — the class rounds 12 and 13 closed at the scan, the arbiter, the pin, the pending surface and the export — was still readable through the two reads that reach a single row. reminderOwned is that identity on its own, without the liveness half those two reads must not have (a fired reminder on an archived item is history worth showing). The write paths reach a row only through GetReminder, so scoping it scopes them; a row no door can write needs no door to delete it. ListRemindersForItem now takes the workspace its caller already resolved the item in. The raw-row test asserts both reads refuse the row from both sides; the reminderOwned-removed mutant surfaces it through GetReminder. Refs IDEA-2641 * fix(reminders): codex round 16 — an archived item's reminders are readable, and its verbs say "archived" The doors resolved the item live. Listing an archived item's reminders answered 409 from a GET, and ack/re-arm/delete answered a bare 404 for a reminder that exists on an item that exists — while the store, since round 14, deliberately keeps that history readable. The API already has a posture for archived items: GET reads them, mutations answer 409 "archived … restore it before editing" (writeItemResolveError). The list now follows handleGetItem; the lifecycle verbs load the item include-deleted, run the visibility check first, and then answer the same 409 every other item mutation does. One test walks archive → list 200 / ack 409 / arm 409 → restore → ack 200 on the same rows. Refs IDEA-2641 * fix(reminders): codex round 17 — one suggestion per item, the archived 409 by slug, and the door courtesy named Three findings on the server pass. (1) An item that was both a fired reminder and an ordinary candidate appeared in suggested_next twice; the ordinary entry is dropped, the reminder entry (which carries the ack id) stays, and two reminders on one item remain two entries. (2) Round 16's 409 for an archived item's reminder was written by re-resolving item.Ref, which is derived and empty for a legacy item with no item_number — so the class most likely to be legacy fell through to a bare 404. The slug is handed over instead. (3) The archived check in resolveReminderForWrite is check-then-write, and an archive landing in between lets the verb through: accepted and documented — it is the posture of every item mutation here (UpdateItem's UPDATE has no liveness clause), the outcome is benign, and putting liveness in AckReminder's WHERE would re-create the no-match ambiguity round 12 removed. Refs IDEA-2641 |
||
|
xarmian
|
c2bb1bad34 |
fix(mcp,server): close three codex round-11 findings, one of them my own bad refutation (BUG-2850)
Two P1 and one P2. The P2 is the important one, because I had already
dismissed it in round 10 and was wrong.
[P2 — CORRECTION] Artifact imports DO drop undeclared-field warnings,
and the case is reachable. Round 10 refuted this on the grounds that
artifact.Decode populates Fields only from FieldKeysForKind, so no
undeclared key could arrive. That check was real, and it was the WRONG
SIDE of the comparison: UndeclaredFieldKeys compares the field map
against the DESTINATION COLLECTION'S SCHEMA, not against the artifact
format's key list. The destination schema is editable, so a canonical
artifact key can be undeclared THERE while being perfectly legal in the
artifact.
Verified before reinstating, not argued: narrow the conventions
collection's schema to declare only `status`, import an ordinary
convention carrying trigger/scope/priority — the blob stores all three
and UndeclaredFieldKeys names all three. The merge is back, and the
comment now records the correction rather than the refutation, so the
next reader inherits the right reason.
What I got wrong is worth naming exactly: I verified a true fact and
then drew a conclusion one step wider than it supported, because I never
asked what the OTHER operand of the comparison could be. "No key outside
the artifact's list arrives" does not imply "no undeclared key arrives"
unless the destination declares every key on that list — an assumption I
never stated and never checked. The test I wrote at the time could not
pass, and I read that as confirming the refutation instead of as the
setup being wrong.
[P1] An empty hierarchy value inside `fields` was a silent no-op.
`fields:{"parent":""}` promotes onto the top-level `parent`, where both
doors treat empty as NOT PROVIDED — so the call reported success and
detached nothing. Refused now, pointing at clear_parent and the raw
`field:["parent="]` form. Not silently promoted to a clear: that decides
what this door MEANS, and v0.19 already made clear_parent canonical so
the empty string would not have to carry it.
[P1] The v0.16 compat ID params were not conflict-checked against
`fields`. Never schema-declared by design, they are invisible to
padItemPromotedFieldKeys and took the generic path, where the check only
consults the `field` array. `assigned_user_id:"A"` with
`fields:{"assigned_user_id":"B"}` made the doors disagree outright — the
remote mapper reads A, stdio emits only `--field assigned_user_id=B`,
because the top-level form has no CLI flag behind it. One call, two
different people assigned. Conflicting values refuse; equal ones
collapse to one form.
Also corrected, per CONVE-23: the round-10 test asserting that
`fields:{"parent":""}` conflicts with the plan alias now refuses for a
DIFFERENT reason and its stated rationale had become false. The case
moved to the new test with the right reason, and the field-array clear —
which really is an effective directive — stays where it was, with a
control leg proving the new refusal did not swallow it.
Mutation matrix, each mutant from a file backup:
revert the empty-hierarchy refusal -> only EmptyHierarchyValueInFieldsRefused fails (both subtests)
revert the compat-ID check -> only CompatIDConflictRefused fails (both subtests)
revert the import warning merge -> only the narrowed-schema import test fails
gofmt clean · go vet clean · go test ./... green (29 packages)
|
||
|
xarmian
|
0a71ad3aa9 |
fix(mcp): an empty parent param is not a hierarchy directive (BUG-2850)
Codex round 10: three P2, no P1. One fixed, one already filed, one
refuted and reverted.
[FIXED] An empty top-level `parent` was counted as an alias directive,
so `parent: ""` with `fields:{"plan":"X"}` refused a perfectly good
call. Every declared string param on this tool treats "" as NOT
PROVIDED — it is why promotedParamValue does, and why `assign: ""` is
deliberately inert — so a client that fills declared optional params
with their zero value rather than omitting them got a refusal for
asking one hierarchy question. My own round-9 snapshot carried this
forward from the out[]-based check it replaced.
Deliberately NOT applied to the other empty forms: `field:["parent="]`
and `fields:{"parent":""}` are the documented CLEAR signal
(BUG-2013 / BUG-2078), so they are semantically effective and still
conflict. One is a param left blank, the other is an instruction that
happens to look like one. Both directions are pinned, and the mutation
matrix drives both:
remove the empty-param exclusion -> only EmptyParentParamIsNotAnAliasConflict fails
extend it to the field-array clear -> only EmptyClearFormsStillConflict fails
[ALREADY FILED] The transport-dependent whitespace finding is BUG-2870,
ruled out of this PR's scope. Round 10 did add something the filing
missed and BUG-2870 now records it: the divergence covers VALUES too
(`--field "cost= 3"` stores the number 3 remotely and the string " 3"
over stdio), which is worse than the key half because both doors report
success and only the stored type differs.
[REFUTED, REVERTED] "Artifact imports discard createItemChecked's
undeclared-field warnings." True as a code reading — this handler builds
its own response shape and ignores item.Warnings — but the condition is
unreachable. artifact.Decode populates Fields exclusively from
FieldKeysForKind via a closed switch over a typed frontmatter struct, so
a key outside that per-kind list never enters the map. Verified both
ways before reverting, because the import door takes raw bytes and the
hand-written case is the one that mattered: Encode drops extra keys, and
Decode of a hand-written artifact carrying extra frontmatter keys drops
them too.
I had written the merge and a test for it before checking; the test
could not pass through the public door, which is what exposed the
finding rather than my fix. Reverted to a comment recording the
mechanism, so the next reader — or the next round — does not re-find it.
A branch nothing can enter is not defence in depth, it is a claim that
something is handled when it never happens.
gofmt clean · go vet clean · go test ./... green (29 packages)
|
||
|
xarmian
|
7f25283a41 |
test(server): pin the last three coercion call sites (BUG-2850)
Five of the eight `CoerceFields` sites had a test that goes red if that site alone is dropped. Move, bulk move and bulk update did not, so the PR's "typed on every door" claim rested on reading the code — CONVE-19 and the shape rounds 2-5 of this unit kept finding. Why the move pins are faithful rather than green-for-free: migrateValue already permits text->number (migrate.go:190), but it returns `value`, the ORIGINAL, not a parsed float. So a text field holding "42" reaches a number-typed destination as the STRING "42", and only CoerceFields at the move site turns it into a number before validation. Assertions are on the STORED NATIVE TYPE, re-read from the item rather than taken from the mutation's own response, so a handler that answered 200 and stored the string is still red. Bulk update merges request STRINGS (status, priority), so it is observable only where the schema declares one of those keys as a non-string type; a collection declaring `priority` as a number is unusual but legal and is the honest way to reach that site. Bulk ops answer 200 with per-item failures in the envelope, so the pins read the envelope too — a status-code-only assertion would pass on a dropped coercion. Per-site mutation matrix, run this turn against these tests, each mutant applied and reverted from a file backup (never `git checkout`, which would have taken the uncommitted tests with it): drop coercion at handlers_items.go:2314 -> only TestItemFieldsCoercedOnMove fails drop coercion at handlers_items_bulk.go:683 -> only TestItemFieldsCoercedOnBulkMove fails drop coercion at handlers_items_bulk.go:499 -> only TestItemFieldsCoercedOnBulkFieldUpdate fails Each mutant is the defect at the site the test targets — not a call-site patch next to a still-correct function (CONVE-28) — and each kills exactly one test, which is the per-site discrimination the PR claims. Files restored and verified identical after the matrix; suite green. |
||
|
xarmian
|
2cf9f0035a |
fix(mcp,server,cli): three codex round-2 findings (BUG-2850)
1. [P1] The structured-value refusal was in the wrong place and killed the fix. It went into BuildCLIArgs, which env.Dispatch runs for BOTH transports before handing off to whichever Dispatcher is configured — so it blocked the remote /mcp door too, and the native-field handling that is the whole point of this change was never reached. Moved into ExecDispatcher, which IS the stdio door. My own test could not see this: it called mapItemCreate directly, so it vouched for the mapper and not for the path that reaches it — CONVE-19's exact shape, in a unit where I had already written binding tests for the other half. The tests are now split along the two claims the first version conflated: nested values REACH the dispatcher (the remote door is unblocked), and refuseStructuredFieldsOverCLI refuses them at the CLI door naming the transport. 2. [P2] The CLI warning sat after the `--format json` early return, so the caller most likely to have sent a mistyped key — one piping stdout into a parser — was the one caller who never saw it. Moved above the return, and out of the `ref != ""` branch it was also trapped in. Still stderr. 3. [P2] A nil value in fields_patch DELETES the key (store/items.go), so reporting it as an undeclared field told the caller a field was stored that the same request removed. Filtered at the patch site, not inside UndeclaredFieldKeys, because nil means "store JSON null" on the full-fields path where reporting it is correct. Gates: gofmt clean, go vet clean, go test ./... 29 packages ok. Claude-Session: https://claude.ai/code/session_011Q4b1iHtJtSyMs7BA2ySxo |
||
|
xarmian
|
dc3fc2d50e |
feat(server,cli): name undeclared field keys on the write response (BUG-2850)
Undeclared keys are ACCEPTED — the census found 168 live values under 14 such
keys, and refusing them would break read-modify-write on items nobody edited
wrongly. But once stored, a typo and a deliberate extra field are
indistinguishable, so the write now says which keys it did not recognize.
- models.Item gains `Warnings *ItemWriteWarnings` with `undeclared_fields`,
omitempty and additive. NEW API SURFACE: item write responses carried no
warnings element before. Wrapping the response as {item, warnings} was the
alternative and would have broken every existing parser; a clean write is
byte-identical to before.
- items.UndeclaredFieldKeys consults models.IsReservedItemField rather than
re-listing the reserved set — that set exists so callers ask, and its doc
comment records what re-listing cost last time. So a write carrying
implementation_notes or github_pr reports nothing.
- fields_patch reports only the PATCHED keys. A stray key already on the item
is not something this write introduced, and naming it on every touch would
train the reader to ignore the field.
- The CLI prints one line to STDERR. Never stdout: `--format json` output is
piped into scripts, and a warning there would corrupt the JSON they parse.
- CLAUDE.md documents the element as new surface.
Controls: never attaching the warnings fails the pin; reverting the HTTP
mapper's native overlay fails the remote-door type test; dropping the
reserved-key exclusion fails its own test.
Two coverage gaps the controls FOUND rather than confirmed, both now closed:
the remote door's native overlay was covered by no MCP test at all (a revert
left the package green), and the reserved-key exclusion had no test either.
Both were written after the control survived, which is the only reason they
exist.
Gates: gofmt clean, go vet clean, go test ./... 29 packages ok.
Claude-Session: https://claude.ai/code/session_011Q4b1iHtJtSyMs7BA2ySxo
|
||
|
xarmian
|
b451fb50de |
test(server): bind the coercion to its call sites, and enforce copy/preflight agreement (BUG-2850)
CONVE-19: wiring is a claim. The previous commit threaded CoerceFields through eight validate sites; a test at the items package vouches for the function, not for any of those bindings. - Three HTTP-door tests (create, update, fields_patch) assert the stored NATIVE TYPE, not that the request returned 201 — a test that only checked the status passes on an implementation that stores the string, which is the shape the reporter described. - A text field holding "42" must stay a string in every one of them. Fixing this bug by coercing anything that parses would retype real data. - An un-coercible value must still be REFUSED with the validator's existing message, so coercion is not quietly widening what the server accepts. - TestCopyAndPreflightCoerceIdentically makes the cross-package invariant real. The preflight validates in internal/server and the copy in internal/store; both files carry a comment saying they must match, and a comment protects nobody. The assertion is agreement FIRST — whatever they do, they must do the same thing — and only then that both accept and the copy stores a number. Controls, each run against the mutated tree: - CoerceFields reduced to the identity function (the unfixed build) fails the two door tests and the items typing test. - Dropping the call at CREATE alone fails only the create test; dropping it at FIELDS_PATCH alone fails only that one. The bindings are individually covered, not covered in aggregate. - Coercing in the copy but not the preflight FAILS, and so does the reverse. Both drift directions are caught. BOUNDARY, stated rather than implied: four of the eight sites — move, bulk update, bulk move, and the migrated-schema paths they share — are wired identically but have no test that fails if that specific wiring is dropped. They are covered by the existing suites for their own behaviour, not for coercion. A follow-up should extend the door tests to them. Gates: gofmt clean, go vet clean, go test ./... 29 packages ok. Claude-Session: https://claude.ai/code/session_011Q4b1iHtJtSyMs7BA2ySxo |
||
|
xarmian
|
ae793e6fa6 |
fix(server,store,items): coerce field values to their declared types server-side (BUG-2850)
The write doors disagreed about what `key=value` means. The CLI has coerced by schema type since BUG-1125, and local stdio MCP inherits that by shelling out to the binary — but the remote /mcp transport builds its field map in ingestFieldKVP with `dst[key] = val`, so every value arrives as a string. validateFieldType then correctly refuses a string for a declared number or json field, and the net effect was that an MCP agent on that transport could not write those fields AT ALL: every attempt a 400, not a mis-typed value. Measured before writing anything (repro table on BUG-2850's trail): CLI and stdio MCP store 42 and an array; the HTTP door 400s on both; an UNDECLARED key is stored as a string on every door. items.CoerceFields(fields, schema) converts strings to the declared type — number via ParseFloat (NaN/±Inf refused, because json.Marshal cannot encode them and the ignored downstream error would silently drop the whole payload), json/multi_select via Unmarshal, checkbox via ParseBool — and is applied immediately before every Validate* call. Three deliberate non-behaviours, each with a test: - A value that will not parse is left as the string for the validator, so the existing "must be a number" error still fires. Coercion invents no error path, and cannot turn a currently-PASSING write into a failure. - Non-string values pass through untouched; an int stays an int. - Text-typed fields holding "42" stay strings. Coercing anything that parses would retype real data while fixing the bug. Not folded into ValidateFields, though that would be the single call site: a function named Validate that mutates its input is a trap, and two callers re-marshal the map they pass. THE POPULATION IS 8 CALL SITES, and finding them took two sweeps. The first was scoped to internal/server and found 7; the copy path validates in internal/store (items_cross_workspace_copy.go), which only a repo-wide sweep sees. The preflight and the store-side copy now carry cross-references to each other: the preflight exists to PREDICT the copy, they live in different packages, and that is exactly how they would drift unnoticed. The undeclared-key half of BUG-2850 is untouched and marked as a decision point in CoerceFields — refuse/warn/keep is with Dave. A test pins today's keep behaviour so the ruling lands as a deliberate change. The CLI's parseFieldFlag deliberately STAYS: it is why two of four doors are correct today, and removing it alongside its replacement would put all four at risk of one mistake. Retiring it is a follow-up. Claude-Session: https://claude.ai/code/session_011Q4b1iHtJtSyMs7BA2ySxo |
||
|
xarmian
|
178b6b5010 |
fix(server,store): two more from codex rounds 3 and 4 (BUG-2810)
**The import repair could silently change what gets imported.** It decodes into map[string]any, where a repeated object member keeps only the LAST value. The TYPED decode that runs next does not agree: encoding/json unmarshals members in order into the same struct field, so two `"workspace"` objects MERGE there and collapse here. A body with duplicate members would therefore import differently with --repair-nul than without, which is outside what a flag by that name may do. It now DECLINES such a body: returns it untouched, lets the gate judge it exactly as it would without the flag, and says why in the refusal — "the payload repeats the member X, and repairing it would change which value is imported". Detection is a token walk, because a decode is what loses the information: by the time there is a map the duplicate is gone. The detector's own test carries the false positive that matters — the same member name in SIBLING objects is not a duplicate, and a single shared set of names would decline every real export, since items all carry `id`, `title`, `slug`. Rewriting such a body faithfully wants a token-preserving pass, which is BUG-2812's token-walk and not a rider on this. A real export cannot contain duplicate members (json.Marshal does not emit them), so declining costs nothing an operator meets by accident. The tally now owns the repair — decodeJSONRepairingNUL takes it and calls Apply — so the count and the declined reason come back through one object instead of a return value a caller has to remember to record. That is the same mistake this branch already made once, when the JSON path dropped the count and the header reported 0 for an import that had rewritten a value. **A row the repair could not address was reported as a failure.** A NUL in a key column the list does not protect, on a row whose violation is elsewhere, makes the address unbindable: Layer A inspects every bound parameter, including a WHERE clause's, so the lookup is refused before SQLite is asked to find the row. It landed in Failed carrying "invalid text parameter: parameter 2" — the same information phrased as a fault in the repair rather than a property of the row. Now detected up front and reported as a skip with the reason, alongside the two skips that already existed. **One finding NOT fixed, deliberately, and recorded instead.** Round 3 raised that the scan misses a NUL in a value shadowed by a LITERAL duplicate key, so such a row passes the migrate-to-pg preflight and then fails during the copy — the exact failure the preflight replaces, surviving for one shape. That is textguard.KnownGaps: a blind spot every layer shares on purpose, which DOC-2823 forbids closing in one layer alone, because layers disagreeing about one value is the defect this cluster is made of. So it is named in ScanNUL's doc comment, in the preflight's, and in docs/backup.md for the operator, and TestScanNULInheritsTheRecordedKnownGaps pins the miss and FAILS when it stops being one — the notification that BUG-2812 has landed and those three prose sites need updating. The consequence is recorded on BUG-2812's trail. Round 2's single finding was refuted rather than fixed: it predicted TestRepairFlagReachesTheNestedAndObliqueForms would fail, on a mechanism that describes the raw-byte scanner this branch had already replaced. The test passes; the outer decode resolves the oblique spelling before the walk sees it. |
||
|
xarmian
|
49bd342e4c |
fix(store,server,cli): three defects from codex round 1 (BUG-2810)
**The import flag could not repair the column it exists for.** `--repair-nul` scanned the RAW body for a live escape, which is right for a value the gate reads at the top level and wrong for the one that actually matters. An item's `fields` blob travels through an export as a STRING: a NUL escape in the stored blob marshals into the body with a DOUBLED backslash, which a raw scan must leave alone because at that layer it is literal text — while the gate refuses it anyway, since it decodes the body and re-parses that string as the document it is. So the repair now walks the DECODED body with the same classing bodyDecodesNUL uses, one verb changed: where the gate asks textguard whether a value decodes to a NUL, this asks textguard to repair it. Two walks of one shape in one package is a real risk, and the mitigation is that they are measured against the same corpus in both directions rather than reviewed for similarity — TestBodyRepairMirrorsTheGateOverTheCorpus drives every case through the body shape and asserts refused-becomes-accepted and accepted-stays-byte-identical. Two consequences worth stating. The walk also reaches the OBLIQUE spelling — the backslash written as its own escape, so the six characters never appear in the raw bytes at all — which the scanner could not, so the test that pinned that limit is replaced by one asserting the capability. And re-encoding is now possible, so it is bounded: UseNumber, so an integer wider than float64 is not silently re-emitted in scientific notation; SetEscapeHTML(false); and a body with nothing to repair is returned byte-identical rather than round-tripped. The mutation that removes UseNumber turns 9007199254740993 into ...992, and a test says so. The header is now X-Pad-Repaired-NUL-Values, because at the decoded layer an escape is not a thing that exists any more and one nested document may have carried several. **The scan could not run on the databases it exists for.** Several protected tables carry a NULLABLE workspace_id — activities, api_tokens, mcp_audit_log — and the scan selected it into a plain *string, which fails with "converting NULL to string is unsupported" and takes the scan, the repair and the migrate-to-pg preflight down with it. Every column is now scanned as sql.NullString: SQLite also permits NULL in a declared PRIMARY KEY that is neither INTEGER PRIMARY KEY nor NOT NULL, which no other engine does, and a NULL key cannot address a row for an UPDATE — such rows are reported and skipped with the reason rather than handed a WHERE that matches nothing. Verified against the unfixed code: the scan returned `scan activities.actor row: sql: Scan error ... converting NULL to string`. It needed a VIOLATING row in such a table, which is why every fixture that planted its rows in `items` missed it. **--force by accident.** The repair skipped the running-server check whenever --from was given — and the most natural --from an operator types is the path `pad db scan-nul` just printed, which IS the live database. The check is now on the resolved path (Abs + EvalSymlinks, so a symlinked data directory or a relative path still matches), and a --from naming an unrelated backup stays unguarded, which is correct: nothing is writing it. The ordering moved with it. `store.New` runs pending migrations, so the refusal now happens BEFORE the database is opened; opening first and refusing second made the guard arrive after the thing it guards against. |
||
|
xarmian
|
63da2f4f5f |
feat(store,server,cli): count and repair the legacy NUL population (BUG-2810)
Layers A and B stop the value being written. Neither makes a row that
already carries one go away, and BUG-2810's filing is what that costs: an
affected workspace exports with a 200 and re-imports with a 400, so a
self-hoster restoring their own backup is blocked with no path forward in
the product, and `pad db migrate-to-pg` fails partway through the copy
against PostgreSQL's jsonb parser rather than up front.
This is DOC-2823's S3, on Dave's day-54 rulings: U+FFFD as the replacement,
repair standalone only with a migrate-to-pg preflight that refuses and
prints the command, `--repair-nul` on import shipping default-strict.
ONE REPAIR, beside the one predicate. textguard.Repair lives next to
ParameterRefused because four layers that agree about what is REFUSED and
disagree about what a repair PRODUCES is this bug family arriving one step
later. Its contract is a property over the same corpus, in both directions:
every refused value becomes one all four layers accept, and every accepted
value comes back IDENTICAL. The second half is the load-bearing one — a
repair that tidies values nobody complained about rewrites
`{"a":"x\\u0000y"}`, six literal characters after a doubled backslash, and
corrupts it.
The JSON arm is a string-literal SCANNER, not decode-walk-remarshal, which
is what the recon write-up proposed before it was written. Re-marshalling
changes four things nobody asked to change — object key order, insignificant
whitespace, integers wider than float64, HTML-ish characters — and silently
drops one of a document's LITERAL duplicate keys, which is a gap BUG-2812
owns and the last thing a repair should do. Scanning copies every byte it
does not deliberately rewrite, so an untouched document is byte-identical
without that having to be argued. A substring replace is not equivalent and
the test that proves it took a mutation to find: a doubled-backslash literal
ALONE never reaches the scanner, so the discriminating fixture is one
document carrying a live escape AND a literal.
THE COUNT IS COMPUTED IN GO. Measured on the read path in this worktree: a
row planted with `bad<NUL>name` reads back into a Go string with all 8 bytes
and the NUL intact, while `length(name)` in the same database answers 3.
TASK-2824 found that C-truncation and concluded no DB-side REPAIR could be
trusted; the same measurement on the read path says no DB-side COUNT can be
either. SQL narrows — `instr(col, char(0))`, plus the escape prefix on
JSON-classed columns, which is textguard's own pre-filter — and never
decides. The decision stays ParameterRefused with isJSON from the shared
86-column list, i.e. Layer B's classing.
Row addressing is read from the live schema rather than a hand-kept map:
39 tables carry protected columns, one (item_wiki_links) declares no primary
key and is addressed by rowid, two have composite keys, and five have a
single key that is not `id`. The repair checks RowsAffected because an
address that stopped selecting its row would otherwise commit an UPDATE that
touched nothing and report it as repaired — the one failure an operator
cannot see in the output.
`email_optouts(email)` is both a protected column and its own primary key.
Repairing it changes the row's identity and can collide with an existing
row, which in that table means somebody starts receiving mail again. It is
reported and skipped, with the reason.
The import flag is NOT an exemption from the gate. `--repair-nul` buys the
body one repair attempt and then runs the same `bodyDecodesNUL` on the
repaired bytes, which still decides — a decode path that skipped the check
is the door BUG-2803 spent thirty rounds closing, on the endpoint carrying
the largest attacker-controlled body in the product. Only the ESCAPE form is
repaired: a raw NUL byte makes the document invalid JSON, and widening what
parses is not this flag's job. Both doors are covered, JSON and tar.gz,
because giving them different answers is how one of them keeps being
forgotten.
Postgres is settled with evidence rather than sent up as a ruling: it cannot
hold either defect (22021, 22P05) and the four-way differential test already
pins that, so the scan reports not-applicable WITH the reason rather than
returning a zero a reader could mistake for a clean database.
Spellings settled here, per the dispatch: `pad db scan-nul` and
`pad db repair-nul` as siblings rather than `repair --nul`, matching
`migrate-to-pg`'s hyphenated compound — a repair verb that errors when given
no flag is a worse shape, and there is no second repair to share it with.
scan-nul IS the dry run, so repair-nul grows no --dry-run. It refuses while
the server is running unless --force, on the `pad db restore` precedent: the
report is a claim about a database, and one somebody else is concurrently
writing makes it a claim about a moment that has passed.
docs/backup.md's section on this is rewritten. It still said the rule lives
in the binary and not the database, which S2 made false, and it pointed at
this item for a preflight and a repair that now exist. Its import examples
also showed `pad workspace import < file`, which has never worked — the file
is an argument.
Closes BUG-2810.
|
||
|
xarmian
|
f54a0e41d4 |
docs(store,server): four comments and one log line that had stopped being true (BUG-2827)
Codex round 6. No logic finding; it confirmed the refusal ordering, the
split-budget claim and the first-tick scan as sound. Five statements in
the branch's own prose were untrue of the code as it stands:
- MaxOutboxPayloadBytes' comment still argued from 64 MiB ("two orders
below both ceilings") after the constant became 128 MiB, which is 4x
under the lowest ceiling, not two orders.
- maxOutboxClaimBytes' comment counted a scan-into-string-then-copy
transient that round 1 removed; the scan lands straight in []byte.
- maxOutboxClaimRows' comment used the item BODY mean (~2.4 KB) as the
payload mean; the measured payload mean is ~3.5 KB, so 5,000 rows is
~17 MiB, not ~12.
- emitBulkItemEventTx's early-out said the numbers the caller sees still
come from writeOutboxTx; when the early-out fires they come from the
projection, and the error says so in Measured.
- The drain's oversized-row log said "not claimed". OversizedPendingOutbox
filters only dispatched_at, so during a rolling upgrade a binary older
than the ceiling may be holding a claim on the row it names. The line
now states what the query establishes: this instance will not claim it.
The doc says why claimed_at is deliberately not a filter.
|
||
|
xarmian
|
0836808ca5 |
fix(store): drop a past-the-hop-bound event before judging its size, and correct three comments (BUG-2827)
Codex round 5. No production defect found; one ordering edge and three comments that had stopped being true. THE ORDERING. writeOutboxTx has two refusals that disagree about the mutation. The hop bound drops the event and lets the mutation stand, because only the cascade it would extend is illegitimate. The size cap fails the mutation, because there the mutation and the event are the same fact. An event that trips BOTH was judged for size first, so it failed a mutation over a row that was never going to be written. The hop drop now comes first. Unreachable today - nothing propagates a hop - which is exactly why the ordering is worth pinning before something does. TestAnOversizedEventPastTheHopBoundIsDroppedNotRefused fails with the two checks swapped back (run before the crash that interrupted this round, and again on this tree). THREE COMMENTS. The drain-limit constant said whole batches are claimed past it; the byte budget and row cap can now split one. The claim candidates' doc said every sibling; it is as many as the budget still allows. OversizedPendingOutbox's doc named the write cap while its query uses the claim ceiling, and said it ran every tick when the caller throttles it to once per five minutes. Gates: gofmt clean, golangci-lint 0 issues, full SQLite suite exit 0, full Postgres suite exit 0. |
||
|
xarmian
|
57caa7f92e |
docs,test(store): correct five comments and strengthen the shrink fixture (BUG-2827)
Codex round 4. Its one P1 does not hold, but the test it named as weak genuinely was, and five comments in this branch had drifted from the code they describe. THE P1, CHECKED RATHER THAN ARGUED. The claim measures size at candidate selection and never rechecks it, so a payload that GREW during a concurrent scrub could be claimed over the ceiling. The proposed growth path was Go's HTML escaping: json.Marshal writes < as its six-character unicode escape, where the source had one byte. Measured against Postgres 16: a one-key object whose value holds the four characters x<y>z&w, written with those characters LITERAL -> 16 bytes the same object written with < > and & as their six-character JSON unicode escapes instead -> 16 bytes Postgres parses the escapes and stores the characters, so the escaped and literal forms are the same size and the round trip cannot grow the row. On SQLite the payload is stored exactly as Go wrote it, so re-marshalling is idempotent. The rejection from round 3 stands, now on a measurement instead of an assertion about key removal. But the test defending it was weak, and codex was right about that: its fixture was one repeated ASCII letter, which cannot tell any of these encoder paths apart. It now carries <, >, & and non-ASCII, so it exercises the divergence rather than asserting past it. Mutation note worth keeping: a whitespace-padding mutant is caught on SQLite and NOT on Postgres, because jsonb discards insignificant whitespace — the mutant does not actually grow the stored row there. The faithful mutant adds a key, and that one dies on both. FIVE COMMENTS THAT SAID SOMETHING UNTRUE, all introduced by this branch: - OversizedOutboxPayloadError was documented as the write cap's error; three sites raise it, against two different limits. - "The two things Measured can name" listed three. - measuredStoredRow said "as the database stored it", but OctetLength measures what the driver hands back, which on Postgres is the ::text rendering rather than storage. - OversizedPendingOutbox described its threshold as the write cap while querying the claim ceiling. - ClaimPendingOutboxEvents still said batches are claimed whole, which the byte budget and row cap deliberately interrupt. Tests also now assert Measured at all three refusal sites — without it the field could be blank everywhere and every existing assertion still passes — and the claimability property test's refusal branch checks the row actually rolled back, so "refused" cannot be satisfied by a write that committed anyway. Gates: gofmt clean, golangci-lint 0 issues, full SQLite suite exit 0, full Postgres suite exit 0. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
213142c4b5 |
fix(store,server): honest refusal figures and a throttled oversized scan (BUG-2827)
Codex round 3. Two of three findings acted on, one rejected with an invariant test in place of the change it asked for. REFUSAL FIGURES SAID SOMETHING FALSE. The store refuses on two different measurements against two different limits — the member content before marshalling, against the write cap, and the row exactly as stored, against the claim ceiling — and both reported the number as "a %d-byte payload". For the first that is untrue: projectedBulkPayloadBytes is explicitly a lower bound, so the message named a size the payload did not have. A caller seeing two different numbers for one mutation had no way to reconcile them. OversizedOutboxPayloadError now carries what it measured, and both the error and the 413 say so. THE DIAGNOSTIC WAS THE MOST EXPENSIVE THING THE DRAIN DID, and it was most expensive when it found nothing. OversizedPendingOutbox has a non-sargable size predicate and no index to help it, so an empty result means evaluating octet_length over every pending row — on Postgres, detoasting and serializing each JSONB payload — and it ran every 5s tick. Now throttled to once every 5 minutes. Latency is the cheap thing to spend here: the rows it reports are permanently unclaimable and sit until the 7-day retention takes them, so a five-minute alarm delay changes no decision anyone makes about them. The first tick after a restart still scans, so an existing oversized row is reported promptly. REJECTED: that the claim needs to revalidate size, because a concurrent scrub could grow a payload between candidate selection and the claim UPDATE. It cannot. scrubOutboxRowTx is the only UPDATE of payload in the tree and it removes keys and re-marshals compactly, so a rewrite is strictly smaller — and shrinking is harmless, since a row judged claimable stays claimable. That is load-bearing for the claim needing no revalidation, so it is now stated at the function and pinned by TestScrubOnlyEverShrinksAPayload rather than left as an assumption for the next person adding a payload rewrite to break silently. The throttle test found its own gap on the way in. Written first against the helper, it stayed green when the call site was mutated to `if true` — a helper nothing calls is still correct in isolation. TestOutboxDrainTickConsultsTheThrottle covers the call site through the stamp the tick leaves behind. Also caught by the gate rather than by review: the field carrying the throttle clock landed on outboxDrainSettings as well as outboxDrainConfig, because the edit matched a line both structs have. Tests passed with both; lint named the dead one. Gates: gofmt clean, golangci-lint 0 issues, full SQLite suite exit 0, full Postgres suite exit 0. Mutants: growing the written payload kills TestScrubOnlyEverShrinksAPayload, removing the throttle call site kills TestOutboxDrainTickConsultsTheThrottle. One mutant discarded as unfaithful — corrupting the payload BEFORE the compare-and-swap is neutralised by the retry, which re-reads and redoes the work correctly, so it tests the retry rather than the invariant. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
5b12d5eeeb |
fix(store,server): bound the outbox's unbounded reads and writes (BUG-2827)
item.bulk_updated marshals every cascaded member body into ONE
event_outbox row, and nothing bounded either the row or the drain's
reading of it. The v1 doc called the size deliberately unbounded and
named a follow-up condition; measurement met it.
MEASURED FIRST, against an 8,434-item instance:
widest wiki-title cascade 23 members / 175.6 KiB of bodies
widest wiki-ref cascade 50 members / 365.5 KiB
worst option rename 4,429 members / 7.34 MiB
(tasks.status="done")
marshal ratio ~1.46x (item.updated payload mean 3,518 B
against an item body mean of 2,416 B)
So renaming one status option on that instance emits ~11 MiB in a single
row today, from a user clicking rename in the collection editor. That
kills the obvious design: a write cap tight enough to bound the drain's
memory refuses routine work, and one loose enough for routine work
bounds nothing. The two therefore became two numbers.
WRITE CAP (128 MiB) in writeOutboxTx, the single INSERT INTO
event_outbox in the tree, so all seven emit paths inherit it rather than
an enumerated site list. Refusing FAILS the mutation: the hop bound in
the same function drops the event and keeps the mutation, correctly,
because there the cascade is what is illegitimate, while here the
mutation and the event are the same fact. 128 rather than 64 MiB so it
clears MaxItemRenameCascadeBytes — a cascade squeaking under that 64 MiB
bound marshals to ~96 MiB, and a 64 MiB cap here would let the vaguer
refusal preempt rename_cascade_too_large on the very renames that bound
describes. Surfaces as 413 event_payload_too_large, following that
precedent rather than inventing a second spelling.
CLAIM BUDGET (64 MiB per pass), spent by the primary candidates AND by
batch siblings. Spending it on siblings knowingly relaxes "batches are
claimed whole": read literally that rule makes the budget bypassable by
construction, since one large batch is an unbounded read no row limit
touches, and groupOutboxDeliveries already defines the split. The first
candidate is always taken whatever it costs, so the bound meant to keep
the drain alive cannot starve a row instead.
CLAIM CEILING (2x the write cap), and it is NOT the write cap — a
distinction the Postgres leg had to teach. The cap measures the Go bytes
json.Marshal produced; the claim measures what the driver hands to Scan,
which on Postgres is the JSONB ::text rendering, one space inserted
after every colon and comma. A 40,000-byte payload reads back as 40,001.
Thresholded at the same number, a payload written at exactly the cap was
admitted by the guard and then permanently excluded by the claim:
delivered to nobody, reported as nothing, reaped seven days later.
TestARowWrittenAtTheCapIsStillClaimable fails on Postgres and passes on
SQLite against the 1x version.
Rows above the ceiling are excluded IN THE PREDICATE, not filtered in
Go — filtering in Go leaves them occupying candidate slots and starves
everything behind them, which is the same jam wearing different clothes.
They are logged every tick and left pending for the existing 7-day
undispatched retention, not stamped dispatched_at, which would record
that an event went out when it did not.
SCRUB. ScrubOutboxUserRefsTx collected every LIKE-matching payload at
once — the same unbounded read through a different door. Now batched by
a keyset cursor on id. READ FULLY THEN WRITE is preserved PER BATCH, and
the cursor keeps the per-row UPDATEs in ascending id order across
batches, so batching does not quietly reintroduce the BUG-2409 deadlock
it was written to avoid.
Prose sweep: MaxItemRenameCascadeBytes' comment asserted this vector had
no bound, and emitBulkItemEventTx's asserted the payload was unbounded
by decision. Both now say what is true, the second keeping its original
reasoning because it still explains what the bound does NOT do.
Mutation matrix, each mutant compiled and run unfiltered: removing the
row-cap check, > to >=, removing the budget break, removing
always-take-one, removing the SQL size exclusion, removing the sibling
budget, removing the scrub cursor advance, and 2x to 1x — all die, the
last on Postgres only. One survivor recorded as unfaithful rather than
as a gap: setting the scrub cursor to the first row read still advances
monotonically over an ascending id > cursor query, so termination holds
and only the pass count degrades.
Not addressed, named rather than left to be found: a workspace that
outgrows the write cap cannot shrink its own cascade, so the refusal
leaves no recourse. The answer is chunking one bulk event across rows
sharing a batch_id, which the drain's fold already supports.
Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm
|
||
|
xarmian
|
054b192e5d |
feat(store): make the NUL invariant a property of the DATABASE (DOC-2823 S2)
Closes BUG-2813's old-binary half. S1 put the rule in the binary, which cannot help in the window this bug is about: an older binary serving the same SQLite file has no guard, so a rollback, a staged rollout or a second instance writes rows the invariant forbids. A trigger is enforced by the FILE. 194 BEFORE INSERT/UPDATE triggers over 97 columns — TASK-2825's 86-column census plus the second ring Ruling 2 admitted wholesale. ONE LIST, THREE CONSUMERS, which is the shape TASK-2825 asked for. The migration is GENERATED from internal/store/nulcolumns.go, a guard test compares that list against the LIVE migrated schema, and a second test pins the generated SQL against the list so neither can drift from the other by hand. The census guard is a BASELINE rather than a demand that all 405 text columns be classified. The census classified the 86 that can carry caller text and left ids, timestamps and hashes unenumerated; requiring an entry for each would be 300 lines nobody reads. The 301 known-outside columns are recorded, and ANY change to that set — added, renamed, removed — fails and asks for a decision. The predicate is TASK-2824's, measured on the driver Pad actually embeds: instr for the raw byte (length() C-truncates and cannot be used), and json_tree for the decoded escape, in values AND keys, with a json_valid guard because json_tree raises on a non-document. The doubled-backslash case stays literal, which is the false positive this predicate family exists to avoid. Trigger aborts are classified into the SAME typed error Layer A produces, so a caller cannot tell which layer refused and the handler's existing 400 mapping covers both. That discharges Ruling 2's condition for admitting the second ring: a header-derived user agent or IP hitting a trigger must not surface as a 500 or a broken login. THE FOURTH LEG IS LIT. S1 built the differential harness with three legs and left this one dark; all 17 corpus cases now agree across the HTTP gate, Layer A at the driver, Layer B via a real UNGUARDED SQLite write, and native Postgres. Four independent enforcers, one corpus — the property DOC-2823 named as the actual deliverable. Two consumers needed changes, both legitimate. The FTS trigger census excludes the new triggers by prefix, so adding a protected column does not require editing that test. And a timeline test that INJECTS a NUL to build its fixture now brackets the injection by dropping and restoring the two triggers — it needs a row that violates the invariant, because what it tests is the handling of legacy data, and Layer B exists to stop such a row being written. SQLite only. Postgres refuses a NUL in text natively and an escape decoding to one in jsonb, so it already owns the rule there. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
cc8bf888d2 |
fix(store,server): close codex round 3 — inspect what is actually bound
Eight findings, four P1. The three parameter ones share a cause: the guard inspected one value and forwarded another. [P1] Value() was called for the CHECK and the original driver.Valuer was forwarded, so pgx called it again — a stateful valuer could show the guard clean text and the database a NUL. Resolution now happens once and the resolved value is written back into the argument. [P1] A typed-nil valuer, (*sql.NullString)(nil), was called directly and panicked. database/sql special-cases it as SQL NULL; so does the guard now. [P1] Only an exact `string` was recognised. pgx implements NamedValueChecker and binds *string, named string types and json.RawMessage unconverted, so each carried text the guard never saw. Resolution is by reflected KIND now, not by a list of types. [P1] Registration opened a probe connection for BOTH drivers, so creating a SQLite store attempted a live PostgreSQL connection against whatever host the environment happened to name — network access as a side effect of opening a local file — and swallowed the error, skipping the guarantees in exactly the case the check could not run. The assertions moved to connect and prepare time, where the object being asserted about is the one in hand. [P2] The statement-interface assertions were claimed in a COMMENT and existed nowhere. guardStmt forwards neither NamedValueChecker nor ColumnConverter, so a driver gaining one would silently lose its own argument conversion. Asserted now, and the parity test covers the statement level too. [P2] The cross-workspace copy gave a guard refusal its ambiguous "may or may not have landed" message. That refusal fires at parameter binding, before any statement executes — nothing landed, and telling the caller to reconcile invites exactly the manual work DR-13's wording exists to prevent. [P2] The Valuer integration test passed for the wrong reason by default: SQLite's driver lacks NamedValueChecker, so database/sql unwrapped the NullString before the guard ran. The three properties are unit-tested now, independent of which driver is present. And the NULL control asserted only that the write succeeded — it reads the row back and asserts SQL NULL, which an incorrect conversion to "" would otherwise have passed. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
74c786d590 |
fix(store,server): close codex round 2 — the wrapper must mirror its base
Seven findings; two were P1 and one of them was a real hole on Postgres only. [P1] driver.Valuer bypassed the guard. checkParams type-asserted `string`, and pgx implements NamedValueChecker — so it ACCEPTS a sql.NullString unchanged rather than letting database/sql's converter unwrap it, and the guard never saw the text. Measured before the fix: a NUL-bearing sql.NullString on Postgres passed Layer A entirely and was refused by the server as SQLSTATE 22021, i.e. a 500, while the identical value on SQLite got the typed 400 — the dialect split reappearing in the response shape. wiki_links.go binds sql.NullString today. [P1] driver.DriverContext was dropped, so sql.Open used a legacy connector that ignores the context and a cancelled request could leave a pgx dial running to its 60-second timeout. The rest share one cause, and it is the thing to remember: database/sql BRANCHES on whether an optional interface is present, so a wrapper advertising one the base lacks CHANGES behaviour rather than adding a no-op. Measured, the two drivers differ — pgx has no Validator, sqlite has no conn NamedValueChecker and no DriverContext — and the single wrapper type claimed all of them. So the wrapper now MIRRORS its base: four conn variants over the two interfaces that vary, a separate driver type for DriverContext, and the non-varying ones asserted at registration so a driver bump fails loudly instead of degrading. I made the same mistake inside the fix — implementing OpenConnector unconditionally, which broke every SQLite open — and then a third time, where guardConnector.Driver() returned the inner wrapper and a pgx pool reported no DriverContext. The third was caught by a new parity test on its first run, not by review: it asserts wrapped and base advertise EXACTLY the same interfaces. Also: both gap guardrails SKIPPED when their slice was empty, so deleting an entry made the suite green — the opposite of their purpose. They assert counts now. And the oracle test compared the two walkers without pinning any answer, which is how round 27 left both wrong about scalars; known answers are pinned against Postgres, and doing that caught me re-pinning one from a stale comment. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
ef792bdf35 |
fix(store,server,textguard): close codex round 1 on S1 — seven findings
All seven verified before acting; two were production regressions I had
introduced and one was a defect older than this branch.
[High] The wrapper HID the optional connection interfaces. Measured: the raw
modernc conn implements Pinger, SessionResetter and Validator; the wrapped one
implemented none, so Ping succeeded without pinging, pooled connections stopped
being reset, and dead connections stayed in the pool. Each is now forwarded,
falling back to exactly what database/sql does for a conn lacking it.
NamedValueChecker is forwarded too, or the wrapper silently narrowed the
argument types pgx accepts.
[High] Statement-level driver.ErrSkip was wrong. At the CONN level ErrSkip is
the documented fallback signal; at the STATEMENT level database/sql propagates
it, so a base statement without the context interface would have failed rather
than degraded. Both now fall back to the positional form, refusing a NAMED
argument rather than binding it by position.
[High] Scalar JSON documents bypassed the check — and the database settles it:
SELECT ('"a<escape>b"')::jsonb;
ERROR: unsupported Unicode escape sequence
A bare JSON string is a complete jsonb document. The object/array-only shape
test is right for the HTTP gate's "is this a nested document" question and
wrong for the store's "will a jsonb parser read this". Both now use a widened
form, so the gate refuses it at the door instead of leaving it to the store.
That widening broke the gate's independent oracle, which had been NARROWED in
BUG-2803 round 27 to make the two walkers agree. The disagreement was real; the
direction of the fix was not — nobody measured which one matched Postgres, and
it was the oracle. Restored, with the measurement in the comment. Two
implementations made to agree are not thereby correct.
[High] The 400 mapping was not the single funnel I claimed. createItemChecked,
bulk ops and the cross-workspace copy carry their own error envelopes by
design. One shared CLASSIFIER now serves all of them; the envelopes stay
different, the classification and wording do not. Copy keeps its
retry-discouraging message deliberately (PLAN-2357 DR-13).
[Medium] The over-refusal justification was FALSE as written. It claimed a
JSON-shaped text value carrying a live escape "is a value Postgres would refuse
the moment anything parsed it". Nothing parses a text column; Postgres stores
it fine. The trade is now stated honestly, and pinned: textguard.
StoreOverRefusals records the case, and a test fails when it is paid down.
[Medium] Duplicate JSON keys are a real under-refusal, inherited from the
shared predicate's map model. DOC-2823 requires Layer A NOT to fix it alone, so
it is recorded in textguard.KnownGaps with a test that fails when it CLOSES —
the signal that BUG-2812's token-walk landed.
[Medium] Test weaknesses, and the fixes caught a real one. The census now sees
ALTER TABLE ... ADD COLUMN. The store corpus leg now requires accepted cases to
SUCCEED, which immediately exposed a case passing for the wrong reason: a
non-JSON value written to the fields column failed on SQLite's own JSON parser,
never reaching the guard.
Full Go suite green on SQLite and Postgres 17; lint 0 issues.
Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm
|
||
|
xarmian
|
3baf984b41 |
feat(server,store): the differential test's three live legs + the 400 mapping
Completes S1's deliverable except Layer B, which stays dark until S2. THE THREE LEGS, all driving the SAME corpus: - HTTP gate (internal/server): each case enters as a request body, classed the way a request is — by KEY NAME, "fields" for JSON, "content" for text. - Layer A at the driver (internal/store): each case is a real write, classed by column. - Native Postgres (internal/store): each case is a real INSERT through the RAW pgx driver, no guard in the path, so what is measured is Postgres's own verdict rather than ours reflected back. All 15 cases agree across all three. That is the calibration result the design needed: the guard neither over-refuses (breaking writes Postgres accepts) nor under-refuses (leaving the dialect split BUG-2831 was about). The Postgres leg additionally asserts the SQLSTATE, because "some error occurred" would have been satisfied by a typo in the test's own SQL. THE 400 MAPPING is in writeInternalError — the single funnel every 500 already passes through, so one insertion covers every handler present and future. The item-title unit had to find the same error block three times in one function before a structural test caught the third; this avoids the enumeration entirely. 400 rather than 500 because the request is understood and will be refused identically on retry, and 400 specifically because it is what the gate answers for the SAME value refused at the door — two statuses for one rule would be the layers disagreeing in the response instead of the predicate. The residual is stated rather than hidden: a value can reach the store from something the SERVER composed (BUG-2814's re-emit population), and for those a 400 misattributes the fault. It is still better than 500, and if that case ever needs its own status it needs its own error type first. Mutation matrix extended: 3 more mutants, all killed — mapping removed (500 returns), mapping swallowing every error (the control leg), and the gate ceasing to delegate to the shared core, which is the layers-diverge scenario the corpus exists to catch. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
52c480c6d2 |
feat(store): enforce the NUL invariant at the driver (DOC-2823 S1 Layer A)
Closes BUG-2814 and the current-binary half of BUG-2813: the fixed binary can no longer WRITE a decoded NUL from any path, request or re-emit. The guard is a database/sql driver wrapper, not a seam in this package, and the two measurements that decided it are in nulguard.go's header. The design named the Queryer seam; Queryer is Query + QueryRow, its own doc says "the read-only subset", and no store write passes through it. Writes reach the driver by four receivers — db.Exec (139 sites), tx.Exec (99), stmt.Exec (2 prepared), and a passed executor — and a wrapper at the *sql.DB level cannot see the prepared ones at all. The deeper reason is this cluster's own lesson: a seam every write site must be EDITED to route through is an enumeration wearing a seam's costume, and nothing stops the next site taking the raw handle. The wrapper never reads SQL text, only bound parameters, so Sprintf-built statements are covered without being understood and a new call site is covered before it is written. It covers Query as well as Exec, which is not symmetry: three writes ride the Query path today (UPDATE ... RETURNING in password_resets and email_verification, INSERT ... RETURNING in yjs_updates). A single-line regex for that shape found ZERO; a structural scan found all three, each a multi-line raw-string literal. CLASSING: string parameters are checked, []byte is exempt. The invariant is about text and JSON columns, and in this store []byte binds BINARY — item_yjs_updates.update_data, the only BLOB/BYTEA column in either schema. The first version checked []byte and refused every Yjs op-log append; the existing collab suite caught it immediately. The exemption is pinned by a census test that fails when a new binary column appears, because the supporting sweep was source-level and this unit is a catalogue of source-level sweeps missing things. Mutation matrix: 13 mutants, 12 killed by named tests. Four survived first and each taught something rather than needing a weaker claim — the Query path, both prepared routes, and a Prepare branch that turned out to be DEAD for both drivers (database/sql routes to PrepareContext when the conn implements it), so that duplicate was removed rather than tested. Full Go suite green on SQLite and Postgres 17. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
d0644a60fc |
refactor(textguard): extract the one decoded-NUL predicate and its corpus (DOC-2823 S1)
Groundwork for S1's store-layer guard. The predicate lived in internal/server and could not be reached from internal/store, so a store guard would have had to reimplement it — two implementations of "does this decode to a NUL", which is the disagreement class this whole cluster exists because of. internal/textguard is a leaf package holding the predicate and the adversarial corpus. The HTTP gate now DELEGATES to it rather than carrying its own copy: stringIsJSONDocument and nestedDocumentDecodesNUL become one-line forwards, and valueDecodesNUL's unclassed arm is textguard's walk verbatim. Its existing 418-line corpus passes unchanged, which is the extraction's proof. What is deliberately NOT shared is named in the package doc: how a layer decides a value IS JSON. The gate derives that from request-body KEY NAMES; the store derives it from the COLUMN a parameter is bound to. Those cannot be merged, so every entry point takes the classification as an argument and the differential test pins that both derivations agree on one corpus. The corpus ships in the package rather than in a _test.go because three packages measure against it and three copies would drift. Fifteen cases, each carrying the reasoning or measurement that put it there — including the doubled-backslash false positive that made the raw-byte parity filter unsound, the raw-NUL-inside-a-JSON-value regression codex round 9 caught, and the nested-document asymmetry where the same escape is fatal one layer up. The escape text and the NUL character are constants, and a test asserts they did not decay into each other by comparing against a value it BUILDS rather than one it types. That mistake was made three times during BUG-2803 and twice more while writing this file. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm |
||
|
xarmian
|
0d0f1c9125 |
fix(items): require and bound item titles at every write door (BUG-2833, BUG-2831)
`PATCH {"title": ""}` was accepted and applied while `POST` refused the same input
with 400 "Title is required": the guard was an inline literal inside
handleCreateItem, so the sibling handler on the same field never had it. Item
titles were also unbounded, and the slug derives from the title with no
truncation, so the same input was accepted on SQLite and refused by Postgres at
the UNIQUE(workspace_id, slug) btree with an unmapped SQLSTATE 54000 — a latent
`pad db migrate` failure as well as a create-path one.
One models.NormalizeItemTitle / models.ValidateItemTitle pair now backs every
door, enforced authoritatively in store.CreateItem and store.UpdateItem so a
future door inherits the rule rather than having to repeat it. The handlers keep
a pre-lock copy that REFUSES ONLY: it may answer 400 early and must not alter
the input, because its view of the row predates the write lock.
- trim: whitespace-only titles are refused, widening the create door. Artifact
import already trimmed while create tested == "" exactly, and its comment
claimed to mirror the gate it was stricter than.
- bound: 255 runes, matching MaxDocumentTitleRunes but justified for items —
slugify emits only [a-z0-9-] at one byte per rune and truncates nothing, so
255 runes bounds the slug well under the btree index-tuple cap. That cap is
2704 bytes in practice, not the 8191 the filing quoted; both figures and the
readings behind them are in the constant's comment.
- non-retroactive: a title identical to the stored one is not a rename, is not
validated, and is dropped rather than re-applied — so rows predating the bound
stay editable and a no-op echo cannot move an item's slug.
- import coerces rather than refuses (empty -> "Untitled", over-long ->
truncated, both logged, colliding truncations resolved), matching
coerceJSONForImport's recorded disposition three lines away. Refusing would
break restoring archives of data this product already accepted.
- cross-workspace copy propagates a legacy source title, by ruling. It takes no
title from the caller, so it cannot mint one.
The guarantee that holds across every path is narrower than "every stored title
satisfies the bound", and the comments say so: no CALLER-SUPPLIED title is
stored without being validated.
Seven codex rounds, 23 findings, ending CLEAN. Two of the findings were defects
introduced by earlier fixes in this same unit — an empty-title hole opened
through the legacy-protection clause, and a handler-side decision that dropped a
concurrent rename — both recorded on BUG-2833's trail. 38 mutants; every
behavioural fix has a mutant that is the defect at its site, killed by a named
test.
Prose sweep per CONVE-23: three comments asserting item titles are unbounded,
and a cost model resting on a ~2 MiB single-request title, corrected in place —
the guards they document still hold, because the bound is non-retroactive and
the cascade charges STORED titles.
Filed rather than bundled: BUG-2836, BUG-2839, BUG-2840, BUG-2842.
Closes BUG-2833, BUG-2831.
Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm
|
||
|
xarmian
|
21b4d8d9c4 |
fix(store,links): remove both quadratics in the item rename cascade and bound it (BUG-2804) (#1224)
* test(store): measure the item-side rename cascade amplification (BUG-2804) Measurement only — no production code changes, no cap. The dispatch is measure-first, and BUG-2798's arithmetic is explicitly non-transferable to this path, so nothing here reuses its numbers. internal/store/items_rename_probe_test.go sweeps three axes with a k=0 negative control and a precondition that fails if the cascade would not be exercised (an empty cascade would otherwise report a flat line and read as "no amplification"). Instrument is MemStats.TotalAlloc, the same counter documents_rename_bounds_test.go uses, so the two paths compare directly. What it establishes, all measured on this machine: - 2.01x the body allocated per bracket, linear across four intervals (1 -> 8 -> 64 -> 512 -> 4096 brackets, agreeing to three significant figures). Bracket count is not independent of body size: the cheapest link is 5 bytes and nothing caps links per item, so the cost is O(C^2) in ONE linker's body, in ONE request, with no accumulation needed. Confirmed directly: doubling C multiplies allocation by 3.83 / 3.92 / 3.95, converging on 4. - 21.3x-22.2x the body per added linker at 8 brackets each, linear in k. - The outbox payload carries every rewritten body in a single row, at exactly 1.00x the body set, persisted. emitBulkItemEventTx documents this as deliberately unbounded in v1 and invites the measurement. Peak residency is reported as a coarse floor and carries no claim. The sampler sleeps between reads deliberately: ReadMemStats stops the world, and a spin loop turned a sub-second cascade into minutes of wall clock on this probe's first run. * test(store,links): decompose the per-bracket constant into its two sites (BUG-2804) Still measurement only. Checkpoint 2 reported a marginal cost of 2.01x the body per bracket without establishing what the 2 was made of. These probes account for it exactly, and the answer changes the fix shape. - internal/links/probe_alloc_test.go isolates RewriteBracketAt: 1.00x the body per call. So the rewriter is only HALF the per-bracket cost. - items_rename_probe_test.go's SELECT probe finds the other half, and it is not in the rewriter at all. cascadeTitleRename's SELECT joins item_wiki_links to items and projects s.content, returning one row per LINK, each carrying a full copy of the source body. The scan loop de-duplicates by source id only AFTER rows.Scan has allocated a fresh string per row. Measured dead flat at 1.00x the body per row against a SINGLE distinct source: 4096 rows on one 256 KiB item allocate 1.07 GB. 1.00 + 1.00 accounts for the measured 2.01. That makes TWO independent quadratics with different fixes. A single-pass rewriter would leave the SELECT's copy untouched and halve the cost rather than remove it. A third probe tests, and REFUTES, an earlier hypothesis that the second copy was the cascade's `rewritten != newContent` full-body comparison: a different-length new title measures 2.05x against the same-length 2.02x, no drop. The comparison allocates nothing, so it could never have appeared in an allocation counter. Kept as a negative result so the refuted reading is not re-derived. * refactor(links): single-pass RewriteBracketsAt, RewriteBracketAt delegates (BUG-2804 M2) The per-bracket decision moves into one unexported helper; RewriteBracketAt becomes the one-element case so the two cannot drift. Differential tests pin behaviour against the PRE-REFACTOR implementation, frozen verbatim in the test file as an oracle. Comparing RewriteBracketAt to RewriteBracketsAt-of-one would be vacuous now that the former is defined as the latter, so the oracle is what makes the test able to fail. * fix(store): bound the item rename cascade and remove both quadratics (BUG-2804) Three mechanisms, landing together because the first two are factors of one product — either alone still leaves O(C^2) while measuring "2x better". M1: cascadeTitleRename's SELECT no longer projects s.content. It returns one row per LINK, so projecting the body made the driver materialise a full copy per bracket, de-duplicated only after rows.Scan had allocated each one. Measured at 1.00x the body per row against a SINGLE source. Each source's content is now read once. M2: the rewrite is a single pass. links.RewriteBracketsAt splices every recorded bracket with one strings.Builder; RewriteBracketAt is reimplemented as the one-element case so the per-bracket decision has exactly one copy and the public contract is unchanged. M3: MaxItemRenameCascadeBytes (64 MiB) bounds the total linking-item content one rename may process, charged before each body is built so a refusal never allocates the thing it refuses. The number is derived from the live workspace's measured distribution, not inherited from documents.go, and the receipt is in the constant's doc comment. Measured, before -> after: doubling C 3.83/3.92/3.95x -> 2.04/2.21/2.06x (quadratic gone) per bracket 2.01x body -> 0.01x body per linker 21.3x body -> 7.3x body 4096-row SELECT 1,074,850,200 B -> 1,180,856 B (910x) store suite 180.9s -> 86.8s Equivalence is pinned against the PRE-REFACTOR implementation, frozen in the test file as an oracle, over a hand-enumerated behaviour corpus plus 20,000 randomised inputs; and the single pass is pinned against the descending fold the cascade used to perform, over 5,000 randomised inputs with shuffled, duplicated and corrupted offsets. The k-linear retention that remains is in the outbox member snapshots, filed separately as BUG-2827. * fix(store,server): codex R1 findings + a dialect-dependent test instrument (BUG-2804) P1 — ProjectRewrittenLen built a replacement string per bracket to measure it, reintroducing allocate-then-refuse one layer below the cap it feeds. The per-bracket helper now returns the SEGMENTS and never concatenates; projection does length arithmetic only, and the rewrite writes the parts straight into its builder, removing an allocation from the hot path too. P2 — the item rename cascade's refusal had no errors.As arm in handleUpdateItem, so a deliberate, permanent decline reached clients as a 500 implying a retry might help. Now 413 rename_cascade_too_large, composed from the error's typed fields so the internal call path is not published. Pinned by an exact-status test plus a counterfactual that a blanket-413 handler would fail. Checked the sibling write paths rather than assuming the reported one was the population: bulk update and version restore never set Title, so the cascade cannot fire there and this handler is the whole surface. P2 — `applied` counted brackets MATCHED, not brackets CHANGED, so a rewrite reproducing content byte-for-byte would still write the row, bump seq and emit events. Counting changes restores what the pre-fix whole-body comparison did. Cascade reachability is NOT established and the test says so. Separately, found by make test-pg: the refuse-before-building guard used a TotalAlloc ceiling, and the same refusal allocates ~69 MB on SQLite against ~242 MB on Postgres — the ceiling was measuring the driver. Replaced with an exact count of bodies built via a store test seam. The first version of that seam ALSO failed to discriminate: a mutation moving the cap check between the build and the seam call went undetected, so the build and the count are now one function with no third position for the defect to hide in. Gates on this tree: gofmt, go vet, go build ./... clean; go test ./... PASS on SQLite AND on Postgres (own container, private port 5446 — never the shared 5445, whose teardown would kill a sibling seat's container mid-run). * fix(links,server): codex R2 findings (BUG-2804) R2-1 — ProjectRewrittenLen advanced its cursor past a no-op bracket while RewriteBracketsAt did not, so the two disagreed about which overlapping rewrites the guard skips. On `[[A[[B]]]]` with a no-op at 0 and a change at 3, projection reported 10 bytes / 0 applied while the pass produced 13 / 1 — the cascade charged the read and never charged the rewrite it then performed, so the bound leaked on exactly the corrupt and duplicated offsets the defensive paths exist for. Projection now follows the pass, not the reverse: the pass's behaviour is pinned to the descending fold the cascade used to perform, so moving it would have changed cascade semantics under cover of a bug fix. The lockstep property test that pins this needed MIXED per-position target titles — the first version shared one title across a call and passed against the broken code, because the reproducing shape needs a no-op bracket overlapping a changing one, which a shared title cannot express. Cascade rows carry per-row target_title, so mixed is the realistic case. R2-2 — the 413 mapping covered one of THREE places handleUpdateItem reaches UpdateItemWithParentLink. The post-R1 population sweep asked whether other HANDLERS reach the store call and never asked whether this handler reaches it more than once. All three now share writeItemRenameCascadeTooLarge. The collab-edit half had a deeper cause than a missing arm. In applyContentViaCollabOnce's prune-and-direct-write fallback, a failing directWrite() had its error DISCARDED and the original collab error returned in its place, so the caller read a deterministic refusal as a recoverable routing problem and fell through to its own direct write — re-deriving the identical refusal from scratch. Measured 64 rewritten bodies built for one request against an expected 32. Deterministic failures now survive that branch, scoped to the three errors every call site already treats as final (open-children rejection, update conflict, cascade refusal). Everything else still returns the collab error, preserving the graceful-degradation contract the branch exists for. Status alone cannot discriminate the double-work fix — both behaviours end in 413, since the fall-through reaches the plain path's arm — so the store's build observer is now reachable from the server package via an exported test-support setter, and the test asserts the work count. Gates: gofmt, vet, go build ./... clean; go test ./... PASS on SQLite and Postgres; go test -race PASS on internal/links and internal/store with zero data races. Note -race on store runs 705s, past the 600s default timeout. * test(links): make the multi-bracket fold oracle independent (BUG-2804, codex R3) The descending-fold equivalence test built its oracle by folding RewriteBracketAt — which now delegates to RewriteBracketsAt, the function under test. The comparison was therefore circular: it could agree with itself while both diverged from the pre-refactor behaviour it exists to pin. This is the same vacuity I flagged and avoided for the ONE-element assertion, missed one case over in the multi-bracket one. The fold now uses rewriteBracketAtV0, the already-frozen pre-refactor primitive, which is the only genuinely independent oracle available. The test still passes, so the single pass does match true pre-refactor behaviour on every generated input including the overlapping ones — that agreement is now evidence rather than a tautology. Mutation-verified after the change: the fold test kills the dropped-sort and dropped-overlap-guard mutants, two of which it had previously left to other tests. Test-only. `git diff |
||
|
xarmian
|
ba1255881d |
fix(server): refuse a decoded NUL in a JSON request body (BUG-2803) (#1220)
* fix(server): refuse a decoded NUL in a JSON request body (BUG-2803)
The body half of BUG-2782 (path) and BUG-2784 (query). A caller-supplied
string reached a Postgres text parameter, Postgres refused it, and the
handler answered 500 — the honest answer is 400.
WHY THE TRANSPORT RULE CANNOT BE EXTENDED, which is the whole reason this
is a different fix rather than a wider middleware. ValidateQuery works
because a decoded query value is a substring of the raw query with ASCII
substitutions: the bad byte in the raw text IS the bad byte in the value.
That property fails for a JSON body — the reachable NUL arrives as the
six-character escape, all ordinary ASCII — so no request middleware can
find it without decoding the body, which is the handler's job.
MECHANISM, each premise measured against encoding/json rather than
reasoned about:
raw NUL inside a string -> decode ERR (invalid character in string literal)
raw NUL after the value -> decode ERR
the escape in a value -> decodes to a string CONTAINING a NUL
the escape in a KEY -> same
a DOUBLED backslash -> decodes to literal text, NO NUL
the uppercase spelling -> not a JSON escape at all
So the escape is the only vector and its substring is a sound FAST PATH
(absent -> no NUL possible), but not a sufficient test: a doubled
backslash carries the same six characters and decodes to text. In this
product that is not hypothetical — items and documents store markdown,
and a document about JSON escapes is an ordinary thing to write. The
exact step is json.Decoder.Token(), which returns DECODED strings, covers
object keys and arbitrary nesting (an item's fields blob), and needs no
knowledge of the destination type.
NOT REFLECTION over the decoded value, the other obvious design: it sees
[]byte fields AFTER base64 decoding, so a body carrying legitimate binary
({"b":"AQAC"} -> bytes 01 00 02) would be refused for a NUL that is not
text. A token walk sees the base64 characters. No request struct has such
a field today (searched: []byte with a json tag in internal/server and
internal/models, non-test — only models.YjsUpdate.UpdateData, which no
handler decodes from a body); the token walk is chosen so adding one
later cannot silently start rejecting valid requests.
BUFFERING IS NOT A COST. json.Decoder.Decode already holds the whole
top-level value in memory — refill accumulates into dec.buf and grows it
by doubling (encoding/json/stream.go) — so streaming never avoided the
copy. Measured on the 64 MiB workspace-import shape, total allocation:
stream+Decode 354.7 MiB, ReadAll+Unmarshal 256.5 MiB, ReadAll+Decode
512.5 MiB. Peak heap is order-dependent and does not discriminate; the
first run of that measurement showed a 0.77x peak win that vanished when
the legs were swapped, so only the allocation figure is claimed.
POPULATION, measured on Postgres 17 through the real router with a
control leg on every endpoint (92 mutating routes enumerated via
chi.Walk; 13 probed):
before: 12 of 13 DOOR (control 201 / NUL 500, SQLSTATE 22021)
after: 0 of 13 — every NUL leg 400, every control leg unchanged
Confirmed doors: workspace name, collection name, item title, item
content, item fields value, item title via PATCH, comment body, agent
role name, view name, document title, webhook secret, workspace import.
workspace-token name is UNMEASURED, not clean — its control leg 500s on
an unrelated FK in this fixture. The other 79 routes are unprobed, not
claimed clean; the completeness argument is structural instead, and
enforced by a test rather than asserted.
SECOND DEFECT, named rather than slipped in: the six handlers that
decoded straight off r.Body had no http.MaxBytesReader either — the cap
decodeJSON has always applied — so each was an unbounded body read.
Routing them through decodeJSON closes that too.
COMPATIBILITY: json.Unmarshal refuses trailing non-whitespace after the
JSON value where Decode ignored it. Deliberate, same direction as this
fix, and the only behaviour change beyond the refusal. Trailing
whitespace still passes. An EMPTY body still returns a wrapped io.EOF,
because handlers_playbooks.go reads errors.Is(err, io.EOF) as "no
arguments supplied" — caught by TestPlaybookRunAcceptsEmptyBody, which is
exactly the wiring a helper-level change is blind to.
No call site changed for the refusal itself: all 65 decodeJSON callers
already turn a decode error into a 400 carrying err.Error().
Release note: a NUL character in a JSON request body now returns 400
instead of 500 on Postgres deployments.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): follow the NUL refusal into JSON-encoded string fields (BUG-2803)
Codex round 1 on #1220: the check scanned ONE JSON layer, and several
fields cross the wire as JSON-ENCODED STRINGS rather than nested objects
— an item's fields, a collection's schema, a workspace's settings. The
OUTER decode of {"fields":"{...}"} yields the inner document as literal
text, in which the escape is still six ordinary characters and no NUL
exists, so the single-layer token walk passed it.
MEASURED on Postgres 17 with a control leg on each, after the
single-layer check was already in place:
item.fields as a JSON-encoded string 500 control 201
collection.schema as a string 500 control 201
workspace.settings as a string 500 control 201
The error is DIFFERENT from the rest of this family, which is why it is
worth reading rather than assuming:
insert collection: ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)
22P05, not the 22021 the path and query halves produce. The outer string
is pure ASCII so it never trips the text-encoding check; this is
Postgres's own JSON parser refusing the escape inside a document bound
for jsonb, which cannot represent a NUL. After this change all three
answer 400 with their control legs unchanged.
THE FIX: when a decoded string is itself a complete JSON object or array
— the class this API re-parses downstream — walk it too, to a depth
bound of 8. Recursion terminates on its own (each level is a strict
substring of the one above); the bound keeps a hostile body from buying
many full re-parses, and AT the bound the body is refused rather than
passed uninspected, since the escape is known to be present and the walk
has stopped looking.
WHAT THIS OVER-REFUSES, by design and pinned by a test: the rule is
structural, not destination-typed, so a plain TEXT field whose ENTIRE
value is a valid JSON document carrying the escape is refused too, even
though its column would have stored it. Prose ABOUT a JSON escape does
not parse as a bare document, so the case is narrow, and a value of that
shape breaks any consumer that parses it. The destination-typed
alternative — an allow-list of the fields that arrive JSON-encoded — is
exactly correct and goes stale in silence, which is the failure mode
ValidateQuery's comment rejects when it explains why per-site query
validators could not be written.
Tests: nested documents (fields/schema/settings/array/twice-encoded),
with controls for ordinary content, a doubled backslash INSIDE the
nested document, a string that starts like JSON but does not parse, and
prose that merely mentions the escape; the over-refusal pinned as a
decision rather than left as an accident; the depth bound; and a wiring
leg through the real router on SQLite, where the write would otherwise
SUCCEED so a green cannot be the database doing the work. Fixtures build
their JSON-encoded strings with encoding/json rather than hand-written
backslashes, since the escaping rules are the subject under test.
All four new tests fail with the recursion removed.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* test(server): build the NUL-bearing timeline fixture through the store (BUG-2803)
TestTimeline_NeverEmitsACursorItWouldRefuse built its fixture through the
API on a premise its own comment stated: "a structured id comes from the
item's fields blob, which nothing validates on write". BUG-2803 made that
false — decodeJSON now refuses a body whose strings decode to a NUL,
including one nested inside a JSON-encoded `fields` string — so the API
can no longer produce the row and the test 400'd on its fixture.
Repaired rather than deleted, because the DEFENCE it covers is still
live: rows in this shape can predate the rule, and the store has no such
check of its own, so a migration, an import or any future non-HTTP writer
can still produce one. The timeline must keep refusing to hand out a
cursor it would then reject.
The fixture now writes the blob directly, injecting the six-character
JSON escape rather than a raw NUL — the blob is JSON text and both
backends reject a raw NUL in it; the NUL comes into existence when Go
DECODES the blob, which is exactly how the timeline ends up with one
inside an entry id. The test is not vacuous under the change: it asserts
the NUL-bearing id took the positional fallback, so an injection that
failed to produce a NUL fails the test rather than passing quietly.
This is the CONVE-23 case — a change that falsifies existing prose owes a
sweep for that prose. The stale sentence was found by the test failing,
not by the sweep, which is the weaker of the two ways to find it.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* docs(server): correct the timeline comment BUG-2803 falsified (CONVE-23)
The entryID fallback's comment said note and decision ids "come from the
item's fields blob and nothing validates them on write". BUG-2803 made
that half false: the HTTP API now refuses a request body whose strings
decode to a NUL, including one nested inside a JSON-encoded `fields`
string. The sentence was true when written and nothing in this branch's
diff pointed at it.
The fallback still has to exist, and the corrected comment says why:
the STORE has no such check, so rows predating the rule — and anything
writing a blob by another path, a migration, an import, a future
non-HTTP writer — can still carry one.
SWEPT AND DELIBERATELY LEFT: two nearby comments
(handlers_timeline_id_collision_test.go, handlers_timeline_structured_test.go)
also say "nothing validates them on write". Both are about id FORMAT and
DUPLICATION — an imported artifact carrying a UUID-shaped id, a
hand-written blob repeating one — and this change validates neither. In
context those sentences remain true, so they are left alone rather than
edited into noise.
Sweep command: grep -rniE "nothing validates|not validated on write|no
validation on write|unvalidated" --include=*.go internal/ cmd/ — six
further hits, all about other subjects (github_pr raw writes, terminal
schema keys, push payload format, decodeJSON's size bound).
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): scope the nested-NUL walk to JSON-encoded fields (BUG-2803)
Codex round 2 on #1220. The nesting check from the previous commit
recursed into ANY string that parsed as a JSON document, on the argument
that a structural test beats a destination-typed one. That argument was
wrong in a way I had written down as an accepted trade and should have
weighed as a defect: a plain-text `content` value holding a JSON snippet
that merely MENTIONS the escape was accepted before this branch, is
stored in a text column that has no problem with it, and was newly
refused — including on RE-IMPORT of an export carrying it.
Refusing input the server itself produced is a worse failure than the
door the unscoped recursion was closing. Measured before the fix: a
workspace whose item content held such a snippet exported 200 and
re-imported 400.
The walk now descends only under keys whose STRING value is a JSON
document something downstream re-parses: config, events, fields,
metadata, phase_data, plan_overrides, schema, settings, tags, traits.
WHY A LIST IS SAFE HERE, when ValidateQuery's comment rejects exactly
this shape for query parameters: there the set of names is unbounded by
design (parseItemListParams turns any unrecognised parameter into a field
filter), so no list could be complete. Here the set is a closed property
of the wire model — a field is JSON-encoded because a Go struct declares
it as a string holding JSON — and
TestJSONEncodedFieldKeysCoversTheModels derives it from internal/models
and fails when a new one appears. The list cannot go stale in silence.
Over-inclusion is the safe direction and the list takes it: a listed key
that is not really JSON-encoded costs one parse attempt and can only
refuse a complete JSON document carrying the escape, while a missing key
reopens a door. `traits` is listed for that reason — it carries JSON but
its declaration has no comment saying so, which is exactly how the
derivation test would have missed it, so the test asserts coverage in one
direction only and the list is allowed to be a superset.
The walk also changed shape: decoding into `any` and walking the value,
rather than a token stream, because key context is needed to know which
subtree is JSON-encoded. The []byte reasoning is unchanged and still
holds — decoding into `any` never produces a []byte, so a base64 field is
seen as its ASCII text rather than as decoded bytes that might contain a
legitimate 0x00.
Tests: text fields carrying a JSON document are ACCEPTED (five keys),
with a leg proving the same document under a JSON-encoded key is still
refused, so the pair differs only in the key; the derivation test; and
the depth-bound fixture now nests under a JSON-encoded key at every
level, since nesting under an ordinary key would never start the
recursion and would have passed for the wrong reason.
STILL OPEN, and the lead holds it: a LEGACY row whose stored fields blob
already carries the escape still exports 200 and re-imports 400. That is
data this fix cannot make importable without weakening the write-side
refusal, and the disposition (repair sweep, flagged import, or documented
acceptance) is a product ruling. Recorded on the item.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): close the three body doors codex round 3 found (BUG-2803)
All three verified before fixing, none taken on the reviewer's word.
1. BUNDLE IMPORT BYPASSED THE REFUSAL (P1). handlers_import_bundle.go
parses pad-export.json itself rather than through decodeJSON, so the
SAME workspace import — reached with Content-Type application/gzip
instead of application/json — walked straight past the NUL check into
Postgres. The bundle's export blob is now checked with bodyDecodesNUL
before ImportWorkspace, answering the same 400. Test drives a real
tar.gz through the router with a clean-bundle control leg, because this
path answers 400 for a dozen unrelated reasons (bad gzip, out-of-order
tar, duplicate entries) and a bare 400 would prove nothing.
2. ONE CALLER SWALLOWED THE NEW ERROR (P2). handlers_admin.go's
test-email endpoint read `if err := decodeJSON(...); err != nil ||
input.To == ""` and fell back to the admin's own address, so a body
carrying a NUL answered 200. An ABSENT body legitimately means "send it
to me"; a body that is present and REFUSED is a different thing, and
collapsing the two turns a validation error into a success. The two
cases are now separated on errors.Is(err, io.EOF).
3. THE COMPLETENESS TEST COULD NOT SEE PAST TWO CALL SHAPES (P2). It
scanned for json.NewDecoder(r.Body) and io.ReadAll(r.Body), so it was
blind to io.ReadAll(io.LimitReader(r.Body, n)) — a shape ALREADY in the
package — and to any alias or helper. A completeness test that misses a
live example is worse than none, because it reads as coverage. It now
scans for the thing that cannot be spelled around, a reference to the
request body at all, and requires every FILE touching one to be
accounted for with a written reason. Both directions are asserted: an
unaccounted file fails because a door may have opened, and an accounted
file that no longer touches a body ALSO fails, so the list cannot rot
into stale excuses that quietly cover a future reader. Verified with a
positive control (an added body reference in an unlisted file fails) and
a negative one (a stale entry fails).
FOUND BY THAT WIDENED SWEEP, and fixed here rather than filed: the raw
artifact import (POST /workspaces/{ws}/import-artifact) takes TEXT, not
JSON, so it never went through decodeJSON and inherited neither the NUL
refusal nor the path/query rule — a body is neither. A raw NUL or
invalid UTF-8 reached the store and Postgres answered 22021, which the
handler turned into a 500 for what is a client error. It now applies
bindableText, the same predicate ValidatePath and ValidateQuery use, and
answers 400 invalid_body. Note the shape difference from the JSON half:
there the ESCAPE is the vector because a decoder rejects a raw NUL;
here the RAW BYTE is, because nothing is in the way.
Each fix has a mutation run against it: disabling the bundle guard fails
the bundle test, disabling the artifact guard fails the artifact test,
and both controls still pass.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): the escape gate was unsound, and YAML has its own (BUG-2803)
Codex round 4, two P1s, both reproduced before fixing.
1. THE FAST PATH LET A REAL NUL THROUGH. bodyDecodesNUL gated on "does
the raw body contain the six-character escape". That is unsound: the
BACKSLASH itself can be written as an escape, so a body carrying
\u0000 contains no literal six-character sequence anywhere in its
raw bytes, while the OUTER decode manufactures one inside the string —
and if that string is re-parsed as a JSON document (jsonEncodedFieldKeys)
the second parse turns it into a real NUL.
Measured through the real router before the fix: the oblique spelling
answered 201 where the direct one answered 400.
The mistake was applying a fact about how a NUL is spelled INSIDE a
decoded string to the RAW BYTES, where the backslash can itself be an
escape. That is the same layer-confusion this whole bug is made of, for
the third round running.
The gate is now a BACKSLASH. Every JSON escape mechanism requires one, so
a body with no backslash has decoded strings byte-identical to its raw
bytes, and a raw NUL cannot survive the decoder — no backslash therefore
means no NUL, at any depth, however spelled. Bodies WITH one pay for an
exact answer, a larger set than before (any nested JSON carries a
backslash-quote), which is the cost of being correct. The same
correction applies to the per-string pre-filter one level down.
2. YAML HAS ITS OWN ESCAPE VOCABULARY. The raw bindableText check added
last commit passes a double-quoted scalar `title: "a\0b"` — no NUL in
the request bytes — and the YAML decode manufactures one. Measured
before the fix: that artifact imported 201 with a NUL in the item title.
The decoded artifact is now checked too: title, body, and every
frontmatter field value, walked because a playbook's `arguments` is a
nested structure rather than a scalar. Keys are checked as well as
values, on the same precautionary grounds ValidateQuery states for
query parameter names.
Same shape as the JSON half in both cases: a value that is harmless
until a SECOND parse, checked at the layer that can see it.
Tests: the oblique spelling joins the nested-document table, and the
YAML escape joins the artifact table. Each is mutation-verified —
reverting the gate to the substring fails the oblique case only, and
disabling the post-decode artifact check fails the YAML case only, with
the raw-byte cases still killed by the raw check. That per-leg
discrimination is the point: it shows each check earns its own keep
rather than being covered by its neighbour.
Prose corrected where this falsified it: jsonNULEscape's "it is the ONLY
spelling" is true of the escape and was being used to justify a filter on
the raw bytes, which is a different claim. Both now say so explicitly.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): multipart text fields and the bundle manifest (BUG-2803)
Codex round 4's two P2s. Both are the same shape as the rest: a
caller-supplied string reaching a text comparison through a door the
earlier fixes did not cover.
1. MULTIPART TEXT FIELDS. The multipart body is deliberately exempt from
the JSON rule — its payload is binary blob content and must not be
scanned for text validity — but its TEXT fields are a different thing.
`item_id` goes to ResolveItem and into a database comparison exactly as
the query-string channel does, and that channel has been validated at
the transport since BUG-2784; the form channel was not. multipartValues
now drops values that are not bindable text, which makes an unusable
value indistinguishable from an absent one — the disposition
resolveUploadItemID already applies to empty values.
The uploaded FILENAME gets the same predicate, with a fallback to a
generic name rather than a refusal: the bytes are fine, only the label
is unusable.
A NEGATIVE RESULT worth recording, because it changed the test: a RAW
NUL in the multipart header is NOT the vector. Go's multipart reader
refuses it as a malformed MIME header line before any handler sees it
(measured: 400, "malformed MIME header line"). The reachable spelling is
the RFC 5987 encoded form, filename*=UTF-8''sh%00ot.png, which the
header parser accepts and percent-decodes afterwards. The first version
of this test used the raw form and was testing a vector that does not
exist.
2. THE BUNDLE ATTACHMENT MANIFEST. A second JSON document inside the
tar.gz, parsed directly like pad-export.json was, so it needed the same
check. Without it a NUL in a manifest string reached
rehydrateAttachment, whose failure is logged and SKIPPED — so the import
reported success while silently dropping the attachment. The
skip-on-failure behaviour is pre-existing and deliberate (a partial
restore beats none); refusing the bad INPUT is what stops it being
reached this way. Left as it is, and named rather than quietly changed.
A VACUOUS ASSERTION THE MUTATION CAUGHT, recorded because the test would
otherwise have shipped as coverage: the filename leg first asserted
`!strings.ContainsRune(body, 0)` on the RESPONSE, which is JSON — a NUL
in the filename comes back as the six-character escape, not as a 0x00,
so the check passed whether or not the fix was present. It did pass with
the fallback disabled. Now it decodes the response and asserts the
replacement name. The item_id leg had the mirror-image weakness: it
asserted "not a 500", which is the Postgres-only symptom, so on SQLite it
would have passed either way; it now asserts the request behaves exactly
like the no-value control.
Every fix in this commit has a mutation against it, and each kills only
its own leg.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* refactor(server): drop the now-unused escape constant (BUG-2803)
The gate became a backslash check, which was the last production use of
jsonNULEscape; golangci-lint's unused check failed on the next run. Its
documentation was load-bearing, so the explanation moved into
bodyDecodesNUL's comment rather than being deleted with the variable —
including the distinction that made the old gate wrong (the escape has
one spelling INSIDE a decoded string, which is not a claim about the raw
bytes).
Caught by re-running lint on the tip after the previous commit rather
than trusting the run from the tip before it.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): rune-safe truncation and User-Agent sanitising (BUG-2803)
Codex round 5 was asked for the POPULATION rather than a confirmation —
"enumerate every remaining way a caller-supplied string can reach a
database text or jsonb parameter without passing a validity check" — and
returned three residual classes with their sinks. Two are fixed here;
the third is filed, because measuring it needs a fixture this unit
should not grow.
1. TRUNCATION CAN UNDO THE VALIDATION. Four sites cut a caller string
with a plain byte slice (name[:120], input.Name[:200]). If the boundary
lands inside a multi-byte rune the result ends in a partial sequence and
is no longer valid UTF-8 — so a value that PASSED the body check a few
frames earlier arrives at the store unbindable, and Postgres answers
22021 for a request the server already accepted.
This is the interesting one, because no input-side round could have
found it: the defect is downstream of validation, and it is invisible
with ASCII fixtures, which is what every test in that area used.
truncateBindableText walks back off continuation bytes and drops the
straddling rune. Tested with 2-, 3- and 4-byte runes so an off-by-one
walk-back cannot pass them all, and with a counterfactual leg asserting
the naive slice really does produce unbindable output for the same
input — without it the cases would pass against an implementation that
did nothing.
2. USER-AGENT REACHES TEXT COLUMNS. It lands in activities.user_agent
(three document paths, the connected-apps revoke) and
sessions.user_agent (three login paths), and no rule here sees a header.
The disposition is SANITISE, not refuse, and that is deliberate: a
header is metadata this server chose to record, not something the caller
asked for, so a malformed one must not turn an otherwise fine request
into a 400. The two sites that HASH the header are left alone — sha256
over arbitrary bytes is well defined, and changing what is hashed would
invalidate every stored UAHash.
The filing's own earlier probe had recorded User-Agent as NOT
reproducing on the item-create path. That was true and did not
generalise; these are different sinks.
3. NOT FIXED, FILED: the OAuth form-encoded bodies
(/oauth/token, /oauth/authorize/decide, /oauth/revoke,
/oauth/introspect) parse url-encoded form data outside the shared body
validator, with connection_name reaching oauth_connections.name and
client_id reaching the oauth_clients.id lookup. This was the ORIGINAL
subject of BUG-2803 before the filing was re-scoped, and it was recorded
then as unreachable without a fosite-backed fixture. That is still true,
and round 5's sink list is far more than the filing had. Filed rather
than guessed at.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): narrow the gate, stop refusing natural-shape fields (BUG-2803)
Codex round 6 plus one measurement of my own. Three changes, one of them
a revert of something I got wrong in the previous commit.
1. THE GATE COST TOO MUCH, so it is narrower and still sound. The
previous commit gated the walk on "does the raw body contain a
backslash", which is correct but catches every body carrying nested JSON
(each `\"` is a backslash). Measured on a ~377 KB import-shaped body:
60106 allocs/op with that gate versus 30073 with the walk disabled — the
walk was running on ordinary traffic.
The gate is now the four bytes that begin any \u escape for a character
below U+0100. The argument: to manufacture the six-character NUL escape
inside a decoded string, each of its characters arrives either literally
from the raw bytes — in which case the raw contains the escape, which
begins with that prefix — or from a \u escape of its own, and the three
characters involved (backslash U+005C, 'u' U+0075, '0' U+0030) all sit
below U+0100, so those escapes begin with it too. Back to 30073
allocs/op, identical to the walk-disabled build.
That argument is the same KIND of reasoning that was wrong two rounds
ago, so it does not stand on its own: a differential test runs the gated
function against an UNGATED walk over a corpus built to attack it —
oblique backslash, upper-case hex, an escaped 'u', an escaped '0', a
doubled backslash — and fails on any disagreement. It also asserts the
corpus contains both answers, since agreement over a one-sided corpus
would be vacuous. Reverting the gate to the old substring fails it.
2. THE CHECK REFUSED THE NATURAL SHAPE OF ITS OWN FIELDS. `tags` and
`fields` accept both a JSON-encoded STRING and their natural array/object
form, and the walk propagated "this subtree is JSON-encoded" into
containers — so a free-form tag whose whole value happened to be a JSON
document was refused, though nothing re-parses it. Measured: refused
before, accepted now, while the JSON-encoded spelling of the same field
is still refused. The flag now marks only a direct STRING child of a
listed key.
3. REVERTED: I wired the three LOGIN paths to the User-Agent sanitiser
last commit, before reading store.CreateSession. It HASHES the header
and stores no text — the round-5 enumeration named "sessions.user_agent"
and I took the name for a column. The change would have been actively
harmful: login would store sha256(sanitised) while middleware_auth still
compares sha256(RAW), so every session from a client with a non-UTF-8
User-Agent would fail validation. A sink named in a review is a pointer
to verify, not a finding. The real sink is activities.user_agent, from
three document paths and the connected-apps revoke.
4. And the wiring leg codex asked for, on that real sink: a request
through the router with a malformed header, reading the STORED value out
of the activities row, with a control asserting an ordinary header is
kept VERBATIM. Unwiring the production call site fails it; the helper's
unit test does not notice.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): apply the key rule at every level, not once (BUG-2803)
Codex round 7, both findings, the first confirmed by measurement.
1. THE RECURSION WENT ONE LEVEL TOO DEEP. Once the walk descended into
a JSON-encoded string it treated the WHOLE subtree below as
JSON-encoded, so a value nested two levels down — an ordinary string
inside a `fields` blob that happens to hold JSON text — was refused.
That is a false rejection, and the measurement says so plainly. With the
depth-2 check disabled, on Postgres 17:
depth 1 (the fields blob itself) -> 400 (correct: Postgres parses it)
depth 2 (a string INSIDE the blob) -> 201 (accepted, no error)
control -> 201
The handler parses `fields` ONCE. The inner text is re-escaped when the
blob is written, so what Postgres receives has a doubled backslash and no
escape at all. Only the document Postgres itself parses can carry a fatal
one.
The nested call now passes false rather than true, which makes this a KEY
RULE APPLIED AT EVERY LEVEL rather than a depth limit: a JSON-encoded key
INSIDE a document still recurses (pinned by a test), an ordinary one does
not. Same correction as round 6's natural-shape fix, one level further in
— I fixed the sibling case and left this one, which is CONVE-18's lesson
about my own enumeration being a sample too.
I checked whether anything re-parses a value inside the blob before
loosening this, rather than assuming: `arguments` was the candidate, and
parsePlaybookArguments asserts it is a native ARRAY (raw.([]any)) rather
than a JSON string, so it is covered by the natural-shape rule and needs
no second parse.
2. AN ERROR MESSAGE THAT SENT CLIENTS THE WRONG WAY. The OAuth dynamic
client registration handler prefixed every decode failure with "Request
body must be JSON". A body carrying a NUL is valid JSON, so that message
sends a client hunting a syntax error it does not have. The two failures
are now distinguished.
Round 7 also reports no break in normal CLI, MCP or web-client request
generation — they marshal JSON and encode paths and query parameters —
which is the first thing any round has said about the client surface.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): complete the artifact check, make the walk path-aware (BUG-2803)
Codex round 8. It confirmed round 7's two fixes, then found two real
defects and two inaccurate comments — the comment half being the angle
the round was asked for.
1. THE ARTIFACT CHECK MISSED TWO REACHABLE FIELDS. artifactIsBindableText
walked the decoded artifact by TYPE, so it never covered Provenance —
whose strings are rendered into a Markdown footer appended to the stored
content — and never matched Arguments, declared []map[string]any, a
concrete slice type the walk's []any case does not match. A YAML NUL
escape in either reached storage.
It now MARSHALS the artifact and searches the output for the escape
encoding/json produces. A type switch over a struct that grows is a list
that goes stale in silence; marshalling covers every exported field,
including ones added later. The one thing it cannot see is invalid UTF-8
(which marshals to U+FFFD), and it does not need to: step 2 rejects that
in the request bytes, and YAML cannot manufacture it from valid input —
its escapes name code points, where \0 names a NUL.
Both new cases fail with the check disabled; the raw-byte cases still
pass, killed by the raw check, so each leg is discriminating.
2. THE WALK WAS NOT PATH-AWARE. A collection may declare a user field
literally named `schema` or `tags`. The walk consulted the wire-key list
at every level, so `{"fields":{"schema":"..."}}` treated a user field
name as a wire key and refused valid text holding a JSON example.
The key list is now consulted only OUTSIDE caller data — not under a
natural `fields` object, not inside an element of a `tags` array, not
inside a re-parsed document. Combined with round 7's fix that makes the
descent exactly one level deep BY CONSTRUCTION, which is why the depth
counter is gone: with the flag no longer inherited, a bound could never
fire, and dead protection reads as protection. The depth-bound test is
replaced by one that pins the property directly — an escape IN the
parsed document is refused, one BELOW it is accepted, and a
wire-key-shaped user field does not restart the descent.
3. THREE COMMENTS CORRECTED, all mine, all of the kind a reader would
believe without checking:
- MaxBytesReader: Close FORWARDS to the underlying body rather than
being a no-op, and with a nil writer there is no automatic 413 — the
cap surfaces as a read error the callers turn into 400. Behaviour
unchanged; only the claim was wrong.
- parseArtifactRequest said "three checks" while implementing five, and
its returns list omitted ErrArtifactUnbindableText. Both added by this
branch, which is exactly the prose a change is most likely to falsify
(CONVE-23).
- errJSONBodyNUL claimed all 65 callers surface its message. The STATUS
is uniform; the wording is not — several substitute a generic string.
4. And one in a test: the timeline fixture said both backends hold a
CHECK constraint a raw NUL violates. items.fields is a plain TEXT column
with no CHECK on SQLite. What was OBSERVED is "SQL logic error:
malformed JSON"; the likely source is an expression index over
json_extract, and that attribution is recorded as NOT verified rather
than asserted.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): a regression this branch introduced, and the same trap again (BUG-2803)
Codex round 9. Both findings are mine, one of them a regression from the
round-8 restructure two commits ago.
1. THE ROUND-8 RESTRUCTURE REOPENED THE ORIGINAL DOOR. Taking the
JSON-encoded branch for a listed key skipped the plain "does this string
contain a NUL" check and asked only "does the document this string
carries hold an escape". Those are different questions. So
{"fields":"a<NUL escape>b"} — a direct NUL in the fields value, the very
first case this whole change closed — was accepted again.
Both checks now run. The test pins all three legs: a direct NUL in the
fields string, an escape inside the fields document, and an ordinary
fields string that must still be accepted, so the first two cannot pass
merely because everything under a listed key is refused.
2. THE ARTIFACT CHECK FELL INTO THE TRAP IT WAS WRITTEN AGAINST. It
searched the MARSHALLED bytes for the escape sequence, and a value
holding the six LITERAL characters marshals to a doubled backslash which
still contains that sequence as a substring — so valid content was
refused. Artifacts are documentation; text about a JSON escape is
exactly what one carries.
Worse than the bug: the comment I wrote asserted the ambiguity "cannot
arise here". It was the same doubled-backslash case bodyDecodesNUL exists
to resolve, one function away, and I wrote a sentence explaining why it
did not apply instead of checking. The marshalled form is now decoded
again and walked with the same machinery — the round trip is what makes
every field reachable without a type switch, the walk is what makes the
answer exact.
Its test asserts literal escape TEXT is accepted in title, body and a
field value, with a counterfactual leg asserting a real NUL in each of
those places is still refused, so acceptance cannot come from the check
doing nothing.
Reverting either fix fails its test and only its test.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* docs(backup): the one case where an export is not importable (BUG-2803)
Codex round 12, an operational pass. It found no migration or config
requirement, and two documentation gaps.
docs/backup.md promises that application-level export/import is portable
across SQLite and PostgreSQL. Since BUG-2803 that has one exception: a
workspace whose stored data contains a NUL exports fine and is refused on
import. It can only affect data written before the rule existed and only
on SQLite, which accepted it — a PostgreSQL instance never stored one.
`pad db migrate-to-pg` has the SAME problem and reports it worse: it
copies rows directly and never passes through the import guard, so a
legacy row fails against PostgreSQL's JSONB parser partway through the
copy rather than being refused up front. That is the likelier way an
operator meets this, since it is the operation that puts an entire old
SQLite database in front of PostgreSQL for the first time. Recorded on
BUG-2810, which owns the preflight and repair.
Round 12's other finding — that the PR's stated release note covered the
JSON 500-to-400 change and none of the rest — is fixed in the PR body
rather than in the tree.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* test(server): close two blind spots the tests themselves had (BUG-2803)
Codex round 13, asked whether the new TESTS are sound. Five findings;
these are the two that were self-contained. The other three are recorded
on the item with what each needs.
1. THE COMPLETENESS SCAN WAS BLIND TO FORM BODIES. It matched only
`.Body`, so FormValue / ParseForm / MultipartForm — which read the
request body just as surely — were invisible. It therefore reported full
coverage while the OAuth form-encoded handlers were entirely outside its
view. Widened, and it immediately failed on handlers_oauth.go, which is
the instrument working.
That file is now ACCOUNTED FOR AS A KNOWN GAP rather than as safe: the
OAuth handlers read form-encoded bodies that no rule in this family
covers (the transport rules see the query half of r.Form, not the body
half), tracked as BUG-2811 and needing a fosite-backed fixture to
measure. The test now STATES the gap instead of being blind to it, which
is the difference between a completeness claim and a completeness
appearance.
2. THE TRUNCATION TEST ADMITTED AN IMPLEMENTATION THAT RETURNED "". Its
assertions were: within the limit, bindable text, a prefix of the input.
An empty string satisfies all three. It now also asserts that an input
fitting the limit comes back UNCHANGED, and that no more than one rune
(4 bytes) is lost to the boundary — so a truncator that drops too much
fails, not just one that keeps too much.
Both were found by asking whether a broken implementation would pass,
which is the question CONVE-12 is about and which I had applied to the
production code and not to these two tests.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* test(server): the three remaining round-13 gaps (BUG-2803)
Codex round 13's other three findings, all of the same shape: a test
that would stay green with the production change reverted.
1. THE MANIFEST CHECK WAS UNTESTED. The bundle test built archives
containing only pad-export.json, so disabling the INDEPENDENT attachment-
manifest check left the suite green. The new test builds a bundle with
both entries, differing only in the manifest, so a refusal cannot come
from the export half. Verified by disabling each check separately: only
the matching test fails, so the two are independently covered.
2. THE TEST-EMAIL CHANGE HAD NO HANDLER-LEVEL TEST. Every existing leg
exercised decodeJSON, so reverting handlers_admin.go to default EVERY
decode failure to the admin's own address passed them all. The new test
drives the real endpoint with a wired mock sender and pins the
distinction that used to collapse: an ABSENT body still means "send it
to me" (control), an ordinary body still sends (control), and a body that
is present and refused answers 400 rather than being reinterpreted as
the default recipient.
3. THE MULTIPART LEG CHECKED ONE BYTE CLASS. A filter rejecting NULs
while letting malformed UTF-8 through would have passed it. It now drives
both, which matters because invalid UTF-8 is the class that reaches
Postgres as 22021 on a UTF8 database.
Round 13 was asked whether the new TESTS are sound — deterministic,
order-independent, and failing on broken code. It reported the fixtures
isolated and found five ways they were not discriminating. Two were
fixed in the previous commit; these are the rest.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* test(server): pin the wiring at every call site, not one (BUG-2803)
Codex round 14 confirmed round 13's five, then found the same shape one
level out: reverting a SINGLE call site back to the unsafe form left the
whole suite green, because the surviving fixtures are ASCII and a
helper's unit test does not care who calls it.
TestTextSafeHelpersAreUsedAtEveryCallSite asserts the wiring STATICALLY
rather than adding a fixture per site (an OAuth connection, a cloud
login, four audit paths). A byte-slice truncation of a caller string
fails it, and so does a raw User-Agent read outside the exempt set. Both
directions are checked: finding none of the SAFE form also fails, so a
scan that silently matched nothing cannot pass forever.
The User-Agent exemptions carry counts rather than being blanket, so a
NEW raw read in an exempt file still fails. All four reads in
handlers_auth.go are exempt because they feed a HASH — CreateSession
hashes the header and stores no text — and sanitising before hashing
would be actively harmful: login would store sha256(sanitised) while the
session check still hashes the RAW header, failing validation for every
client with a non-UTF-8 User-Agent. middleware_request_text.go's one raw
read is requestUserAgent itself.
Verified by reverting one truncation call site and one User-Agent call
site independently; each fails the test.
Round 14's third finding is fixed behaviourally rather than statically,
because the static scan cannot see it — handlers_oauth.go is already
listed for its form-body reads. TestOAuthRegisterRefusesNULBody drives
the real dynamic-registration endpoint with cloud mode and an OAuth
server wired, with a control leg registering successfully, and pins both
the refusal and the message split: the body IS valid JSON, so the answer
must not send a client hunting a syntax error.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): match wire keys the way the decoder does (BUG-2803)
Codex round 16, asked whether this change is consistent with its siblings
in the same file and extensible by someone who did not write it. It found
a live bypass instead.
encoding/json matches an incoming key to a struct field by an exact match
first and a CASE-INSENSITIVE one otherwise, so {"Fields":...} and
{"FIELDS":...} land in ItemCreate.Fields exactly as {"fields":...} does.
The walk looked the key up case-SENSITIVELY, so it skipped the nested
document for a body the handler went on to accept, and the database
answered the original 500.
Measured before the fix: `fields` refused, `Fields` and `FIELDS`
accepted.
This is the same defect shape as everything else in this unit — a check
that agrees with one layer's rules while the layer that actually consumes
the value uses different ones — which is why the fix is a PREDICATE
rather than a wider map: the map is the vocabulary, and the matching RULE
belongs to the consumer. Someone adding a key should not also have to
remember to add its spellings.
The test drives six spellings including mixed case, with a control
asserting an unlisted key stays caller data in any casing, so this is
case-insensitive matching rather than matching everything.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): fold keys the way encoding/json folds them (BUG-2803)
Codex round 17, first of five findings. The previous commit fixed the
ASCII half of key matching and left the Unicode half, which is this
bug's own pattern one more time.
encoding/json matches with Unicode SIMPLE FOLDING, not lower-casing.
U+017F LATIN SMALL LETTER LONG S folds to 's', so "ſchema" reaches the
`schema` struct field while strings.ToLower("ſchema") is unchanged and
missed the allowlist — a nested NUL under that spelling reached the
handler undetected.
Matching is now strings.EqualFold against each canonical key. The test
carries both a lower-case fold spelling and an upper-case one alongside
the ASCII cases, and keeps its control asserting an unlisted key stays
caller data in any casing.
The other four round-17 findings are recorded on the item rather than
patched here: they are genuine layer disagreements (duplicate keys
merging differently in a typed decode than in a map, a scan-failure
disposition on inputs the typed decode tolerates, and unknown-field
policy) whose fixes are design decisions rather than corrections, and
this seat is near its context bar. Each is written up with the
measurement it needs.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
* fix(server): pin that a NUL-bearing manifest refusal keeps the partial workspace (BUG-2803)
Codex round 18. The comment on the manifest NUL branch said refusing the
input "stops it from being reached this way" and stopped there, which
reads as though the refusal undoes the import. It does not.
A plain error with a non-nil workspace keeps the partial workspace,
exactly as every other manifest failure in this loop does — the rollback
branch fires only for *importStatusError, and mid-stream manifest
failures intentionally keep what was imported (TASK-896). Returning a
rollback-shaped error here would give NUL-bearing manifests different
semantics from malformed ones, which is a change to the bundle-import
contract rather than a fix to this bug.
So the behaviour is unchanged and now DELIBERATE: the comment states it,
and the test asserts the persisted state rather than only the HTTP
answer. Mutation: routing the branch through *importStatusError makes
the refusal roll back, and the new assertion fails naming the release
note it would falsify. The pre-existing status/body assertions do not
notice.
Claude-Session: https://claude.ai/code/session_01AUvLoXsKdS5sdpYju6rj4p
* docs(server): record the four map-model disagreements as dispositions, and pin them (BUG-2803)
Lead ruling day-68 is land-and-follow: this branch lands on its measured
commits, and the token-stream rewrite is the BUG-2812 unit's spec rather
than a late restructure of an 18-commit branch under review pressure.
That makes the four open findings from rounds 16-17 something to WRITE
DOWN precisely, not something to leave in a trail comment.
The doc comment on bodyDecodesNUL now carries all four, with the one
root cause named: this scan decodes into map[string]any and the typed
decode does not agree with that model about keys. Two under-refuse
(duplicate-key merge; scan-failure passthrough) and are BUG-2812's spec
- both dissolve under a walk that never builds values. Two over-refuse
(unknown fields; case-variant duplicates) and are ACCEPTED, because
refusing is the safe direction. The asymmetry is stated rather than
smoothed over: within the map model, (1) and (4) are one defect seen
from two sides and only one of them fails safe.
Finding (3) is an observable compatibility change - a forward-compatible
field carrying a NUL escape now gets a 400 where it got a 200 - so it
goes in the release note as well as here. A qualification only protects
where the actor meets it.
All four are pinned by a test, measured on this tip rather than carried
over from the round-16/17 write-up. The two known-gap legs assert the
WRONG answer on purpose: when BUG-2812 lands they FAIL, naming the doc
comment and the release note as what to update. Both gap legs carry a
premise assertion - the same bodies with the disagreement mechanism
removed ARE detected - without which they would pass against a check
that detected nothing.
Claude-Session: https://claude.ai/code/session_01AUvLoXsKdS5sdpYju6rj4p
* test(server): wire release-note item 10 to the router, with its before-state measured (BUG-2803)
The disposition test proves bodyDecodesNUL RETURNS true for an unknown
field carrying a NUL escape. The release note claims the API answers
400. Those are different claims and only the second one is what an
operator or client author reads - CONVE-19, my own convention: a
direct-call test vouches for the component, not its binding.
Two legs, and the control is the load-bearing one. An unknown field with
an ordinary value must still be ACCEPTED, so this pins "refused for the
NUL" rather than "refused for being unknown". The handler does not
reject unknown fields; if it ever started to, the note's explanation
would be wrong while its status code stayed right, and no
status-code-only assertion could see that.
The before-state is measured rather than asserted from memory. Disabling
the check makes the same request answer 201 - which is main's behaviour,
since decodeJSONWithLimit there unmarshals straight into the typed value
and the key is dropped. So "answers 400 where it answered 200" is a
measurement in both directions, not a recollection of one.
Claude-Session: https://claude.ai/code/session_01AUvLoXsKdS5sdpYju6rj4p
* docs(backup): the NUL rule lives in the binary, not the database (BUG-2803, BUG-2813)
Codex round 19, the fresh-angle deploy/rollback/mixed-version pass.
docs/backup.md said a NUL-bearing row "can only affect data written
before that rule existed, and only on SQLite". The second half is true.
The first half is false, and the reason is the interesting part: the
guard is in decodeJSONWithLimit, so the invariant is a property of the
running BINARY, not of the database.
On SQLite any window where an older binary serves the same database can
still write one - a rollback after upgrading, a staged rollout with an
old and a new instance sharing a database, a second older instance on
the same file. The window closes, the guard returns, and the rows are
already stored, behaving exactly like genuinely old ones. A rollback is
an ordinary operational move, so this is not an exotic path.
The doc now states the binary-version dependence, says which dialect is
affected and why PostgreSQL is not (it refuses a NUL itself, at every
version), and gives the operational answer: drain writes from older
binaries before the new one serves, or roll forward rather than back.
Store-layer enforcement - so the running build stops mattering - is
filed as BUG-2813 rather than added here. It is a dialect-level change
and the day-68 ruling on this unit is land-and-follow.
The same false implication was carried by the PR's release note calling
such a workspace "legacy"; corrected there too.
Claude-Session: https://claude.ai/code/session_01AUvLoXsKdS5sdpYju6rj4p
* docs(server): cite the ruling in house style, not the team-room day counter (BUG-2803)
"lead ruling day-68" is the internal day counter, which means nothing to
anyone reading this repo and is inconsistent with every other citation
in it - the codebase cites a lead ruling by DATE or by BUG ref, never by
day-N. Replaced with the bug ref, which is the part a reader can
actually follow.
Claude-Session: https://claude.ai/code/session_01AUvLoXsKdS5sdpYju6rj4p
* docs(server): drop a commit count I had already measured as wrong, and stop asserting a cause I borrowed (BUG-2803)
Two defects in a comment I wrote an hour ago, both of the kind this
unit's trail keeps recording.
"an 18-commit branch" - the branch was 20 commits at
|
||
|
xarmian
|
50499bceb8 |
fix(server): charge workspace imports against the plan limit (BUG-2793) (#1219)
* fix(server): charge workspace imports against the plan limit (BUG-2793) `POST /workspaces` enforces the user-scoped `workspaces` plan limit before creating. `POST /workspaces/import` did not, and it mints a workspace through the same store.CreateWorkspace — so a user at their plan's limit could exceed it by exporting any workspace and importing it back. Cloud only; enforceUserPlanLimit is a no-op when cloudMode is off, so self-hosted was never affected. Dave ruled the shape on day 63: an import IS a new workspace and counts, with no exemption for re-importing something you previously owned. Export provenance is not trustworthy enough to gate billing on, and the at-limit case that deserves relief — undoing a delete — is served by the restore endpoint, which mints nothing. The call is one line; the PLACEMENT is the fix. It sits beside the #1212 consent gate, ABOVE the Content-Type dispatch, for two reasons: - handleImportWorkspaceBundle is reachable only through that dispatch, so a gate below it would cover the JSON path and leave the tar.gz path — the one that carries attachments, and the one a real export produces — wide open. - Above either body read, so a refused caller never uploads. The two paths have very different size bounds; the gate precedes both. That is not a hypothetical: the mutation matrix includes it. Moving the gate below the dispatch fails ONLY the bundle test and leaves the JSON test green, which is exactly the false confidence a placement-blind fix would have shipped. Five tests, four of them controls, because a gate is easy to get green and hard to get right: the JSON path refuses at the limit, the bundle path refuses at the limit, under-the-limit is NOT refused (a gate wired to the wrong feature key would pass the first two), self-hosted is unaffected (this must not introduce a limit where there are no plans), and a request with no resolved user is not charged — mirroring the create side's `userID != ""` guard, which is not defensive padding but the difference between "no limit applies" and a nil lookup. This is the SECOND gate on workspace creation the import door skipped; the first was the OAuth consent gate (IDEA-2756, PR #1212). Two have now diverged this way, which is the argument for the shared pre-step helper — tracked separately rather than folded in here. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2793 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * test(server): make the import controls assert success, not merely non-refusal (BUG-2793) Codex round 1. The three control tests were vacuous and I would have shipped them. WorkspaceExport.Version defaults to 0 and the import requires 1, so the under-the-limit, self-hosted, and no-resolved-user cases were all failing with a 500 long before they reached anything this change is about. They passed because they asserted only "not 403" — and a 500 is not a 403. That made all three useless in the same direction: a fix that broke imports outright, or a gate wired to refuse everything with a non-403 status, would have sailed through them while the two refusal tests stayed green. The controls existed precisely to catch that, and could not. Fixed by setting Version: 1 and asserting the real success status, 201. A control that cannot tell success from a server error controls nothing. Mutation matrix re-run after the change, because a matrix over vacuous tests proves nothing either: removing the gate still fails exactly the two refusal tests, and the three controls now pass on genuine imports rather than on identical 500s. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2793 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * docs(server): scope what the no-resolved-user import test actually pins (BUG-2793) Codex round 2 pointed out that this test locks in a 201 for a caller with no resolved user, and that a reader will take that as approval of userless workspace creation. It is not. The test pins the GUARD — that import behaves as create does when no user resolves — and it drives the handler directly, so it does not prove a real legacy workspace token reaches this code at all. Said so in the test rather than leaving the 201 to speak for itself, and pointed at BUG-2809 for the question it does not answer. Round 2's three findings are all real and all filed rather than folded, because this unit's ruling is specifically the plan limit on the import door: - BUG-2808 — enforceUserPlanLimit is check-then-act, so concurrent requests can exceed any cap. A property of the helper, shared with the create door and every other feature it gates; this change inherits it rather than introducing it. - BUG-2809 — import and create still enforce different preconditions on the same mint: required-name validation (a live defect — an empty name yields an empty SLUG, which is a routing key), settings normalization, source attribution, and the userless case above. Filed as a class because the mechanism is one thing and the record now shows it failing twice. The reviewer also confirmed two non-doors, which is the useful negative: autoCreateWorkspace is intentional first-workspace provisioning, and `pad db migrate-to-pg` calls ImportWorkspace directly as an operator-only migration outside HTTP entirely. BUG-2793 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * test(server): pin the JSON-path placement and assert "not charged" as data (BUG-2793) Codex round 3, on the tests. Two ways they could pass without proving what their names say. 1. The JSON refusal test sends VALID json, so a gate placed after the decode would still return 403 and it would stay green. The bundle test covers the gzip half of the placement claim; nothing covered the JSON half, and the code comment claims the gate sits above EITHER body read. Added an at-limit case with an undecodable body: reaching 403 rather than a decode error is only possible if nothing read the body first. 2. The no-resolved-user test asserted only a 201. A regression that quietly attributed the import to the at-limit fixture user would also return 201 and pass. It now asserts the fact instead of inferring it — the user's workspace count is unchanged across the request, and the created workspace has no owner. The mutation matrix now separates the two placements, which is the point of having both tests: - gate below the Content-Type dispatch (still above the decode) -> only the BUNDLE test fails. - gate below the JSON decode -> the bundle test AND the new JSON test fail. Neither mutation is caught by the original refusal test, which is what "passes for the wrong reason" looked like here. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2793 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN |
||
|
xarmian
|
427540706c |
fix(documents): bound the rename cascade at the title and at the projected total (BUG-2798, BUG-2796) (#1218)
* fix(documents): bound the rename cascade at the title and at the projected total (BUG-2798, BUG-2796) A document rename rewrites [[oldTitle]] into every linking document. Neither factor of the output size was bounded: titles had no length validation, and the cascade holds every rewritten body in memory before writing any of them. One rename could project 10 GB from a 500 KB input -- 20,000x, measured -- and OOM while holding the workspace rename lock. Two walls, per Dave's day-63 ruling. 1. Title length, bounded at write time (models.MaxDocumentTitleRunes = 255). Runes, not bytes: "255 characters" is what a user and a UI counter mean. Existing over-limit titles stay valid until their next rename -- no retro-breakage of stored data. 2. The cascade's projected TOTAL, bounded at 16 MiB (store.MaxRenameCascadeProjectedBytes), accumulated across the linking set and refused before the first rewrite is built. The total is the right quantity and a per-document cap would not have been. Measured, with the title bound already in place: one linker holding the largest body a 2 MiB request can carry projects 108,632,370 bytes -- 51.8x -- and the aggregate is linear in the number of linkers (108.6 / 217.3 / 434.5 MB at k = 1/2/4, allocation tracking output at ~1.02x). A per-document cap of C still admits k * C, which is the same unbounded shape one level up. The 16 MiB figure has a receipt in the constant's doc comment: it sits above the absolute ceiling of any cascade this development instance could produce (its entire wiki-linking corpus is 10,077,476 bytes) and 6.5x below the single-document attack. The refusal is permanent-shaped and deliberately NOT in ErrLinkCascadeContention's family: 413 with the projection in the message and no Retry-After. Contention means "someone got there first, try again"; this means "this rename cannot be performed as asked". Answering it from the retryable family would tell a client to retry forever. BUG-2796 folds in at the same validation point, as ruled -- a title containing wiki-link syntax is emitted raw by links.ReplaceTitle, so renaming to `A]] [[A` produced two broken links and reported success. The rule is derived from the two mechanisms that consume a stored bracket (the grammar at markdown.ts:327 and the unescaper at markdown.ts:753) rather than from a character blacklist: the first version of this fix banned `]`, `\` and `|` because all three "look like wiki-link syntax", and the round-trip test refuted two thirds of that. `|` in particular is a title shape resolveWikiBody contains a dedicated branch to support, and `[` passes the grammar untouched. Doors enumerated rather than assumed (CONVE-24): store.CreateDocument and UpdateDocument have exactly two callers between them, both HTTP handlers. No CLI, import, or seed path writes a document title. Update previously validated doc_type and status and NOT title -- the one field that drives the cascade -- so the handler tests drive real requests through both doors (CONVE-19). BUG-2798, BUG-2796 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): count retained bytes, bound the retry path, escape the cascade's LIKE pattern (BUG-2798) Codex round 1 on #1218. Three findings, all real, all fixed here. 1. The guard bounded projected OUTPUT, which bounds nothing when the new title is SHORTER than the old one. Renaming a 255-character title to a one-character title makes each 2 MiB linker project ~40 KiB while the cascade still retains its 2 MiB read for the compare-and-set, so hundreds of linkers exhaust memory while the counter reports well under the cap. The counter now sums RETAINED bytes — read plus written, both alive at once — so the cap is a statement about resident memory rather than about output. MaxRenameCascadeProjectedBytes becomes MaxRenameCascadeRetainedBytes and moves 16 -> 32 MiB, because the legitimate ceiling it clears doubles under the new metric (that instance's whole wiki-linking corpus retains ~20,154,952 bytes); the single-document attack retains 110,729,522, so it is still refused by 3.3x. 2. The compare-and-set's retry path bypassed the guard entirely. On contention it re-reads the linker and calls ReplaceTitle on whatever the winner wrote — a NEW input, bounded by nothing the scan had checked — so a content edit landing inside the cascade's window could grow a linker from harmless to enormous and walk the rename back into the amplification it would have been refused for. Each document's compare-and-set now carries the cap less what the other linkers hold, and re-checks the grown body against it. 3. The cascade's `content LIKE ?` search term went in unescaped, so a document TITLE decided how the pattern was read. `\` is the default LIKE escape character on Postgres and NOT on SQLite, so `[[Alpha\Beta]]` was searched for as itself on one dialect and as `[[AlphaBeta]]` on the other: linkers not found, cascade rewrites nothing, rename reports success, every link left stale. Silent and dialect-dependent. Codex named the backslash; `%` and `_` are the rest of the class (CONVE-18) — wildcards on both dialects, so a title carrying them selects documents that do not link it. An explicit `ESCAPE '\'` clause plus escapeLikePattern makes both dialects agree, rather than leaving SQLite correct by accident. Finding 3 also constrains finding 3 of the ORIGINAL fix: models' validator allows a lone backslash in a title on the grounds that both renderers handle it, which was true of rendering and false of cascading. That comment now records the dependency — allowing it is only correct while the cascade's pattern stays escaped. Tests, four new, each mutation-verified against the code it guards: - CountsRetainedBytesNotJustOutput — the shrinking rename. Asserts as a PRECONDITION that the projected-output total stays under the cap, so the test cannot pass for the old reason. - RetryRecheckesTheBudgetAgainstTheGrownBody — drives the real race through the afterLinkCascadeRead seam. POSTGRES ONLY and skipped loudly elsewhere: SQLite's BEGIN IMMEDIATE closes the window structurally, so a green run there would be a property of the DSN. - FindsLinkersWhoseTitleContainsABackslash — Postgres only, same reasoning inverted: SQLite is the dialect that was accidentally right. - DoesNotSpendTheBudgetOnDocumentsThatDoNotLinkTheTitle — `%` and `_`. Its first version asserted the decoy's content was untouched and passed against the unescaped pattern, because over-matched rows rewrite to themselves. The observable harm is that they spend the caller's budget, so that is what it now asserts. Mutation matrix for this round: output-only counter -> only the shrinking test fails; retry check removed -> only the retry test fails (PG); LIKE unescaped -> the budget legs fail on SQLite and the backslash test fails on PG. Gates: `go test ./...` under Postgres 17 EXIT=0; SQLite packages EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798, BUG-2796 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): tighten the retry budget, stop charging no-op rewrites, order the typed check first (BUG-2798) Codex round 3 on #1218, an edge-case angle over the new arithmetic and control flow. Three findings fixed, one declined. 1. The retry budget credited back this document's own share, on the reasoning that the retry replaces it. It does not: the original read and rewritten bodies stay reachable through `updates` while the write loop runs, so the re-read and its rewrite are allocated ON TOP of them. The bound could be exceeded by up to one document's share while the arithmetic still reported it satisfied. The budget is now the genuine headroom, `cap - retained`. 2. A concurrent edit that REMOVES the link left a body with no occurrences, which cascadeRetainedBytes still charged twice — once for the read and once for a rewritten copy that does not exist, because strings.Replace returns its input unchanged when there is nothing to replace. That could refuse an otherwise valid rename for memory the cascade never allocates. 3. The handler classified this error by PROSE before testing it by identity. The UNIQUE-constraint arm matches a substring, and the refusal error embeds the caller's title verbatim, so renaming a document to a title containing the words "UNIQUE constraint" came back as a 409 name collision — advice to pick a different name, for a rename that was refused for size and would fail identically under any name. Typed sentinel now tested first. DECLINED: unchecked int64 arithmetic in the projection. The multiplicands are derived from the length of a string already resident in memory, so overflowing int64 needs a single document body of roughly nine exabytes; and the accumulator returns as soon as it passes the cap, so it cannot run away either. Saturating arithmetic here would be guarding a state the machine cannot reach. Tests, three new, each mutation-verified: - RetryBudgetExcludesThisDocumentsOwnStrings — deliberately separate from the existing retry test, because that one catches the check being ABSENT and this one catches it being too GENEROUS. The grown body is sized to fall BETWEEN the two budgets; a body far over the cap cannot tell them apart. - ConcurrentEditThatRemovesTheLinkDoesNotRefuseTheRename — its first version sized the link-free body against the CAP rather than against the retry's real headroom, so the refusal it caught was correct behaviour and the test was wrong, not the code. Re-sized against the headroom: fits when charged once, does not when charged twice. - IsNotMisreportedAsATitleCollision — at the handler, since the defect is entirely in its classification order. Mutation matrix for this round: credit the share back -> only the tight-budget test fails; charge the no-op body twice -> only the link-removed test fails; order the substring arm first -> only the misclassification test fails. Gates: `go test ./...` under Postgres 17 EXIT=0; touched packages re-run after the lint fix EXIT=0; gofmt clean; `make lint` 0 issues. CI green on |
||
|
xarmian
|
771ec5bbaa |
fix(server): refuse invalid-UTF-8 and NUL query values at the transport (BUG-2784) (#1217)
The query-string half of BUG-2782. A caller-supplied query value reached a
Postgres text comparison, Postgres refused the parameter, and the handler
answered 500 — the honest answer is 400, because the caller asked about
something that cannot exist.
ValidateQuery is a root-router middleware beside ValidatePath, refusing 400
invalid_query when a decoded query key or value is invalid UTF-8 or carries
a NUL. Both share one predicate, bindableText.
Measured on Postgres 17 (server_encoding UTF8) at
|
||
|
xarmian
|
1933041027 |
fix(events): surface the SUBSCRIBE error and refuse callers instead of admitting a dead stream (BUG-2764) (#1215)
* fix(events): surface the SUBSCRIBE error and refuse callers instead of admitting a dead stream (BUG-2764) * fix(watchevents): surface the SUBSCRIBE error at construction and on resubscribe (BUG-2764) * docs(events): the idle-cycle prose and metric Help now name the third install-nothing reason (BUG-2764) * fix(events): retry uncovered workspaces, wait on in-flight records, refuse after Close (BUG-2764 codex round 1) * fix(events): post-loop check waits on an in-flight record before trusting a live entry (BUG-2764 codex round 2) * test(watchevents): assert what the failed subscribe leaves, not how long it takes (BUG-2764 codex round 3) * docs(events): prose says what the code guarantees — delivery failure or shutdown, replacement refusals do not count as cycles (BUG-2764 codex round 4) * test(server): the 503 mapping is asserted on both subscribe branches; docs scope the refusal to the activity stream (BUG-2764 codex round 5, BUG-2800) * fix(events): uncovered-workspace retry logs are quiet once the bus is closing (BUG-2764 codex round 6) * fix(events): the uncovered-retry log promises a retry only while subscribers remain (BUG-2764 codex round 7) |
||
|
xarmian
|
cade263fe5 |
fix(store): compare-and-set the wiki-link cascade's writes (BUG-2785) (#1213)
* fix(store): compare-and-set the wiki-link cascade's writes (BUG-2785)
A document rename cascades through every document linking the old title
as a read-modify-write across two statements: SELECT each linker's
content, rewrite the string in Go, UPDATE the row. A content edit to a
linker committing between those two statements was silently overwritten
— the cascade wrote the body it built from the version it read, with no
error and no version row for the loss.
Same lost-update shape as BUG-2770's activity-metadata merge, one table
over, and fixed the same way: the UPDATE now carries `content = ?` with
the body the cascade read, plus a bounded retry that re-reads the row
and re-applies the rewrite.
DIALECT SCOPE, which decides what the tests can prove. Reachable on
POSTGRES only. SQLite's DSN sets `_txlock=immediate`, so UpdateDocument's
db.Begin() takes the write lock at BEGIN and holds it across the whole
read→write window; a concurrent edit cannot commit inside it and
serializes on busy_timeout instead. On Postgres under READ COMMITTED each
statement takes a fresh snapshot and the stale body wins. The CAS is
therefore a no-op on SQLite by construction — the predicate always
matches, because nobody else can have written.
Two consequences, both acted on rather than noted: the new tests SKIP
loudly on SQLite instead of passing for a reason unrelated to this fix,
and the mutation matrix was run under Postgres, where removing the CAS
leaves a SQLite suite entirely green.
The retry rewrites the WINNER's body rather than replaying the original
rewrite — replaying it would reintroduce exactly the text this bug loses.
A mutation that replays instead is in the matrix.
THE ZERO-ROW RESULT NEEDS A PROBE. The UPDATE now has two predicates that
can each refuse it, and RowsAffected cannot say which did: `deleted_at IS
NULL` (the linker was archived — a documented normal outcome, stop) or
`content = ?` (a concurrent edit landed — re-read and retry). Treating
them alike either retries forever against a deleted row or discards a
live linker's rewrite, so a probe distinguishes them, as BUG-2770 needed
for the same reason. The probe reads through tx, never the pool
(BUG-2409).
On retry exhaustion the RENAME fails rather than leaving one linker
holding a title that no longer exists. The alternative — log and continue
— was considered and rejected: it trades a loud retryable failure for a
silent inconsistency, and a rename is atomic in intent. Exhausting three
attempts needs three consecutive commits to the same linker inside one
cascade.
Adds afterLinkCascadeRead, the seam between the cascade's read and its
writes. No existing seam reaches that gap: afterDocumentPreLockRead fires
before the transaction, afterDocumentPreWrite before the renamed
document's own update.
That new seam also closes a gap a previous unit recorded as permanently
open: the `deleted_at IS NULL` guard carried a note calling itself
UNTESTED because reaching its window "would cost a fifth seam". This is
that fifth seam, so the note is removed and replaced by a record of the
closure, and the archived-linker test now drives exactly that window.
Mutation matrix, run under Postgres — 5 mutants, 5 detected, including
one that is literally the pre-fix code:
M1 CAS predicate removed (the unfixed behaviour) → lost-update test
M2 soft-delete probe arm removed → archived-linker test
M3 retry budget cut to one attempt → lost-update test
M4 retry replays the original rewrite → lost-update test
M5 seam removed (control: tests must notice they never raced)
→ lost-update test
Gates: gofmt clean, lint 0 issues, govulncheck clean, full suite green on
SQLite (28 pkgs) and on Postgres 17 (28 pkgs, internal/store 346s vs 79s
— the positive control that the PG legs ran rather than skipped).
Not fixed here, filed as BUG-2795: cascadeTitleRename, the ITEM-side
cascade, has the identical defect and no lock closes its window either.
Its fix does not transfer — it rewrites by POSITION from item_wiki_links
offsets, so a retry must re-derive positions from the winner's body and
re-run replaceWikiLinks, which is a redesign of the retry unit rather
than a predicate on an UPDATE. wiki_links.go's claim that it "matches the
document rename behavior" is corrected to say which half no longer
matches.
* style(store): gofmt the CONVE-23 sweep comment (BUG-2785)
Same failure as the previous unit, same cause, and worth naming rather
than quietly fixing: gofmt wants a blank line between list items once one
item grows a second paragraph, which the note about the previously-
untested deleted_at guard made true.
I re-ran build and the full Postgres suite after those comment edits but
not lint, because the change was 'only a comment' — the exact reasoning
the previous unit's fix commit warned about in writing, one unit earlier.
The PR body's claim that all gates were re-run after the prose edits was
false and has been corrected there too.
The rule as remembered does not work. The mechanical form does: gofmt and
lint are the last action before a push, comment-only changes included.
* fix(links,server): stop a rename hanging the server, and report cascade contention honestly (BUG-2785)
Three findings from Codex round 2 on this PR. The first is a server hang.
1. ReplaceTitle could never terminate. replaceAll looped "find old in
result, splice new in" — re-searching the string it was building,
including the text it had just inserted. When the NEW title contains
the OLD link token it grows without bound.
Measured, not argued: ReplaceTitle("x [[A]] y", "A", "A]] [[A")
builds `[[A]] [[A]]`, which still contains `[[A]]`; a probe against
the old implementation ran 3s without terminating before being
killed. Document titles have no validation, so this is reachable from
user input — and the caller is inside the rename transaction holding
the workspace rename advisory lock (BUG-2778), so the hang would take
every other rename in that workspace down with it while exhausting
memory.
strings.Replace with n = -1 has the semantics that were wanted:
non-overlapping, left-to-right, over the input. Three-case regression
test, all three of which fail against the old implementation, plus a
control that catches a "fix" which terminates by doing nothing.
Pre-existing, and folded in rather than filed: three lines against a
server hang, and this PR's retry calls the helper again per attempt,
which makes it reachable more often than before.
2. Retry exhaustion surfaced as an opaque 500. The rename rolls back
cleanly and retrying can succeed, so "an internal error occurred"
tells the caller the opposite of the truth. Adds the exported
ErrLinkCascadeContention sentinel; the handler now answers 503
lock_contention with Retry-After, reusing the disposition BUG-2778
already established for 55P03/40P01.
3. That 503's message claimed the workspace was "busy with another
rename". 55P03 there is just as likely to be an ordinary content edit
holding the row, and the new arm is definitely one. It no longer
names a cause the server has not established.
Also closes the coverage gap round 2 named around this unit's own
decision: exhaustion now has a test asserting the rename ROLLS BACK
(target keeps its title) and that the concurrent editor's text survives
that rollback. cascadeRewriteAttempts becomes a var so the test can
force exhaustion at 1 rather than arranging three consecutive commits,
which would need a per-attempt hook in production code — the divergence
from its const sibling is noted where it lives.
Records two limitations in the code rather than leaving "the cascade is
safe now" to rot: the MIRROR direction is still open (a content writer
that read before this transaction can commit afterwards and reinstate
the old title — fixing it means giving ordinary content writes a CAS
too), and delete-then-restore of a linker mid-rename brings back the old
title. Both pre-existing, neither worsened here.
Mutation matrix now 7 mutants, 7 detected, run under Postgres:
M6 (%w -> %v, sentinel lost) and M7 (exhaustion swallowed) cover the new
mechanisms.
Gates re-run on the tree being pushed, tests included, after the final
comment edit rather than before it: gofmt clean, lint 0 issues, SQLite
28/28, Postgres 17 28/28 (internal/store 345s vs 79s). gofmt caught an
unformatted test file locally this time, which is the point.
* fix(store,links): correct three prose claims and close the SQLite coverage gap (BUG-2785)
Codex round 4, on SQLite semantics and prose accuracy. One finding says a
bug I FILED is wrong; that is the important one.
1. BUG-2795's premise was false, and this PR repeated it in a comment.
I filed that item claiming the item-side cascade has "the identical
defect" and that "no lock closes its window either", on the strength
of a grep for pg_advisory_xact_lock in items.go that turned up only
the parent-link locks.
It missed acquireWorkspaceSeqLock (items.go:2136), taken
UNCONDITIONALLY by every UpdateItem — content-only edits included —
immediately after Begin and long before cascadeTitleRename, and held
to COMMIT. So two item updates in a workspace fully serialize on
Postgres and an ordinary content edit CANNOT commit inside that
cascade's window. The scenario I filed is not reachable.
Not fully invalid: sweeping every `SET content` writer in
internal/store finds RemapAttachmentReferencesInWorkspace, which
rewrites items.content in its own transaction without that lock. So a
real but far narrower window survives — attachment remap versus
cascade, not user-edit versus cascade. BUG-2795 corrected on its
trail and dropped to low; the comment here now states the lock, the
one surviving writer, and stops claiming parity with the document
cascade.
I searched for the locks I expected rather than for what serializes
that path, then wrote a sentence broader than the search. Same
failure this PR's review has produced repeatedly.
2. "A title nobody should be able to write" was false — the document API
validates doc_type and status, never the title. Correcting it
surfaced a real second defect: renaming to `A]] [[A` now terminates
(round 2's fix) but writes `[[A]] [[A]]`, two links to nothing. Filed
as BUG-2796. The termination test deliberately still asserts the
COUNT rather than the output, so it does not freeze today's broken
rendering as intended behaviour.
3. The hang's blast-radius claim named the workspace rename advisory
lock without qualifying the dialect. That lock is a no-op on SQLite,
where the equivalent damage is the database-wide write lock the
transaction already holds under BEGIN IMMEDIATE. Different mechanism,
same outcome for everyone else.
Also closes the last coverage gap round 2 named. The CAS predicate runs
on SQLite in production and every concurrency test skips there, so
TestUpdateDocument_CascadeRewritesEveryLinkOnBothDialects does not skip:
two linkers, multiple links per body, plus a document that merely
contains the word and must be left alone. Verified it earns its place —
a mutant comparing against the rewritten body instead of the body that
was read compiles, and dies to this test ON SQLITE, where nothing else
would have caught it.
Documents the parallel-test constraint on the now-mutable
cascadeRewriteAttempts, with its boundary: no current t.Parallel test in
internal/store reaches this cascade, but that is a fact about today's
corpus rather than an invariant.
Gates on the pushed tree: gofmt clean, lint 0 issues, govulncheck clean,
SQLite 28/28, Postgres 17 28/28 (internal/store 349s vs 79s).
* docs(store,links): point the cascade comment at the root cause, qualify a claim in its second location (BUG-2785)
Codex round 5, probing cross-connection visibility and transaction
boundaries. Two findings, both mine, neither changing behaviour.
1. The "one surviving writer" framing was too comfortable, twice over.
RemapAttachmentReferencesInWorkspace is not a rare non-interactive
writer: bundle import reaches it on an ordinary user-triggered
import, and the workspace is ALREADY VISIBLE to its owner while it
runs — store.ImportWorkspace commits the workspace row (with
owner_id) in its own transaction before opening the one that inserts
items, and the bundle handler runs the remap as Phase 3 afterwards.
Both verified in code.
So that writer races ordinary item edits, not just the rename
cascade, and it is missing a guard outright rather than being an
exotic pairing. Filed as BUG-2797, which covers the remap itself;
BUG-2795 is now a consequence of it and says so on its trail. The
comment here points at the root cause instead of implying the
cascade's pairing is the whole story.
2. The dialect-unqualified advisory-lock claim survived in a SECOND
location. Round 4 caught it in links.go and I fixed it there; the
same sentence sat in the termination test's comment, and I never
enumerated the sites. That is CONVE-23's verify half failing exactly
as it warns: I fixed the instance I was shown rather than the
population. Both now qualified, and a sweep for remaining unqualified
copies leaves only BUG-2778's own comment, which already carries its
no-op-on-SQLite note.
Worth recording that this is the second time on BUG-2795 that a scope
sentence of mine ran ahead of the sweep supporting it — first "no lock
closes its window either" (acquireWorkspaceSeqLock did), now "not user
triggered" (import is). Both found by a review round rather than by me.
Gates on the pushed tree: gofmt clean, lint 0 issues, SQLite 28/28,
Postgres 17 28/28 (internal/store 348s vs 79s).
|
||
|
xarmian
|
91d92f184f |
feat(server): refuse workspace creation without may_create_workspaces consent (IDEA-2756) (#1212)
* feat(server): refuse workspace creation without may_create_workspaces consent (IDEA-2756)
The OAuth consent screen's "Let this app create new workspaces" checkbox
gated only the post-creation auto-add. A connection whose user left it
unticked could still create workspaces; it simply could not then see
them. A permission that does not prevent the action it names is a
consent mismatch.
Dave ruled it: the checkbox is a permission on whether the connected
token may CREATE, and it has to be true to what a user would honestly
expect from the option. The behaviour-change-for-existing-connections
argument loses to honest consent semantics.
Adds Server.requireWorkspaceCreationConsent, a shared gate at the top of
both endpoints that mint a workspace under the caller's account:
POST /api/v1/workspaces handleCreateWorkspace
POST /api/v1/workspaces/import handleImportWorkspace
Import reaches CreateWorkspace via store.ImportWorkspace, so it is the
same permission at a second door — lead-ruled as an application of the
same rationale, not a new decision. The gate sits above the Content-Type
dispatch, so it covers the tar.gz bundle path (whose only route is that
handler) and refuses before the 64 MiB body read.
Refusal is a 403, mirroring handleAuditLog's consent refusal (BUG-2102):
a hard decline rather than a narrowed response, because there is no
narrower version of creating a workspace.
Three non-refusal cases and one refusal, all but the last with a test:
- not an OAuth grant (PAT, CLI session, local stdio) — creation rides
on ordinary account authority
- ErrOAuthConnectionNotFound (pre-Phase-C grant) — ALLOW, matching the
backfill's may_create_workspaces=ON default. Deliberately asymmetric
with maybeAutoAddCreatorConnection's not-found branch, which declines
a convenience where this one would invent a refusal
- flag set — proceeds; the auto-add is unchanged
- a store I/O error — REFUSED, failing closed with a 500, because
allowing the create when the deciding state could not be read grants
a declined permission on the strength of a database blip. This is
the one branch with no test: injecting a store read failure needs a
fault-injecting store the package does not have, so it is reasoned
rather than measured
Population enumerated before the fix (CONVE-18): five CreateWorkspace
call sites, two of them HTTP endpoints reachable by an OAuth token (both
gated). Excluded with reasons: autoCreateWorkspace (signup-time, no
connection in context), workspace restore (un-deletes an existing
workspace), /oauth/claim (grants access, does not mint), cmd_db.go
(local store copy, no HTTP). Search boundary: the sweep traced
Store.CreateWorkspace callers and did not look for a path that inserts a
workspace by raw SQL.
Ten tests, all driving the real router rather than calling handlers
directly (CONVE-19). Every refusal leg asserts that no workspace of that
name exists afterwards, not merely the status code (CONVE-12) — a guard
that 403s after the write passes a status-only assertion. Seven mutants,
seven detected, including both guard-placement mutations.
MCP tool surface 0.25 -> 0.26. Behaviour bump on the v0.9/v0.16/v0.25
grounds: no tool name, action enum or param shape changed, but
pad_workspace.create now refuses a call it used to permit. Closest
precedent is v0.10; unlike v0.10 there is deliberately no escape-hatch
param, because the gate encodes a decision the USER made at consent time
and a bypass flag would be the app overriding its own grant.
CONVE-23 sweep for prose the change falsified: instructions.md told
agents the create still succeeds and to use the claim flow (it would
have sent them to claim something that was never created); the
TASK-2753 allow-list guard entry asserted the same and posed IDEA-2756
as open; the MCP catalog and CLI help described only the flag=true path;
maybeAutoAddCreatorConnection's flag-off branch is now unreachable from
its sole caller and is documented as dead code kept for contract, to be
deleted only with the guard. CLAUDE.md was already stale at v0.24 (v0.25
bumped the constant without it) — brought to v0.26 with a backfilled
v0.25 line.
The consent screen and console copy are unchanged: they were the
misleading half of this bug, and the fix makes them true.
* docs(server): state the import gate's reachability precisely (IDEA-2756)
The import-side gate is correct but currently unexercised in production,
and the first framing of this change did not say so.
WithMCPTokenIdentity is stashed by exactly one middleware, MCPBearerAuth,
mounted on /mcp alone. An OAuth connection reaches an /api/v1 handler
only through the in-process MCP dispatcher, and that dispatcher's route
table has a workspace create action but no workspace import. So no
OAuth-bound caller can reach handleImportWorkspace today.
The gate stays, and the comment now says why: adding that action later
must not silently reopen the door, which is the state a create-only fix
would have left armed.
Found on a verify pass reading the middleware mount points, not by the
tests — they synthesize the OAuth identity into the request context, so
they prove the handler's behaviour GIVEN an identity and have no opinion
about which routes supply one (CONVE-19). Codex round 2 reached the same
conclusion independently.
* fix(server): correct five overstated claims from Codex round 3 (IDEA-2756)
All five were mine, all P2, none changing the gate's behaviour — four are
claims that were broader than the code, one is a test that proved less
than its name.
1. "Only re-authorization lifts it" was wrong in five places (version.go,
README, CLAUDE.md, the MCP catalog description, CLI help). A user can
also enable the flag on the EXISTING connection via
PATCH /connected-apps/{id}/flags, which the console page drives —
instructions.md said so and contradicted the others. All five now name
both remedies, and both are still the user's, which is the part that
matters: neither is reachable by the app.
2. "This branch is UNREACHABLE ... it is dead code" on
maybeAutoAddCreatorConnection's flag-off branch was false. The gate
reads the connection and that function reads it AGAIN after creation;
a user revoking creation power from the console between those two
reads lands exactly there. It is a real second check across a real
TOCTOU window, failing in the safe direction. The claim was written
from the call graph, which cannot see a concurrent write between two
reads.
3. handlers_import_bundle.go's "Auth: any authenticated user" was made
false by this change and the concept sweep never had a chance at it —
it greps may_create / auto-add / creation power, and that sentence
contains none of them. Corrected in place.
4. The two NonOAuthCallerUnaffected tests claimed PAT, CLI session and
local stdio; each drives one PAT. The comments now state the fixture's
real scope and why one caller stands for the class (the guard branches
on an identity only MCPBearerAuth sets, so callers that skipped it are
indistinguishable) rather than implying three fixtures.
5. The JSON import refusal leg would have passed with the gate below
decodeJSONWithLimit — only the bundle leg pinned placement, and only
for gzip. Adds TestImportWorkspace_ConsentRefusalPrecedesBodyDecode
(malformed body: 400 if the gate is late, 403 if it is early),
mirroring the create-side ordering legs.
Mutation matrix now 9 mutants, 9 detected. M8 (guard below the JSON
decode) is killed by the bundle leg too, so it shows the new test is
covered rather than necessary; M9 gates the bundle path and moves only
the JSON path's guard, and dies to the new leg ALONE. That is the mutant
that justifies the test.
* docs(server): the second consent check narrows the race, it does not close it (BUG-2792)
Round 3 caught me calling maybeAutoAddCreatorConnection's flag-off
branch dead code. The replacement comment then claimed the branch means
a revoked grant cannot silently gain a workspace — which is more safety
than the code delivers, and round 4 caught that.
The read and the AddConnectionWorkspace insert below it are separate
unconditional statements, so a revocation landing BETWEEN them still
adds the workspace. The check narrows the window; it does not close it.
Filed as BUG-2792 rather than folded in: the race is pre-existing and
unchanged by IDEA-2756, and closing it needs an atomic check-and-insert
at the store layer, written and gated for both dialects — materially
more diff and risk than this handler-level guard.
Both mistakes were the same shape in opposite directions: a claim about
concurrency derived from reading the call graph, which cannot see a
concurrent write between two reads.
* style(server): gofmt the doc comment (IDEA-2756)
gofmt wants blank lines between list items once one item spans multiple
paragraphs, which the BUG-2792 note made true.
My error, and worth naming exactly: I ran build, vet and the targeted
tests on this commit but not lint, because lint had passed on the
PREVIOUS commit and the change was 'only a comment'. The gate has to run
on the tree being pushed, not on an earlier one that resembles it. CI's
golangci-lint is pinned to the same v2.11.4 the Makefile installs, so
there was no version skew to blame — the local gate would have caught
this in 51 seconds.
* docs(server): correct ten overstated prose claims from Codex round 8 (IDEA-2756)
Round 8 reviewed only the prose this change adds. Ten claims were
broader than the code. All ten are mine; none changes behaviour. Rounds
3, 4 and 7 each caught one of these, which is why round 8 was pointed at
the class rather than at a new dimension.
The substantive ones:
- "gates every endpoint that MINTS a workspace" — autoCreateWorkspace
mints from registration, bootstrap and oauth-login and is deliberately
outside this gate. The helper doc and the test header now name the two
callers and the exclusion instead of claiming universality.
- "the agent was handed a workspace it could not then see" (version.go,
README, CLAUDE.md) — only true for a connection with an EXPLICIT
allow-list. An all_current_workspaces=true connection is not gated per
slug and could see what it made. The consent mismatch is the constant;
the invisibility was its most visible symptom, not its definition.
- "ErrOAuthConnectionNotFound — a pre-Phase-C grant" asserted a cause the
code cannot know: ANY missing row takes that branch. Now stated as the
expected cause, with the limit of what the code can tell.
- "above the 64 MiB body read" conflated the two import paths. 64 MiB is
the JSON decode's bound; the bundle path has its own, much larger. The
gate precedes both, which is the property that actually matters.
- "the request context is decorated AFTER TokenAuth runs" was false, and
inherited verbatim from the sibling helper this was modelled on
(handlers_oauth_claim_test.go's doClaim), where it is also false. The
wrapper sets the identity BEFORE ServeHTTP; it survives because
nothing on the /api/v1 chain writes that key.
- "lets CreateWorkspace normalize it" — CreateWorkspace slugifies only
when the supplied slug is EMPTY, and import supplies a non-empty one,
so an imported workspace keeps the ?name= value verbatim.
- "The PAT needs a workspace to bind to" — CreateAPIToken takes
WorkspaceID as optional.
And one where the first fix was worse than the finding:
- "Every refusal leg asserts no workspace exists afterwards" was false —
the two ordering legs assert status only. My first correction ADDED
those assertions, which is the trap the finding was pointing at: a
malformed body and an empty name are rejected before creation under
every guard placement, so "no such workspace exists" is true of broken
and working code alike. Reverted; the header now states which legs
carry the counterfactual, and why the ordering legs discriminate on
status instead.
Gates re-run on the tree being pushed, not an earlier one: gofmt clean,
lint 0 issues, internal/server and internal/mcp green, mutation matrix
still 9/9.
* ci: re-trigger CI after a GitHub startup_failure (IDEA-2756)
No code change. The Go job on
|
||
|
xarmian
|
6b5e8be04e |
fix(server): a structured timeline id must not collide with a row id (BUG-2783) (#1210)
* fix(server): a structured timeline id must not collide with a row id (BUG-2783)
A note or decision id comes from the item's fields blob, which nothing
validates on write. It can equal the id of a comment, activity or version
on the SAME item, and then two entries in one timeline payload share an
id — which the client's keyed {#each} and its by-id page dedupe resolve by
hiding one of them. The intra-structured dedupe already prevented exactly
this failure one source-boundary in; `usedIDs` simply never knew about the
other four sources.
Fixed by seeding `usedIDs` from the ids the SQL sources already own. All
three slices are in hand at the call site one line above, so this needs no
extra query and no re-ordering.
The structured side is the one that yields, and that is a decision rather
than a side effect: comment, activity and version ids are real primary keys
AND are what the client sends back as `before_id`, so moving one would break
paging. The unvalidated side moves.
Tests drive the MERGED payload through the real endpoint rather than
structuredTimelineEntries, which is the reason the gap survived: the
builder is correct in isolation, so a unit test of it vouches for the
component and not its binding to the other sources.
Two things the assertions are deliberately specific about. They name WHICH
entry moves — "two entries are present" passes trivially even unfixed,
since the collision is resolved by the client rather than the server, and
"the ids differ" would pass if the comment had been the one to yield, which
is a different bug. And the activity case asserts the row that owned the id
is still in the payload, since the structured entry yielding is only correct
if the thing it collided with survives.
Also closes a pre-existing coverage gap this work surfaced: a mutation
deleting the `for usedIDs[id]` fallback-uniqueness loop survived the entire
timeline suite. It is now covered — and the fixture took two attempts, which
is the part worth recording. A note with no id followed by a note claiming
"note-idx-0" does NOT reach that loop: the second note's raw id is already
used, so the `!usedIDs[raw]` guard diverts it to the fallback branch and it
gets a free name. That fixture passed with the loop deleted. Reaching the
loop requires the fallback NAME to be occupied when the fallback is
computed — note 0 claiming the literal "note-idx-1", note 1 having no id.
Mutation matrix, all four compiling: dropping the seeding entirely, seeding
from comments only, seeding from activities/versions only, and dropping the
fallback loop are each detected, each by the test aimed at it.
Scope note recorded in the code: the fallback half is defence rather than a
live vector across sources, because all three SQL sources mint ids with
store.newID() — including the import path, which re-mints rather than
preserving an artifact's ids — and a UUID cannot equal `note-idx-N`. Within
the structured kinds, where both ids come from the same unvalidated blob, it
is reachable.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(server): make the structured id independent of the page, not of the window (BUG-2783)
Codex round 1, P1, and it refutes the approach the recon proposed and the
lead ratified — reported rather than quietly redesigned, and replaced here
because the defect was mine and introduced by the previous commit.
Seeding the dedupe map from the ids fetched FOR THIS PAGE makes a structured
entry's id depend on the page window. The three SQL windows are cursor-
dependent, so the same note takes its raw id on a page where the colliding
row is absent and a positional id on one where it is present. That id is not
merely a render key: it is the SECOND TERM OF THE CURSOR PREDICATE, so a
window-dependent id makes an entry's own sort position depend on which page
is being built, and paging can then skip or repeat it. Worse than the bug it
was fixing, and a regression I introduced — before the previous commit,
structured ids were a function of the item alone.
Replaced with a SHAPE test: a raw blob id that is UUID-shaped is refused and
takes the positional fallback. That is sound because the only ids it can
collide with are comment, activity and version rows, and every one of those
takes its id from store.newID() — uuid.New().String(). Enumerated rather
than sampled: all six INSERT sites into those three tables pass newID(), and
the import path re-mints rather than carrying an artifact's ids across. So a
blob id can only equal a row id by being a UUID.
The result needs no extra query, consults nothing outside the item, and
changes ids only for blob ids that are well-formed UUIDs — which is exactly
the colliding case. A non-colliding UUID-shaped blob id also loses its raw
id; accepted, since it cannot be distinguished from a colliding one without
consulting the very rows this must not depend on.
Four more findings from the same round, all fixed:
- No version test. A fix knowing only about comments and activities passed
the whole file — which is what the seeding version did, since it
enumerated sources by hand. Added.
- The activity test called t.Skip when it found no activity. Creating an
item writes an activity row, so absence means the fixture stopped building
what the test needs, and a skip reports that as success forever. Now
t.Fatalf. Same for two skips in the new window test.
- The comments described the client as hiding one of a colliding pair. Half
true: WITHIN a page it is a keyed-each duplicate key, which is a Svelte
ERROR; the silent drop is the ACROSS-pages append filter. Both stated.
- "The other four sources" — there are three SQL-backed ones.
Mutation matrix, 5, all compiling: dropping the shape test, inverting it,
making the helper never fire, dropping the fallback loop, and dropping the
intra-structured dedupe are each detected.
The last one first looked like a survivor. It was not — it dies on
TestItemTimeline_StructuredDuplicateIDsAreDisambiguated, whose name my
`-run TestTimeline` filter never matched. A `-run` filter narrower than the
population is the same false-green family as a mutant killed by the
compiler: the harness reports SURVIVED and the reason is the instrument.
Re-run unfiltered before believing any survivor.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(server): a diverted id must not inherit an unstable one (BUG-2783)
Codex round 2. Both findings are about the REPLACEMENT, not the original
bug, and the first is a hazard my own fix widened.
P1: the positional fallback encodes the entry's array index, so inserting
or removing an entry ahead of another renumbers it — and the entry id is
the cursor's tie-breaker, so a renumbered entry can be skipped or re-shown
across a page boundary. That instability PREDATES this work and applies to
entries with an absent or duplicate id. But the shape test diverted every
UUID-shaped raw id onto that same path, which would have taken a small
population and made it a much larger one, for no benefit.
Fixed by giving a shape-refused id a DERIVED id, `<prefix>:<raw>`, instead
of a positional one. It depends only on the entry's own raw id, so it is
stable under any mutation of the blob, and the prefix is what makes it safe:
a row id is a bare UUID, so `note:<uuid>` cannot equal one. It still passes
through the duplicate guard, because a blob may contain a literal
`note:<uuid>` string of its own.
What stays on the positional path is exactly the population that has
nothing else to derive from — absent ids and duplicates. That residue is
filed as BUG-2788 with the two consequences spelled out and three candidate
fixes, rather than being quietly absorbed here.
The new test asserts the property the derived id exists for: the same entry
keeps the same id after an unrelated entry is inserted BEFORE it. That
holds for a derived id and cannot hold for a positional one — the mutant
that removes the divert fails exactly this test and nothing else.
P2: the UUID-shape premise is a property of the WRITERS, not the schema.
All three tables are `id TEXT PRIMARY KEY` with no format constraint and
migrations carry ids verbatim, so a row whose id is not a UUID — from a
future path bypassing newID(), or already present in a database this code
has never seen — would be outside what this refuses. Latent rather than
reachable through any current write or import path. Now stated in the code,
including that the enforcement which would close it belongs at the writers
or the schema and not here, since detecting it needs exactly the row lookup
this design exists to avoid.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(server): close the derived namespace, and pin its contract (BUG-2783)
Codex round 3. Both findings are about the round-2 replacement.
P1, and it falsifies a scoping sentence I had just written on BUG-2788: a
blob holding BOTH `note:<uuid>` and `<uuid>` had the second derive onto the
first's KEPT id, fail the duplicate guard, and fall through to the
positional path. Two distinct, legitimate raw ids, one of them made
unstable — and not covered by the absent/duplicate exception BUG-2788
describes, so that item's scope was wrong within the hour of being filed.
The fix generalises rather than patching the case. Divert any raw id that
could be confused with a derived one — UUID-shaped OR already beginning
with `<prefix>:` — and derive by prepending. Two rules then hold together:
a kept id never begins with the prefix, a derived id always does. So the
namespaces are disjoint BY CONSTRUCTION instead of by luck, two derived ids
are equal only when their raws are equal (the duplicate case), and a bare
UUID row id can equal neither form. BUG-2788's residue is unchanged —
absent and duplicate ids, the population with nothing of their own to
derive from — but now for a sound reason; corrected on its trail.
P2: the tests could not tell this fix from its alternatives. "Stable and
not the raw id" is satisfied by a constant; the collision tests' inequality
checks are satisfied by the very positional fallback this replaced. Two new
tests assert the contract instead of a symptom — the exact `<prefix>:<raw>`
form, distinct raws deriving to distinct ids, a prefix-carrying raw id
deriving to `note:note:<uuid>`, and no entry with a usable raw id landing
on `-idx-`.
The second new test closes a loop this design owed: a derived id is
SERVER-MINTED and the client sends it back as `before_id`, which BUG-2774
taught the server to refuse when the database would. Paging from a derived
cursor is now asserted to be accepted — a new id format must not be
refusable by our own validation.
Matrix now 9, all compiling: the three from this round are stop diverting
the prefix form (dies only on the contract test, confirming the hole was
real and is now covered), derive from a constant, and drop the divert
entirely.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(server): reserve BOTH structured prefixes, not the calling one (BUG-2783)
Codex round 4, and it is the round-3 hole one kind across — which is the
part worth recording, because I fixed round 3's case and called the
namespace "disjoint by construction" while the construction only held
within a single kind.
Notes and decisions are numbered by separate entryID calls but share one
`usedIDs` map. Checking `prefix+":"` therefore checked only the CALLING
kind: a note whose raw id is `decision:<uuid>` was kept, and a decision
whose raw id is `<uuid>` then derived onto exactly that string, failed the
duplicate guard, and landed on the index-dependent positional path. Same
defect, same consequence, one substitution away from the case I had just
closed.
Fixed by reserving both prefixes globally — `hasStructuredPrefix` tests the
raw id against every namespace entryID mints in, regardless of which kind is
being numbered. The invariant is now what I claimed last round: a kept id
begins with no structured prefix, a derived id begins with one, so the two
sets cannot intersect.
The new test asserts forms rather than inequality, and then mutates the
notes array to show the DECISION's id does not move — which it could not
survive if the note had pushed it onto the positional path.
Matrix now 12, all compiling. The three added here: check only the calling
kind's prefix (dies on the cross-kind test), and dropping either "note" or
"decision" from the reserved list — each dies on a different test, so the
list is not covered by one case standing in for the other.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* docs(server): write down what the id scheme closes and what it does not (BUG-2783)
Codex round 5 answered the question I asked it — enumerate every way two
entries can share an id or one entry's id can change without the entry
changing — and returned no new defect. All three residuals were already
known: the positional-fallback instability is BUG-2788, and the two row-id
gaps are the latent writers-not-schema class already documented in
looksLikeRowID.
Recording the enumeration itself, because the next reader should not have
to re-derive it and because "everything else is handled" is exactly the
kind of claim that rots silently. Six closed cases named with the mechanism
that closes each; two open ones named with where to read about them.
One consequence from that round WAS new, and it is on BUG-2788: removing
the FIRST of two entries sharing a raw id changes the survivor's id KIND,
not merely its number — while both exist the second is pushed onto the
positional fallback by the duplicate guard, and once the first is gone the
survivor keeps its own raw id. Same skip-or-repeat consequence, reached
without any index moving. It also rules out that item's option (2):
deriving from content does not help, because two entries with the same raw
id still need a tie-break. Only persisting an id at write time makes an
entry's identity independent of what its siblings do.
Also extended looksLikeRowID's scope note with the second face of the same
gap: the three SQL sources keep their ids verbatim and are not deduped
against each other, so two rows in different tables sharing an id would
collide with no structured entry involved.
Findings by round: 5, 2, 2, 1, 0-new. No behaviour change in this commit.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
|
||
|
xarmian
|
fab38336f4 |
fix(store): never return an activity id alongside an error (BUG-2779) (#1209)
CreateActivity returned the id of the row it had just FAILED to insert. CreateActivityDebounced returned the id of the row it had CHOSEN but not written. The item-update handler discarded the error and linked a user's comment to whatever came back, so the comment appeared under an activity entry describing a different change — attributed, by TASK-2760's agent-name rule, to whoever wrote that entry. The request answered 200. The contract is now: a non-nil error is always paired with an empty id. Three error returns changed and the function's doc states the rule, so no future caller can misuse a value that looks usable. The item-update handler CHECKS the error and logs it. It deliberately does NOT re-zero the id: with the contract fixed that would be a second mechanism for a window the first already covers, it would hide which one is load-bearing, and it would keep passing if the contract regressed. The log is the part nothing else provides. The population the filing left open: logActivityWithMetaReturningID has exactly three callers, and the two comment handlers ALREADY guarded correctly. This brings the third in line with its siblings rather than inventing a pattern — which is why the caller-side half is one line. TESTS, and two instrument failures found on the way: - The first fixture reached a different branch. Closing the database made the CANDIDATE READ fail, so the call fell back to CreateActivity and the merge's error return never executed — the assertion passed for a reason unrelated to the line under test, and the mutation restoring `return existingID, err` survived. BUG-2770's afterDebounceRead seam fires between the read and the write, which is the window that matters. - "No instrument can reach it", written on the classifier's error return, was a fact about the seams I had written rather than about the code. Review pointed at the seam I had not written; afterDebounceRefusal fires between the zero-row refusal and the probe, and that mutation now dies. 5 mutations aimed, 4 die. The survivor is the handler's log line, kept and documented in place: asserting it needs an activity write that fails while the item update succeeds, and a seam existing only so a test can read a log line is not worth a field on Server. Also corrected: a comment in the comment handler that said CreateActivity returns an id on insert failure. True when written, falsified by this change, and missed by my own sweep because I grepped the mechanism's name instead of the claim. |
||
|
xarmian
|
e4e914d399 |
fix(server): reject path segments the database cannot be asked about (BUG-2782) (#1207)
* fix(server): reject path segments the database cannot be asked about (BUG-2782) Every handler that resolves a workspace, collection, item, comment or attachment from a URL path segment passes that segment to the store verbatim, and the store binds it into a text comparison. Postgres refuses a text parameter that is not valid UTF-8 or that contains a NUL (SQLSTATE 22021 / 22P05); the driver surfaces that as a query error and the handler answers 500. SQLite accepts both bytes and matches nothing, so the same request is a clean 404 there — a dialect divergence that leaves the defect invisible to self-hosted installs and live on Pad Cloud. Measured before the fix, driving every route that carries a path parameter with one segment set to "bad-%FF-x" (247 probes, one per parameter position per method, real values elsewhere): Postgres answered 500 to 191 of them, SQLite to 0. After: 0 and 0, all 247 answered 400. Fixed with one root-level middleware rather than at ~112 chi.URLParam call sites, because this is a transport-level input rule and per-call-site fixes rely on every future route remembering. ValidatePath rejects a request whose percent-DECODED path is not valid UTF-8 or contains a NUL, before routing. It validates r.URL.Path rather than what chi hands the handler. chi routes on RawPath when non-empty and Path otherwise, and Go populates RawPath only when the client's escaping is not already canonical — Go escapes 0xff as uppercase "%FF", so the CANONICAL form any ordinary client emits is exactly the one that reaches the store decoded, and the lowercase "%ff" oddity is the harmless one. Validating the decoded path answers both identically and does not depend on chi continuing to prefer RawPath. It cannot refuse Pad's own URLs: store.slugify emits only [a-z0-9-], ids are UUIDs or hex, refs are a prefix plus digits. Valid non-ASCII segments pass through untouched — the database accepts them and they may legitimately name something. 400 rather than 404 because the request is malformed as a URI and the answer does not depend on whether anything exists, so it is not an existence oracle. Scope is the path; the query string is validated at its points of use, per BUG-2774's validCursorID. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(server): the invalid-path rejection must look like every other API error (BUG-2782) Codex round 1, verified before acting on: ValidatePath runs on the root router, so its rejection short-circuits ABOVE the /api/v1 group's cors.Handler and jsonContentType and inherited neither. Measured — the 400 carried a JSON body sniffed as text/plain and no CORS headers at all, while a normal 404 on the same route carried Content-Type: application/json plus the full CORS set. On a cross-origin deployment (PAD_CORS_ORIGINS set) the browser refuses to let the page read a response with no Access-Control- Allow-Origin, so a debuggable 400 arrives as an opaque network error. Fixed without duplicating the CORS configuration: the group's cors.Handler is hoisted into one shared instance, the group mounts it as before, and ValidatePath serves its rejection THROUGH the same instance. Content-Type is set explicitly, since jsonContentType is mounted below and never runs for a rejection. Moving ValidatePath down into the group instead was rejected: two covered routes live outside it — the SPA catch-all and /api/v1/collab/{itemID} — and the mutant that makes that move is caught by exactly those two subtests. A genuine preflight (Origin + Access-Control-Request-Method) to an invalid path is answered 200 by the shared handler, the same as for any other path: a preflight asks whether the method and headers are permitted, not whether the resource exists. The real request that follows still gets the 400, and can now be read. Asserted rather than described. The new test compares each header on the 400 against the SAME route answered normally, for an allowed origin AND a disallowed one, so it pins parity with the API's own errors rather than a header list copied from a spec — and the disallowed-origin leg is what would fail if the rejection echoed origins the shared handler refuses. Mutation matrix, all nine verified to COMPILE first: dropping the CORS decoration and dropping the explicit Content-Type are each detected, and only by this new test. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * test(server): pin the ordering decision the path check makes (BUG-2782) Codex round 2, angle rotated to middleware contracts: a rejected request never reaches TokenAuth, SessionAuth, RateLimit or CSRFProtect, because ValidatePath sits on the root router above that group. The finding is factually right and the ordering is deliberate, but nothing in the diff said so and no test held it — which is the same defect shape as an undocumented invariant: true today, unenforced tomorrow. Verified rather than argued, because "bypasses the rate limiter" reads as a weakening and here the direction is inverted. Before this middleware, the same request ran SessionAuth — a store.ValidateSession round trip — then the limiter, then a handler whose query the database refused, and answered 500. It now costs a UTF-8 scan and a short JSON write with no database contact, so the unmetered path is strictly cheaper than every path the limiter protects. The answer is also constant for all inputs of this shape, independent of auth and of existence, so a flood learns nothing. And the limiter is a plain token bucket per key — no escalating ban, no durable block — so skipping it defeats no state that outlives the request. The alternative, metering it inside the /api/v1 group, trades this for a real coverage hole: the SPA catch-all and /api/v1/collab/{itemID} are mounted outside that group. The test floods 80 invalid paths from one IP (burst is 60), requires all 80 to be 400 and none 429, then requires a VALID request from the same IP to still get the resolver's 404 — proving the budget was untouched. It then asserts its own premise: the same volume of valid requests from a second IP must actually hit the limiter, because an inert limiter would produce an identical reading for the first half. Both mutants land where they should: metering the rejection fails at request 61 (burst 60 + 1, which independently confirms the constant cited above), and disabling the limiter fails the premise check rather than passing quietly. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * docs(server): correct four claims this branch's own measurements refute (BUG-2782) Codex round 3, angle rotated onto prose accuracy. No behaviour changes — every finding is a sentence that was stronger than what was verified, and in two cases stronger than data already sitting in this branch. 1. "It rejects exactly what the DATABASE rejects." Too strong, and inherited verbatim from validCursorID. Postgres refuses these two classes under a UTF8 database encoding; SQL_ASCII accepts the same bytes, and SQLite's sqlite3_bind_text accepts arbitrary sequences with NUL undefined rather than erroring. Pad neither creates nor configures that database — nothing issues CREATE DATABASE or sets client_encoding — so the encoding is the operator's. Now stated as what it is: the strictest reading, applied uniformly so the two backends stop disagreeing about the same request, measured against postgres:17-alpine at its defaults. A first draft of this correction replaced the overstatement with a NEW unverified claim ("the encoding Pad's migrations create"). Grepping for CREATE DATABASE found it only in test helpers. Fixing an unchecked sentence with another unchecked sentence is the same defect wearing the repair's clothes. 2. "Against unfixed code these are 500 on Postgres and 404 on SQLite." False for 56 of the 247 pre-fix probes, and my own sweep output said so — routes whose authorization or configuration gate answers before any store call (admin user lookup; attachments with no storage configured). Replaced with the pasted distribution: 500:191 404:34 403:12 401:4 503:4 400:2. 3. "Passed through untouched" oversold what this middleware guarantees. It does not touch a valid path, but chi still hands the handler the ESCAPED text whenever RawPath is populated: "caf%C3%A9" arrives as "café", the non-canonical "caf%c3%a9" arrives literally, and "%2F" never becomes a separator. Pre-existing chi behaviour, unaffected by this change, written down because the obvious reading is stronger than the truth. 4. "The request is malformed as a URI." It is not — "%FF" and "%00" are syntactically valid percent-encoded octets. The 400 is because the DECODED value cannot be a resource identifier here, which is the actual reason and a different one. Also reconciled the two probe counts that appear in this branch's history (111/94 GET-only, 247/191 all methods) so a reader meeting both does not have to guess which is wrong; they are one sweep at two widths. CONVE-23 sweep: finding 1 falsifies the same sentence in validCursorID (handlers_timeline.go, BUG-2774), which is where this branch inherited it. Corrected there too rather than left standing — the rule that comment describes is unchanged and still right; only its claim about the database was wrong. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * test(mcp): drive the in-process transport seam the path check walks past (BUG-2782) Codex round 4's exploration pointed at the door I had asserted rather than driven: the remote /mcp transport does not reach the server over a socket. HTTPHandlerDispatcher SYNTHESIZES an *http.Request and calls Handler.ServeHTTP in-process, so "the middleware covers every route" was a claim about a path this transport bypasses on its face. Driven, it is covered — and for a chain nothing in the tree stated: Handler is the *server.Server, chi's Mux.ServeHTTP runs mx.handler (middlewares + routeHTTP) on BOTH branches, and buildAuthedRequest forces the fresh-routing branch with a typed-nil RouteCtxKey. Every link is load-bearing and none was written down; this test is what notices if one changes. The counterfactual was worth more than the confirmation. Unfixed, an MCP agent that put an invalid byte in a ref got upstream_error on Postgres — whose hint says the failure is "usually transient, retry" — for an input that can never succeed. An agent obeying that hint retries forever. That is the retry-hostile misclassification family BUG-2675 added a code for, and this change removes an instance of it that nobody had noticed. Now validation_failed: the agent is told its INPUT is wrong. The first version of this test named upstream_error in its comment while running on SQLite, where unfixed gives item_not_found instead — an assertion that would have failed for a reason other than the one it named. The comment now states both backends separately and the fixture takes Postgres when PAD_TEST_POSTGRES_URL is set, so under make test-pg the dangerous half is what actually runs. Control leg included: a valid-but-absent ref must still return item_not_found, or a dispatcher that refused every ref would pass. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * chore(mcp): remove a throwaway probe that was committed by accident (BUG-2782) The probe that established the MCP seam behaviour was meant to be deleted once dispatch_http_invalid_path_test.go replaced it. The 'rm' was written as the first half of a compound command whose second half the tool layer REJECTED, so the whole command never executed — and a later 'git add -A' swept the file in. It duplicates the real test with printf-style output and no assertions. The rule this breaks is one I already hold: verify the mutation, not the report of it. I read 'rm -f X && cat > Y' as having removed X because I wrote it, when the command never ran at all. A rejected command and a successful one look identical in a transcript if you do not look. Caught by a Codex file listing showing an A for a file I believed gone. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * test: five corrections from the final review pass (BUG-2782) Codex round 5, judging the whole change. All five are mine; none needed a behaviour change. **A flake I built in.** The rate-limit test flooded 80 requests against a bucket of burst 60 — but a token bucket REFILLS while the loop runs, at 10/s here, so 20 tokens of headroom is 2 seconds of tolerance and a slow or -race'd run would admit all 80 and fail spuriously. The margin that matters is not flood-vs-burst but how long the loop must take for refill to cover the excess. At 400 requests that is (400-60)/10 = 34 seconds against in-process calls measured in microseconds: four orders of magnitude. The constant now carries that derivation, including the rate and burst it depends on. Both mutants still land, and metering the rejection still fails at request 61 — burst 60 + 1, unchanged by the larger flood. **A claim about MCP that JSON does not support.** The seam test's comment said an agent could put "a raw invalid byte" into a ref. Measured with encoding/json instead of assumed: raw 0xff / lone surrogate / truncated sequence → U+FFFD, valid UTF-8 raw 0x00 → JSON parse error the u0000 ESCAPE → a real NUL So exactly one of the five cases is reachable end to end over a JSON transport, and it is the one only the NUL half of validPathText refuses. The raw-byte cases stay — Dispatch is a Go API and the JSON decode is upstream of that boundary, so they assert the seam holds for callers that do not launder their strings through encoding/json — but the comment no longer offers them as evidence a JSON client can send them. **Two prose overstatements the earlier sweep missed.** The control test still said the rule rejects "only what the database rejects", which the previous commit had already established is false in the permissive direction. And TestValidatePathPostgresNoInternalError was described as reproducing the original 500 when it runs the FIXED server and can only ever observe a 400; the 500 lives in the counterfactual sweep and in the mutation matrix, and a test cannot both apply a fix and witness the bug. **One dead construction**, plus a smaller instance of the same habit: the MCP fixture built a SQLite store and discarded it in Postgres mode. My first attempt replaced the comment with one claiming the branch had been hoisted, and left the code as it was — writing the fix into the prose instead of the code, in the same hour I committed a message about not doing exactly that. Now actually branched. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * docs+test: seven more corrections, and one the emoji route earns (BUG-2782) Codex round 6, reading the three files as they now stand. All prose or assertion strength; one of them changes what the tests cover. **Pad DOES emit a non-ASCII path segment, and I said it never does.** `DELETE /workspaces/{ws}/comments/{commentID}/reactions/{emoji}` — the web client sends `encodeURIComponent(emoji)`. So the justification I gave for "it cannot refuse Pad's own URLs" was false in its premise while true in its conclusion, which is the worst combination: a reader checking the premise finds a counterexample and has no reason to trust the rest. It is also the best possible illustration of why the rule permits valid non-ASCII, so the control test now drives that ACTUAL route with a real emoji rather than relying on an emoji-shaped item slug — the claim is true by construction instead of by careful wording. A new mutant confirms the leg discriminates: a rule that rejects all non-ASCII (the plausible wrong version, not the absurd one) is caught there. **"The handler answers 500" was universal and is not.** Handlers that collapse a resolution error into not-found already answer 404 — the timeline handler's `err != nil || item == nil` is the example. My own measured distribution said so; the sentence did not. **"Self-hosted installs never see it" was wrong about the axis.** The split is by BACKEND, not deployment: a SQLite install never sees it, any Postgres install does — Pad Cloud and a self-hoster on Postgres alike. **A stale cross-reference of my own making.** The previous commit corrected TestValidatePathPostgresNoInternalError's claim to reproduce the 500, and left the sentence POINTING at it still saying it does. Fixing a claim at one site and leaving its pointer false is the CONVE-23 case in miniature. **The MCP test asserted too little.** validation_failed is how the dispatcher classifies ANY 400, so the test could have passed on a mapper-level refusal without ValidatePath running at all. It now pins the middleware's own message, which rides through on the hint. **Two overstatements in the same file.** "Exactly one case is reachable" should be one input CLASS (two cases carry a NUL). And the raw-byte cases do not cover "the stdio path": local stdio MCP is ExecDispatcher, which shells out to the binary and never touches this in-process door. Scope now says HTTPHandlerDispatcher and says what it does not speak for. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * docs(server): two qualifications the file already owed itself (BUG-2782) Codex round 7. Two, both narrow — the review is converging (7 findings last round, 2 this one), and both are internal inconsistencies rather than new ground. "Any Postgres install does" contradicted a qualification made forty lines lower in the same file, where validPathText spells out that SQL_ASCII Postgres accepts these bytes. Now says a Postgres install whose database encoding is UTF8, notes that this is initdb's default, and points at the place the qualification lives so the two cannot drift apart again. validCursorID's paragraph still described the 500 in the present tense, though BUG-2774 fixed it — it is the behaviour the guard PREVENTS, not what the endpoint does. My first attempt at this appended "past tense throughout this paragraph" and left the following sentence in the present tense, which is annotating a problem instead of fixing it. Rewritten so the tense carries the meaning without a note telling the reader to read it differently. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * docs(server): the SQLite half of the claim needed the same narrowing (BUG-2782) Codex round 8, one P3 and it is the mirror image of round 6's. I qualified "the handler answers 500" on the Postgres side and left the symmetric sentence — "the same request is a clean 404 there" — universal on the SQLite side, in the same paragraph. Not every request reaches a store resolution on either backend; a gate that answers first keeps its own status, and my own GET-only sweep recorded 102 x 404 alongside 5 x 403, 2 x 200, 1 x 401 and 1 x 503 on SQLite. Fixing one direction of a symmetric claim and leaving the other is a shape I have hit before and evidently do not catch by intention. The paragraph now says the divergence is in what happens once a value REACHES the store, which is the true and symmetric statement, with the distribution pasted. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
773222368c |
fix(store): serialize document renames and stop reading the pool inside transactions (BUG-2778) (#1208)
Two deadlocks, one in the database and one in the application. THE DATABASE ONE, which is what BUG-2778 was filed about. A document rename takes row locks in two stages inside one transaction: updateLinksInTx writes every OTHER document whose content links the old title, then the final UPDATE writes THIS document. Two concurrent renames of documents that link to each other therefore take the same two locks in opposite orders, and Postgres aborts one with SQLSTATE 40P01 — a 500 on an ordinary rename. A throwaway probe against the unfixed code deadlocked on 12 of 12 rounds; this is deterministic, not theoretical. The fix serializes renames per workspace with a dedicated advisory key (`pad:document-rename:<ws>`), taken before any row lock and whenever a title is supplied. No-op on SQLite, whose single writer cannot produce the cycle. `SET LOCAL lock_timeout = '5s'` bounds the wait, because a transaction that waits with a pool connection already in hand converts contention into pool exhaustion; the handler maps 55P03 and 40P01 to a retryable 503 rather than a generic 500. WHY NOT `ORDER BY id`, which is what I proposed when I FILED this from reading rather than from a repro: each transaction's cascade set is a single row, and the cycle is cascade-then-self, so ordering the cascade leaves it exactly as reachable. Run as a mutation, that fix fails the regression test. Reproducing the bug is what refuted my own diagnosis. THE APPLICATION ONE, found in review and larger than the filed bug. Seven production paths issued a read through the connection POOL from inside an open transaction. Under a saturated pool the second connection never arrives, so the transaction cannot finish and never releases what it holds — no SQLSTATE names this and no lock timeout breaks it. All seven now take their executor: the document and item slug scans, both version checks, the done-field lookup on the item update and move paths, the open-children guard's collection read, and the OAuth startup backfill's workspace lookup. Three of those were found only after asking for the true population rather than for a sample; two were INDIRECT (through GetCollection), which a grep for `s.db` inside transaction bodies cannot see. The instrument is a one-connection pool, which makes the hazard deterministic instead of load-dependent. ALSO FIXED, adjacent and found by the same reviews: - The rename decided from a PRE-LOCK snapshot: the lock made the cascade safe against another rename and then handed it a stale OLD TITLE. It now re-reads under the lock and decides from that row. - A concurrent soft-delete could commit mid-rename, leaving the cascade's rewrites behind while the caller was told not-found. The final UPDATE now carries `deleted_at IS NULL` with a checked row count, so the rename and its cascade land together or not at all. - Class sweep (CONVE-18) of the same unordered-scan-then-per-row-update shape: the attachment remap (items and comments) and the outbox user-ref scrub are now ordered. Scoped claim — it orders those per-row updates, not every lock in those transactions — and NOT reproduced, unlike the rename. 18 mutations aimed, 15 die. The three survivors are written down where they live: one is an equivalent mutant (the cascade is the first row-locking stage, so "lock before the cascade" and "lock at the top of the transaction" are the same order), one is a guard covering a window no seam can currently schedule, and one is a class-sweep fix with no repro. |
||
|
xarmian
|
effea01666 |
fix(server): diff the activity change list against the store's pre-image (BUG-2776) (#1206)
handleUpdateItem built the activity's human-readable change list by diffing the item it read at the TOP of the request — before the permission checks, before the store's locks — against the row the store wrote. Anything a concurrent writer committed inside that window appeared in the difference and was stamped, with this request's actor and agent name, onto whoever sent the PATCH. Agent A sets status, agent B sets priority, and B's entry reads "status: open → done; priority: low → high" over B's name. Unlike BUG-2770, where a change went MISSING, here the timeline gains a confident false statement about who did what — and BUG-2770's debounce then merges that statement forward into the coalesced row, so it outlives the request that invented it. The fix is a carrier, not new machinery. The store has re-read the row under its own locks since TASK-2533, unconditionally, and diffs THAT snapshot for its status and assignment signals; the handler simply had no way to reach it. models.Item.PreUpdate hands the same snapshot back on the returned item — `json:"-"`, transient, populated only by the update path, following the LastMutation precedent. The title, role and assignment arms move onto it too, which makes the whole list committed-vs-committed: what this transaction wrote over, versus what it wrote. A missing pre-image DROPS the change list rather than falling back to the handler's stale read. The fallback is the defect wearing a warning: an entry that says nothing is recoverable, one that names the wrong author is not. The activity row is still written; a slog.Warn names the invariant. Also fixed, same lines: the title arm only recorded a rename when the field diff had produced nothing, so a PATCH that renamed AND edited a field silently dropped the rename. Two test seams are added (Server.afterItemPreRead, Store.afterItemPreLockRead), both nil in production and both documenting their reentrancy requirement. The second exists because of a mutation that SURVIVED the first matrix: with the rival's write landing before the store call, a pre-image taken from the store's pre-lock read is indistinguishable from one taken under the lock — the instrument could not see the difference the fix is about. That mutation now dies. Seven mutations aimed, six die; the seventh (aliasing the pre-image instead of copying it) survives by design and says so in the code. The Postgres leg earned its keep again: two store assertions compared `fields` blobs byte-wise, which passes on SQLite (TEXT, exact bytes) and fails on Postgres (JSONB, re-serialised) while proving nothing either way about which snapshot the blob came from. They compare by value now. |
||
|
xarmian
|
f465d4c51e |
fix(server): a malformed before_id is a 400, not a 500 (BUG-2774) (#1205)
* fix(server): a malformed before_id is a 400, not a 500 (BUG-2774) before_id went from the query string into the cursor predicate unchecked. Postgres refuses a text parameter that is not valid UTF-8 or that carries a NUL (SQLSTATE 22021/22P05), so the store call errored and the handler answered 500 — the server announcing its own failure for a client's bad input, and SQLSTATE noise in the logs for an input problem. SQLite accepts the same bytes and matches nothing, so the identical request was a 200 there; the failure mode diverged by dialect from one line of unvalidated input. validCursorID rejects exactly what the DATABASE rejects rather than what an id should look like, and deliberately carries NO length or format bound: the structured kinds' ids come from the item's own fields blob, nothing validates them on write, and an imported artifact may carry any string — so a cap could only ever fire on a legitimate cursor, while the cost of an over-long one is a single indexed comparison against a parameter the URL length limit already bounds. Its comment says so, because the obvious review question is why there isn't one. Tests: three malformed shapes reachable from a plain URL (%FF, an embedded NUL, invalid bytes mid-string) plus three controls — a UUID, a structured note id, and a long non-ASCII id — without which "reject every before_id" would pass and paging would be dead rather than honest. A store-level test pins the PREMISE where it is real: on Postgres the query itself fails, on SQLite it succeeds and returns nothing, and both halves are asserted in the one place that sees both, each with a message saying what it means if the backend's behaviour has moved. Verified on Postgres 17 (private container, not the shared port). The handler legs fail with the validation removed. Refs: BUG-2774 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(server): never emit a cursor the handler would refuse (codex round 1) The validation had a second direction I had not closed. A structured entry's id comes from the item's fields blob, which nothing validates on write, so a JSON \u0000 escape arrives as a real NUL on SQLite — Postgres's jsonb refuses it at the door, which is why this is a one-backend hazard. That id became the entry's id, the entry's id became next_before_id, and the client sending it back got a 400 from the validation this unit just added: the server handing out a cursor it then refuses, wedging paging on that item. Such an id now takes the positional fallback that empty and duplicate ids already take — the id has to be usable as a CURSOR, not merely unique. One condition on an existing branch rather than new machinery. Its test uses two notes, one NUL-bearing and one clean, so it distinguishes 'replaced the unusable id' from 'stopped using raw ids at all'. Verified against the unmutated branch: the NUL id is emitted verbatim and the leg fails. Refs: BUG-2774 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * test: assert the envelope, the emitted cursor, and split the dialect premise (codex round 3) Three test weaknesses, all mine. Status-only assertions: a bare 400 or the wrong code would have passed, and clients branch on the code rather than the number. The legs now decode the envelope and require invalid_cursor with a non-empty message. The emitted-cursor test claimed to cover next_before_id and did not: at limit=50 the fixture emits no cursor at all, so it only checked entry ids. A third note and a truncating limit make the NUL-bearing one the LAST kept entry, which is what puts it in the cursor — and the test now asserts the cursor is valid AND that paging with it returns 200. The mutant (emitted ids unchecked) still dies, now on the thing the test is named for. The dialect test is CHARACTERIZATION and now says so: it passes with or without this change, because it describes the backends rather than the handler, and that is its job — it is the premise the 400 rests on. Its 'both halves in one place' claim was also false in an unconfigured run, where NewPostgres skipped the whole test and the SQLite half never ran either. Split into subtests: the SQLite half runs everywhere, the Postgres half skips alone. Refs: BUG-2774 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(server): a one-sided cursor is a 400, not a silent no-op (codex round 4) before_id alone could never match anything: the id is the tie-break AT the cursor instant, and before defaults to now+1m, which no row shares. So it was accepted, ignored, and the caller paged from the beginning believing they had a cursor — the accepted-and-does-nothing shape, which is the worst answer of the three available. The other direction stays supported deliberately, and has a control leg saying so: before alone is the external-client case the "g" sentinel exists for, and rejecting the pair symmetrically would break something the handler goes out of its way to serve. In scope for this unit rather than a separate filing because it is the same parameter and the same class of answer — and because BUG-2765's PR body documented "both fields or neither" as the contract, which until now nothing enforced. Codex's other round-4 finding — a structured entry id can collide with a comment/activity/version id, because the dedupe map holds only structured ids — is real, pre-existing, and a cross-source id decision rather than a validation. Filed as BUG-2783. Refs: BUG-2774, BUG-2783 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * docs: the cursor contract is asymmetric, and now says so (codex round 6) BUG-2765's prose said "both fields, or neither" in the handler doc comment and the TS client. This unit then enforced it in one direction only — deliberately, because `before` alone is the external-client shape the id sentinel exists to serve, while `before_id` alone matches nothing and silently pages from the beginning. The slogan was mine and the asymmetry is mine; leaving both in place would have left a reader to discover the difference from a 400. Both sites now state which one-sided form is accepted, which is refused, and why each. Refs: BUG-2774 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
11f67b0a98 |
fix: timeline can answer has_more=true with zero entries, and the client cannot page past it (BUG-2765) (#1202)
* fix(server): timeline returns the cursor for its next page (BUG-2765) The timeline over-fetches 3x per source and drops rows that cannot render (read/searched actions, empty-metadata updates, activities a version or a comment already stands for, collapsed autosave bursts), so a page can carry fewer entries than the rows it consumed — or none, while has_more is true. The client derived its cursor from the last RENDERED entry, which fails in two ways. With no entries it cannot form a cursor at all, so the first page is a dead end. With a fully-dropped window LATER in the history it re-sends the same cursor forever: nothing is appended, the oldest entry does not move, and paging is wedged at that position permanently. The filing named the first; the second is the one that bites an ordinary item, since a run of read activities anywhere in its history is enough. Both are the same root cause — the response says WHETHER to continue and not WHERE — so the server now returns next_before / next_before_id whenever has_more is true: - page truncated: the last entry KEPT, because the ones cut off must be re-fetched. Unchanged from what the client derived. - window exhausted: the NEWEST tail among the sources that filled their window. A short source has nothing older to come back for and must not drag the cursor forward; resuming at the oldest tail instead would step over a newer source's unexamined rows, and repeats are absorbed by the client's dedup while gaps are not recoverable. Progress is guaranteed because every candidate is a row this page fetched and the store's cursor predicate is strict. Tests: an all-dropped window returns a cursor that reaches the history behind it; paging across a dropped MIDDLE stretch terminates and yields each renderable entry exactly once; and a control leg pins that an untruncated page still resumes at its last rendered entry — without it, "always resume from the oldest row touched" passes while silently skipping what truncation cut. The two-full-source selection rule is pinned as a unit test, because a full source whose rows RENDER puts 3x limit entries on the page and takes the truncation branch instead, so the handler cannot cheaply reach it. Mutation matrix, each independently detected: cursor from rendered entries; oldest tail instead of newest; the full-window flag ignored; no cursor at all. Client half follows in the next commit. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(web): page the timeline with the server's cursor (BUG-2765) Client half. The component derived its next-page cursor from the last RENDERED entry, so a page the server had emptied by dropping rows either gave it nothing to page from (the first page, where loadMore returned early on entries.length === 0) or gave it the SAME cursor it already held (a later page, where nothing was appended and the oldest entry did not move). The second case is a permanent wedge with a live button and a running spinner. It now pages with next_before / next_before_id when the server sends them, and falls back to the last entry otherwise — which is exactly the old behaviour, including its wedge, and is there only for a server that predates the field. One press walks at most MAX_EMPTY_HOPS pages while every row keeps dropping. That bound is UX, not correctness, and its comment says so: a single hop is already correct now that the cursor advances; the loop exists so a user crossing a long run of read activity sees entries appear rather than a spinner and nothing, and it is small so a pathological item cannot turn one click into an unbounded request fan. Tests assert the cursors the component ASKS FOR, not only what it displays — a component that shows the right thing by re-fetching page one forever is the bug. Mutation matrix: ignoring the server cursor fails three of four legs (the fallback leg survives, correctly, since that is the path it pins); a single hop fails the advance leg; an unbounded hop count fails the bound leg. vitest 109 files / 1862 tests pass; vite build clean; svelte-check 0 errors (6 warnings, all pre-existing and in other files). Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix: cursor must clear BOTH bounds, and a later page must merge not append (codex round 1) Two findings, both real, both consequences of the cursor itself. P1 — truncation and an exhausted window are INDEPENDENT bounds, and the second is not implied by the first. A source whose rows all drop contributes nothing to the page, so the truncation cursor can sit older than that source's tail, and every unexamined row between the two falls in a gap neither page fetches. The cursor is now the NEWEST candidate across both reasons. Its regression puts one renderable activity in exactly that gap: two comments forcing truncation at limit=1, three read rows filling the activity window above them, and the row at risk in between. Run against the previous commit it comes back 0 times — the row is not late, it is gone. P2 — a later page can legitimately carry entries NEWER than the oldest one already shown, because the cursor deliberately re-covers ground when one source's window ran out before another's. Concatenating printed those below older entries. The client merges by (created_at desc, id desc) now, comparing INSTANTS rather than strings: precision is not uniform — the store writes whole seconds but a structured note can carry a sub-second timestamp — and lexicographically "…:05.123Z" sorts before "…:05Z". Mutation checks: truncation ignoring the exhausted candidate fails the new gap test; concatenating instead of merging fails the new order test and nothing else. internal/server suite green; timeline vitest 9 files / 83 tests green. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(web): stop the hop loop when the cursor does not advance (codex round 2) Against a server that predates next_before, the fallback re-derives the same last entry on every hop, so a single Load More click fired five identical requests where the pre-fix component fired one. The client cannot give an old server a cursor it does not have — that wedge is the old behaviour and stays — but amplifying it was new, and mine. A cursor that did not move cannot make progress, so the loop stops on it. Its test asserts the request COUNT, which is the only thing that distinguishes this from the behaviour it replaces: the entries rendered are identical either way. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(web): order the SSE-refresh merge too, and share one comparator (codex round 3) The refresh prepended its genuinely-new entries on the assumption that a fresh first page is always newer than everything on screen. Normally it is — but a structured note or decision carries a hand-written created_at and can arrive backdated, and the assumption was never stated, only relied on. Both merge points now go through one byNewestFirst, which is the server's own ordering and compares instants rather than strings. Two of codex's three round-3 findings are not folded in: - The refresh not adopting the response's has_more / cursor is DECLINED, not missed. The refresh re-fetches the NEWEST window, which says nothing about where the reader's paging frontier is; the stored cursor stays valid because the refresh consumes no older rows, and adopting the fresh page's has_more after the reader has paged deeper would point the cursor back at history they already hold. One wasted request, absorbed by the no-advance stop, in exchange for a correctness claim I cannot make. - firstPageIds treating any entry missing from a refreshed first page as deleted is real, pre-existing, and a semantics call about what counts as a deletion rather than a patch: filed as BUG-2773. Refs: BUG-2765, BUG-2773 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * test(server): derive the expected entry set instead of naming two ids (codex round 4) The test claimed every renderable entry came back exactly once and checked the two ids it had seeded. The item's own `created` activity could have vanished or repeated underneath that claim — the same partial-verification shape as asserting one direction and writing the symmetric conclusion. The expectation now comes from the store: every activity on the item minus the kinds buildTimeline drops unconditionally, compared in both directions, so a fixture that grows a row cannot fall outside what the test says it covers. Still fails with the cursor withheld. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * docs: state the timeline cursor contract where consumers read it (codex round 5) The TypeScript type documented next_before/next_before_id, but the handler's own doc comment — what a REST consumer reads — still described before + limit only, and the API client method said nothing. A consumer following either could still derive a cursor from its last visible entry, which is precisely the invalid contract this change exists to replace. Both now state the pair, that it must be forwarded rather than re-derived, why (dropped rows make the last entry a different position, sometimes no position), and that the id is the tie-break among entries sharing a second — send both or neither. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(web): clear paging state when the timeline reloads (codex round 6) The previous item's entries stay on screen while the new one's page 1 is in flight — the list renders on `!loading || entries.length > 0` — so Load More is clickable during a switch, and its cursor was the OLD item's position aimed at the NEW item. Pre-existing in shape (the pre-fix code derived the same stale position from the same stale entries), but now it is one line to close, and the dedicated cursor variable is mine. loadTimeline replaces entries with page 1 when it resolves, so clearing the cursor and has_more before the await throws away no state that would have survived; it just stops offering paging for a position that is no longer known. Test holds the switch's fetch unresolved and asserts no request goes out in that window, with the button's absence as the observable. Fails with the two lines removed. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |