The query-string half of BUG-2782. A caller-supplied query value reached a
Postgres text comparison, Postgres refused the parameter, and the handler
answered 500 — the honest answer is 400, because the caller asked about
something that cannot exist.
ValidateQuery is a root-router middleware beside ValidatePath, refusing 400
invalid_query when a decoded query key or value is invalid UTF-8 or carries
a NUL. Both share one predicate, bindableText.
Measured on Postgres 17 (server_encoding UTF8) at 19330410: 8 GET endpoints
x 54 parameter names — every name any handler reads — each probe from its
own source IP because the api limiter is keyed on ip:.
invalid-UTF-8 value: 276 x 200, 56 x 400, 100 x 500 -> 432 x 400
NUL value: 276 x 200, 56 x 400, 100 x 500 -> 432 x 400
control: 376 x 200, 56 x 400, 0 x 500 -> unchanged
Zero 500s in the control, so the 100 are attributable to the value. The
error is `invalid byte sequence for encoding "UTF8": 0xff (SQLSTATE 22021)`.
WHY A TRANSPORT RULE, when BUG-2782 planned per-site validators on
BUG-2774's validCursorID model. Reading the mechanism retired that plan:
parseItemListParams folds every parameter it does not recognise into a
field filter, so ?email=, ?type= and ?anything-at-all= reach a text
comparison exactly as ?search= does — 98 of the 100 failures are those two
endpoints. The set of names is unbounded by design, so there is no finite
list of points to validate.
WHY IT IS NOT A NARROWING of what callers may send, which was BUG-2782's
objection. That objection is sound against a charset rule and does not
reach this one: bindableText requires only valid UTF-8 with no NUL, every
legitimate value here is text, and text is valid UTF-8 in any language.
CONTRACT CHANGE: the timeline's before_id answers invalid_query rather than
invalid_cursor, since the transport rule runs first. Same 400, same
client-error contract, less specific code. validCursorID is NOT dead — two
of its three call sites read ids from the item's own fields blob, which no
request middleware sees — and that reasoning is recorded at the function
definition where someone would land before deleting it.
Keys are validated precautionarily: an invalid-UTF-8 parameter NAME did not
reproduce a 500 in the sweep (7 x 200, 1 x 400), and why it survives is
unread, so they are checked rather than assumed safe.
Gates: go test ./... green; full Postgres suite -timeout=45m green (28
packages, own container, not the shared port); -race on internal/server
green; lint 0 issues; vuln clean; gofmt clean. Mutation matrix 14/14,
including the unwiring mutation run against the Postgres leg to confirm it
fails with the ORIGINAL 500 rather than merely failing.
Ten adversarial review rounds. The first two found a vacuous test (one item
in the workspace meant a handler ignoring ?search passed it) and a
self-contradicting proof. Later rounds found false statements, including
one where my own sweep CLAIM was false. Two real defects were found AFTER
the first MERGE verdict, which is why the rounds continued.
Filed not folded: BUG-2803, an escaped NUL in a JSON body reaching the
store on every JSON write path — a different surface needing decode-time
rather than transport-level validation.
Release note: invalid UTF-8 or NUL bytes in query parameters now return 400
instead of 500 on Postgres deployments.
Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
xarmian