9 Commits

Author	SHA1	Message	Date
xarmian	bd21b97c70	refactor(store): delete the NUL SUSPECT path, now that the predicate sees a NUL behind a repeated key (BUG-3220) (#1561) ... The suspect class held pre-filter matches the NUL predicate cleared, and migrate-to-pg cast each one on the destination, because one shape in it (a NUL shadowed by a literal duplicate key) was invisible to every layer. Since BUG-2812 the token walk reports that shape as an ordinary violation and the ordinary repair fixes it, so the only suspects left were doubled-backslash literals, which PostgreSQL accepts. ScanNUL judges stored bytes, so the same holds for any earlier release's rows (the PostgreSQL 12/17 x UTF8/SQL_ASCII/LATIN1 matrix is on the trail). Deleted: NULSuspect and the Suspects bucket, the destination oracle (CheckJSONBAcceptable and its classification), RepairSuspectValue and its buckets, the preflight's destination cast (preflightNULForMigration no longer takes a destination), the suspect heading in scan-nul/repair-nul, the exported RepairJSONEscapes whose only other caller was the suspect repair, and the docs paragraph. MigratedTables and nulKeyPredicate move out of the deleted file.	2026-09-26 01:09:03 -04:00
xarmian	7376023ff2	fix(server): the request-body gate reads tokens with jsontext, so a NUL behind a repeated key or beside 1e999 is refused, and a repeated request member is refused on every body (BUG-2812) (#1550) ... * fix(server): the request-body gate reads tokens with jsontext, so a NUL behind a repeated key or beside 1e999 is refused, and a repeated request member is refused on every body (BUG-2812) The body gate and the store's JSON-column guard decoded into map[string]any and walked the tree. The tree is a lossy model: it keeps one value per repeated key, so a NUL in a shadowed copy passed (and the typed decode could merge it into storage), and a number no float64 holds made the decode fail, so the gate answered "no NUL" for a body the typed decode then accepted. Walking that tree also cost about 2x on any body that merely mentioned an escape. - internal/jsonscan wraps github.com/go-json-experiment/json/jsontext (pinned v0.0.0-20260820222146-c27c302e5fc3, BSD-3, The Go Authors). It is the module's only importer; its doc comment names the exit to stdlib encoding/json/jsontext. A differential test pins its acceptance to encoding/json v1's, which found that jsontext refuses a lone surrogate v1 accepts; that case falls back to v1's unquote. - scanRequestBody reads every body once, recording a NUL and a repeated request member separately and reading to the end, so malformed input still reports nothing. Repeats count in the request's own structure, and not in caller data under JSON-encoded keys (BUG-2896 collapses those). Names are case-folded where the destination is a struct: decodeJSONBytes reflects its destination's TOP-LEVEL type, so a struct folds its members, a slice of structs each element's, and a map (admin settings, admin user limits) none. Below the top level only an exact repeat counts. decodeJSONBytes and both bundle paths refuse either. - textguard's document walk and artifactIsBindableText read tokens too, so every layer refuses the shadowed NUL together: the one textguard.KnownGaps case moves into Corpus. - The store's suspect path stays, redundant but correct, by ruling; its deletion is its own unit. Its instruments now pin what is true: no refused value is a suspect, and the remaining suspects are the doubled-backslash class the destination accepts. - The map-model walk survives only as the differential's oracle. * fix(server): update vendorHash and the Postgres-only suspect instruments for the token scan (BUG-2812) The Postgres-backed legs of the suspect path fired when the shared predicate learned to see a NUL behind a repeated key: that value is now a violation everywhere, and the store's own guard refuses it before the destination cast. The preflight legs now assert that, the unmigrated- table fixture uses a doubled-backslash literal (still a suspect), and PostgreSQL's native refusal stays measured by TestNativePostgresAgreesWithTheCorpus. vendorHash is the value this PR's Nix job recomputed.	2026-09-25 21:48:30 -04:00
xarmian	57b7ca5f48	feat(store,cli): ask the destination about suspects instead of dropping them (BUG-2810) ... Day-54 lead ruling on PR #1233, and the ruling names the defect precisely: the scan's own SQL pre-filter already surfaces the shadowed-duplicate row as a candidate, and `ParameterRefused` then drops it. So the preflight was discarding information it was holding and going on to promise the migration would go through. I had recorded that as an accepted residual on the grounds that closing it would violate DOC-2823's one-layer rule — but that rule is about what the enforcement layers REFUSE. It says nothing about a preflight throwing away a candidate it had in hand. **The SUSPECT class.** A pre-filter hit the predicate does not refuse. Most are doubled-backslash literals — text that writes ABOUT the escape, which is the false positive this whole predicate family exists to avoid. One member is not: a NUL in a value shadowed by a LITERAL duplicate key, which a map-model decode drops and PostgreSQL refuses. Nothing here can tell them apart, so nothing here tries: `pad db scan-nul` lists them under their own heading, apart from the violations, with what resolves them. **The destination is the oracle.** `pad db migrate-to-pg` casts each suspect on the TARGET connection — `SELECT $1::jsonb`, side-effect-free, and the very cast an INSERT performs — and refuses on 22P05 / 22021. That is exact in both directions precisely because it is not a fourth opinion of ours. Measured against a real server: the literal is ACCEPTED, the shadowed duplicate is REFUSED with 22P05, and a non-JSON value fails for a reason that is reported rather than refused on, because a NUL preflight that quietly grew into a general one would block migrations unrelated to this bug. **The repair had to be measured, not assumed, and the answer changed the design.** `textguard.Repair` leaves the shadowed value completely untouched: its scanner is gated on DocumentDecodesNULAnyShape, a map-model question that answers false for exactly this shape, so it never runs. A preflight that refused the row and printed `pad db repair-nul` would have been printing a command that does nothing to it — a remedy nobody ran (PATTE-135). So the repair reaches the class through the token-level scanner, exported for this, which rewrites the shadowed escape and still leaves the literal byte-identical because it consumes escapes in order. Suspects get their own buckets in the repair report rather than being folded into Repaired, so the dry run's promise and the run's result stay the same number. **Nothing about what any layer REFUSES changed.** textguard.KnownGaps and its pin are untouched, and TestScanNULInheritsTheRecordedKnownGaps still asserts the scan does NOT detect the shape. TestSuspectsCollapseWhenBUG2812Lands fails when the token-walk makes that false, and names every file to delete — the suspect path is a second mechanism that exists only while the predicate is blind. **One defect this found that no test did.** Running the real command against a real Postgres, the refusal announced "0 stored value(s) carry a NUL; nothing was migrated" while listing one — the count used the violations only, and the tests asserted the message CONTAINED "nothing was migrated" without reading the number. Fixed, and the assertion now reads the count. The whole loop is now verified end to end: preflight refuses, `repair-nul` fixes, the migration completes. My own prose from earlier on this branch is corrected with it. ScanNUL's doc comment, the preflight's, and docs/backup.md all said this shape passes the preflight and fails mid-copy, which the same commit makes false.	2026-09-02 16:45:38 +00:00
xarmian	49bd342e4c	fix(store,server,cli): three defects from codex round 1 (BUG-2810) ... **The import flag could not repair the column it exists for.** `--repair-nul` scanned the RAW body for a live escape, which is right for a value the gate reads at the top level and wrong for the one that actually matters. An item's `fields` blob travels through an export as a STRING: a NUL escape in the stored blob marshals into the body with a DOUBLED backslash, which a raw scan must leave alone because at that layer it is literal text — while the gate refuses it anyway, since it decodes the body and re-parses that string as the document it is. So the repair now walks the DECODED body with the same classing bodyDecodesNUL uses, one verb changed: where the gate asks textguard whether a value decodes to a NUL, this asks textguard to repair it. Two walks of one shape in one package is a real risk, and the mitigation is that they are measured against the same corpus in both directions rather than reviewed for similarity — TestBodyRepairMirrorsTheGateOverTheCorpus drives every case through the body shape and asserts refused-becomes-accepted and accepted-stays-byte-identical. Two consequences worth stating. The walk also reaches the OBLIQUE spelling — the backslash written as its own escape, so the six characters never appear in the raw bytes at all — which the scanner could not, so the test that pinned that limit is replaced by one asserting the capability. And re-encoding is now possible, so it is bounded: UseNumber, so an integer wider than float64 is not silently re-emitted in scientific notation; SetEscapeHTML(false); and a body with nothing to repair is returned byte-identical rather than round-tripped. The mutation that removes UseNumber turns 9007199254740993 into ...992, and a test says so. The header is now X-Pad-Repaired-NUL-Values, because at the decoded layer an escape is not a thing that exists any more and one nested document may have carried several. **The scan could not run on the databases it exists for.** Several protected tables carry a NULLABLE workspace_id — activities, api_tokens, mcp_audit_log — and the scan selected it into a plain *string, which fails with "converting NULL to string is unsupported" and takes the scan, the repair and the migrate-to-pg preflight down with it. Every column is now scanned as sql.NullString: SQLite also permits NULL in a declared PRIMARY KEY that is neither INTEGER PRIMARY KEY nor NOT NULL, which no other engine does, and a NULL key cannot address a row for an UPDATE — such rows are reported and skipped with the reason rather than handed a WHERE that matches nothing. Verified against the unfixed code: the scan returned `scan activities.actor row: sql: Scan error ... converting NULL to string`. It needed a VIOLATING row in such a table, which is why every fixture that planted its rows in `items` missed it. **--force by accident.** The repair skipped the running-server check whenever --from was given — and the most natural --from an operator types is the path `pad db scan-nul` just printed, which IS the live database. The check is now on the resolved path (Abs + EvalSymlinks, so a symlinked data directory or a relative path still matches), and a --from naming an unrelated backup stays unguarded, which is correct: nothing is writing it. The ordering moved with it. `store.New` runs pending migrations, so the refusal now happens BEFORE the database is opened; opening first and refusing second made the guard arrive after the thing it guards against.	2026-09-02 15:15:11 +00:00
xarmian	63da2f4f5f	feat(store,server,cli): count and repair the legacy NUL population (BUG-2810) ... Layers A and B stop the value being written. Neither makes a row that already carries one go away, and BUG-2810's filing is what that costs: an affected workspace exports with a 200 and re-imports with a 400, so a self-hoster restoring their own backup is blocked with no path forward in the product, and `pad db migrate-to-pg` fails partway through the copy against PostgreSQL's jsonb parser rather than up front. This is DOC-2823's S3, on Dave's day-54 rulings: U+FFFD as the replacement, repair standalone only with a migrate-to-pg preflight that refuses and prints the command, `--repair-nul` on import shipping default-strict. ONE REPAIR, beside the one predicate. textguard.Repair lives next to ParameterRefused because four layers that agree about what is REFUSED and disagree about what a repair PRODUCES is this bug family arriving one step later. Its contract is a property over the same corpus, in both directions: every refused value becomes one all four layers accept, and every accepted value comes back IDENTICAL. The second half is the load-bearing one — a repair that tidies values nobody complained about rewrites `{"a":"x\\u0000y"}`, six literal characters after a doubled backslash, and corrupts it. The JSON arm is a string-literal SCANNER, not decode-walk-remarshal, which is what the recon write-up proposed before it was written. Re-marshalling changes four things nobody asked to change — object key order, insignificant whitespace, integers wider than float64, HTML-ish characters — and silently drops one of a document's LITERAL duplicate keys, which is a gap BUG-2812 owns and the last thing a repair should do. Scanning copies every byte it does not deliberately rewrite, so an untouched document is byte-identical without that having to be argued. A substring replace is not equivalent and the test that proves it took a mutation to find: a doubled-backslash literal ALONE never reaches the scanner, so the discriminating fixture is one document carrying a live escape AND a literal. THE COUNT IS COMPUTED IN GO. Measured on the read path in this worktree: a row planted with `bad<NUL>name` reads back into a Go string with all 8 bytes and the NUL intact, while `length(name)` in the same database answers 3. TASK-2824 found that C-truncation and concluded no DB-side REPAIR could be trusted; the same measurement on the read path says no DB-side COUNT can be either. SQL narrows — `instr(col, char(0))`, plus the escape prefix on JSON-classed columns, which is textguard's own pre-filter — and never decides. The decision stays ParameterRefused with isJSON from the shared 86-column list, i.e. Layer B's classing. Row addressing is read from the live schema rather than a hand-kept map: 39 tables carry protected columns, one (item_wiki_links) declares no primary key and is addressed by rowid, two have composite keys, and five have a single key that is not `id`. The repair checks RowsAffected because an address that stopped selecting its row would otherwise commit an UPDATE that touched nothing and report it as repaired — the one failure an operator cannot see in the output. `email_optouts(email)` is both a protected column and its own primary key. Repairing it changes the row's identity and can collide with an existing row, which in that table means somebody starts receiving mail again. It is reported and skipped, with the reason. The import flag is NOT an exemption from the gate. `--repair-nul` buys the body one repair attempt and then runs the same `bodyDecodesNUL` on the repaired bytes, which still decides — a decode path that skipped the check is the door BUG-2803 spent thirty rounds closing, on the endpoint carrying the largest attacker-controlled body in the product. Only the ESCAPE form is repaired: a raw NUL byte makes the document invalid JSON, and widening what parses is not this flag's job. Both doors are covered, JSON and tar.gz, because giving them different answers is how one of them keeps being forgotten. Postgres is settled with evidence rather than sent up as a ruling: it cannot hold either defect (22021, 22P05) and the four-way differential test already pins that, so the scan reports not-applicable WITH the reason rather than returning a zero a reader could mistake for a clean database. Spellings settled here, per the dispatch: `pad db scan-nul` and `pad db repair-nul` as siblings rather than `repair --nul`, matching `migrate-to-pg`'s hyphenated compound — a repair verb that errors when given no flag is a worse shape, and there is no second repair to share it with. scan-nul IS the dry run, so repair-nul grows no --dry-run. It refuses while the server is running unless --force, on the `pad db restore` precedent: the report is a claim about a database, and one somebody else is concurrently writing makes it a claim about a moment that has passed. docs/backup.md's section on this is rewritten. It still said the rule lives in the binary and not the database, which S2 made false, and it pointed at this item for a preflight and a repair that now exist. Its import examples also showed `pad workspace import < file`, which has never worked — the file is an argument. Closes BUG-2810.	2026-09-02 14:53:31 +00:00
xarmian	f83de5b904	fix(store,textguard): close codex round 4 — stop chasing shapes, and correct a claim ... [P1] A valuer returning a valuer passed the guard: resolution happened once, so an outer valuer could hand the guard a clean inner value while pgx went on to evaluate it and send a NUL. Resolution is to a FIXED POINT now, bounded so a cyclic valuer is refused rather than looped on. [P1] The text detector still missed shapes pgx binds unconverted, and the next round would have named five more. Two rounds were already spent widening it one shape at a time — exact string, then *string and named types, then json.RawMessage — which is a losing game against a permissive driver. So it is an ALLOW-LIST now: the guard enumerates what this store actually binds (strings, binary blobs, numbers, booleans, times, NULL) and REFUSES anything else rather than passing an unclassifiable value through unchecked. That is the default every previous version had backwards. The full suite on both dialects passes, which is the evidence that the vocabulary is complete. [P2] The typed-nil rule was broader than database/sql's: it nil'd every nil pointer valuer, where the library only does so when the pointer's ELEMENT type implements Valuer and otherwise calls the nil-safe method. Copied exactly now. [P1 residual] And a rationale of mine was FALSE. The duplicate-key gap entry said Postgres "keeps the last too, so it accepts this as well - the two agree today, which is why it is a recorded gap rather than a dialect split." Measured on Postgres 17, a duplicate-key document whose SHADOWED value carries the escape is REJECTED with "unsupported Unicode escape sequence", while the all-clean control deduplicates fine: the parser processes the scalar BEFORE duplicate elimination. So it IS a dialect split — Postgres refuses, this guard accepts, SQLite stores — and I had asserted the opposite without checking. The entry stays in KnownGaps because closing it means replacing the shared predicate's decode with a token walk, which is BUG-2812/S4 and moves both layers together; fixing Layer A alone is the divergence DOC-2823 forbids. The native-Postgres leg ran Corpus only, so the recorded gaps were never measured against the database at all — which is how that wrong rationale survived. It covers KnownGaps now and reports the split explicitly. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm	2026-09-01 15:09:30 +00:00
xarmian	74c786d590	fix(store,server): close codex round 2 — the wrapper must mirror its base ... Seven findings; two were P1 and one of them was a real hole on Postgres only. [P1] driver.Valuer bypassed the guard. checkParams type-asserted `string`, and pgx implements NamedValueChecker — so it ACCEPTS a sql.NullString unchanged rather than letting database/sql's converter unwrap it, and the guard never saw the text. Measured before the fix: a NUL-bearing sql.NullString on Postgres passed Layer A entirely and was refused by the server as SQLSTATE 22021, i.e. a 500, while the identical value on SQLite got the typed 400 — the dialect split reappearing in the response shape. wiki_links.go binds sql.NullString today. [P1] driver.DriverContext was dropped, so sql.Open used a legacy connector that ignores the context and a cancelled request could leave a pgx dial running to its 60-second timeout. The rest share one cause, and it is the thing to remember: database/sql BRANCHES on whether an optional interface is present, so a wrapper advertising one the base lacks CHANGES behaviour rather than adding a no-op. Measured, the two drivers differ — pgx has no Validator, sqlite has no conn NamedValueChecker and no DriverContext — and the single wrapper type claimed all of them. So the wrapper now MIRRORS its base: four conn variants over the two interfaces that vary, a separate driver type for DriverContext, and the non-varying ones asserted at registration so a driver bump fails loudly instead of degrading. I made the same mistake inside the fix — implementing OpenConnector unconditionally, which broke every SQLite open — and then a third time, where guardConnector.Driver() returned the inner wrapper and a pgx pool reported no DriverContext. The third was caught by a new parity test on its first run, not by review: it asserts wrapped and base advertise EXACTLY the same interfaces. Also: both gap guardrails SKIPPED when their slice was empty, so deleting an entry made the suite green — the opposite of their purpose. They assert counts now. And the oracle test compared the two walkers without pinning any answer, which is how round 27 left both wrong about scalars; known answers are pinned against Postgres, and doing that caught me re-pinning one from a stale comment. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm	2026-09-01 13:57:07 +00:00
xarmian	ef792bdf35	fix(store,server,textguard): close codex round 1 on S1 — seven findings ... All seven verified before acting; two were production regressions I had introduced and one was a defect older than this branch. [High] The wrapper HID the optional connection interfaces. Measured: the raw modernc conn implements Pinger, SessionResetter and Validator; the wrapped one implemented none, so Ping succeeded without pinging, pooled connections stopped being reset, and dead connections stayed in the pool. Each is now forwarded, falling back to exactly what database/sql does for a conn lacking it. NamedValueChecker is forwarded too, or the wrapper silently narrowed the argument types pgx accepts. [High] Statement-level driver.ErrSkip was wrong. At the CONN level ErrSkip is the documented fallback signal; at the STATEMENT level database/sql propagates it, so a base statement without the context interface would have failed rather than degraded. Both now fall back to the positional form, refusing a NAMED argument rather than binding it by position. [High] Scalar JSON documents bypassed the check — and the database settles it: SELECT ('"a<escape>b"')::jsonb; ERROR: unsupported Unicode escape sequence A bare JSON string is a complete jsonb document. The object/array-only shape test is right for the HTTP gate's "is this a nested document" question and wrong for the store's "will a jsonb parser read this". Both now use a widened form, so the gate refuses it at the door instead of leaving it to the store. That widening broke the gate's independent oracle, which had been NARROWED in BUG-2803 round 27 to make the two walkers agree. The disagreement was real; the direction of the fix was not — nobody measured which one matched Postgres, and it was the oracle. Restored, with the measurement in the comment. Two implementations made to agree are not thereby correct. [High] The 400 mapping was not the single funnel I claimed. createItemChecked, bulk ops and the cross-workspace copy carry their own error envelopes by design. One shared CLASSIFIER now serves all of them; the envelopes stay different, the classification and wording do not. Copy keeps its retry-discouraging message deliberately (PLAN-2357 DR-13). [Medium] The over-refusal justification was FALSE as written. It claimed a JSON-shaped text value carrying a live escape "is a value Postgres would refuse the moment anything parsed it". Nothing parses a text column; Postgres stores it fine. The trade is now stated honestly, and pinned: textguard. StoreOverRefusals records the case, and a test fails when it is paid down. [Medium] Duplicate JSON keys are a real under-refusal, inherited from the shared predicate's map model. DOC-2823 requires Layer A NOT to fix it alone, so it is recorded in textguard.KnownGaps with a test that fails when it CLOSES — the signal that BUG-2812's token-walk landed. [Medium] Test weaknesses, and the fixes caught a real one. The census now sees ALTER TABLE ... ADD COLUMN. The store corpus leg now requires accepted cases to SUCCEED, which immediately exposed a case passing for the wrong reason: a non-JSON value written to the fields column failed on SQLite's own JSON parser, never reaching the guard. Full Go suite green on SQLite and Postgres 17; lint 0 issues. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm	2026-09-01 12:54:19 +00:00
xarmian	d0644a60fc	refactor(textguard): extract the one decoded-NUL predicate and its corpus (DOC-2823 S1) ... Groundwork for S1's store-layer guard. The predicate lived in internal/server and could not be reached from internal/store, so a store guard would have had to reimplement it — two implementations of "does this decode to a NUL", which is the disagreement class this whole cluster exists because of. internal/textguard is a leaf package holding the predicate and the adversarial corpus. The HTTP gate now DELEGATES to it rather than carrying its own copy: stringIsJSONDocument and nestedDocumentDecodesNUL become one-line forwards, and valueDecodesNUL's unclassed arm is textguard's walk verbatim. Its existing 418-line corpus passes unchanged, which is the extraction's proof. What is deliberately NOT shared is named in the package doc: how a layer decides a value IS JSON. The gate derives that from request-body KEY NAMES; the store derives it from the COLUMN a parameter is bound to. Those cannot be merged, so every entry point takes the classification as an argument and the differential test pins that both derivations agree on one corpus. The corpus ships in the package rather than in a _test.go because three packages measure against it and three copies would drift. Fifteen cases, each carrying the reasoning or measurement that put it there — including the doubled-backslash false positive that made the raw-byte parity filter unsound, the raw-NUL-inside-a-JSON-value regression codex round 9 caught, and the nested-document asymmetry where the same escape is fatal one layer up. The escape text and the NUL character are constants, and a test asserts they did not decay into each other by comparing against a value it BUILDS rather than one it types. That mistake was made three times during BUG-2803 and twice more while writing this file. Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm	2026-09-01 04:24:45 +00:00