17 Commits

Author	SHA1	Message	Date
xarmian	427540706c	fix(documents): bound the rename cascade at the title and at the projected total (BUG-2798, BUG-2796) (#1218) ... * fix(documents): bound the rename cascade at the title and at the projected total (BUG-2798, BUG-2796) A document rename rewrites [[oldTitle]] into every linking document. Neither factor of the output size was bounded: titles had no length validation, and the cascade holds every rewritten body in memory before writing any of them. One rename could project 10 GB from a 500 KB input -- 20,000x, measured -- and OOM while holding the workspace rename lock. Two walls, per Dave's day-63 ruling. 1. Title length, bounded at write time (models.MaxDocumentTitleRunes = 255). Runes, not bytes: "255 characters" is what a user and a UI counter mean. Existing over-limit titles stay valid until their next rename -- no retro-breakage of stored data. 2. The cascade's projected TOTAL, bounded at 16 MiB (store.MaxRenameCascadeProjectedBytes), accumulated across the linking set and refused before the first rewrite is built. The total is the right quantity and a per-document cap would not have been. Measured, with the title bound already in place: one linker holding the largest body a 2 MiB request can carry projects 108,632,370 bytes -- 51.8x -- and the aggregate is linear in the number of linkers (108.6 / 217.3 / 434.5 MB at k = 1/2/4, allocation tracking output at ~1.02x). A per-document cap of C still admits k * C, which is the same unbounded shape one level up. The 16 MiB figure has a receipt in the constant's doc comment: it sits above the absolute ceiling of any cascade this development instance could produce (its entire wiki-linking corpus is 10,077,476 bytes) and 6.5x below the single-document attack. The refusal is permanent-shaped and deliberately NOT in ErrLinkCascadeContention's family: 413 with the projection in the message and no Retry-After. Contention means "someone got there first, try again"; this means "this rename cannot be performed as asked". Answering it from the retryable family would tell a client to retry forever. BUG-2796 folds in at the same validation point, as ruled -- a title containing wiki-link syntax is emitted raw by links.ReplaceTitle, so renaming to `A]] [[A` produced two broken links and reported success. The rule is derived from the two mechanisms that consume a stored bracket (the grammar at markdown.ts:327 and the unescaper at markdown.ts:753) rather than from a character blacklist: the first version of this fix banned `]`, `\` and `|` because all three "look like wiki-link syntax", and the round-trip test refuted two thirds of that. `|` in particular is a title shape resolveWikiBody contains a dedicated branch to support, and `[` passes the grammar untouched. Doors enumerated rather than assumed (CONVE-24): store.CreateDocument and UpdateDocument have exactly two callers between them, both HTTP handlers. No CLI, import, or seed path writes a document title. Update previously validated doc_type and status and NOT title -- the one field that drives the cascade -- so the handler tests drive real requests through both doors (CONVE-19). BUG-2798, BUG-2796 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): count retained bytes, bound the retry path, escape the cascade's LIKE pattern (BUG-2798) Codex round 1 on #1218. Three findings, all real, all fixed here. 1. The guard bounded projected OUTPUT, which bounds nothing when the new title is SHORTER than the old one. Renaming a 255-character title to a one-character title makes each 2 MiB linker project ~40 KiB while the cascade still retains its 2 MiB read for the compare-and-set, so hundreds of linkers exhaust memory while the counter reports well under the cap. The counter now sums RETAINED bytes — read plus written, both alive at once — so the cap is a statement about resident memory rather than about output. MaxRenameCascadeProjectedBytes becomes MaxRenameCascadeRetainedBytes and moves 16 -> 32 MiB, because the legitimate ceiling it clears doubles under the new metric (that instance's whole wiki-linking corpus retains ~20,154,952 bytes); the single-document attack retains 110,729,522, so it is still refused by 3.3x. 2. The compare-and-set's retry path bypassed the guard entirely. On contention it re-reads the linker and calls ReplaceTitle on whatever the winner wrote — a NEW input, bounded by nothing the scan had checked — so a content edit landing inside the cascade's window could grow a linker from harmless to enormous and walk the rename back into the amplification it would have been refused for. Each document's compare-and-set now carries the cap less what the other linkers hold, and re-checks the grown body against it. 3. The cascade's `content LIKE ?` search term went in unescaped, so a document TITLE decided how the pattern was read. `\` is the default LIKE escape character on Postgres and NOT on SQLite, so `[[Alpha\Beta]]` was searched for as itself on one dialect and as `[[AlphaBeta]]` on the other: linkers not found, cascade rewrites nothing, rename reports success, every link left stale. Silent and dialect-dependent. Codex named the backslash; `%` and `_` are the rest of the class (CONVE-18) — wildcards on both dialects, so a title carrying them selects documents that do not link it. An explicit `ESCAPE '\'` clause plus escapeLikePattern makes both dialects agree, rather than leaving SQLite correct by accident. Finding 3 also constrains finding 3 of the ORIGINAL fix: models' validator allows a lone backslash in a title on the grounds that both renderers handle it, which was true of rendering and false of cascading. That comment now records the dependency — allowing it is only correct while the cascade's pattern stays escaped. Tests, four new, each mutation-verified against the code it guards: - CountsRetainedBytesNotJustOutput — the shrinking rename. Asserts as a PRECONDITION that the projected-output total stays under the cap, so the test cannot pass for the old reason. - RetryRecheckesTheBudgetAgainstTheGrownBody — drives the real race through the afterLinkCascadeRead seam. POSTGRES ONLY and skipped loudly elsewhere: SQLite's BEGIN IMMEDIATE closes the window structurally, so a green run there would be a property of the DSN. - FindsLinkersWhoseTitleContainsABackslash — Postgres only, same reasoning inverted: SQLite is the dialect that was accidentally right. - DoesNotSpendTheBudgetOnDocumentsThatDoNotLinkTheTitle — `%` and `_`. Its first version asserted the decoy's content was untouched and passed against the unescaped pattern, because over-matched rows rewrite to themselves. The observable harm is that they spend the caller's budget, so that is what it now asserts. Mutation matrix for this round: output-only counter -> only the shrinking test fails; retry check removed -> only the retry test fails (PG); LIKE unescaped -> the budget legs fail on SQLite and the backslash test fails on PG. Gates: `go test ./...` under Postgres 17 EXIT=0; SQLite packages EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798, BUG-2796 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): tighten the retry budget, stop charging no-op rewrites, order the typed check first (BUG-2798) Codex round 3 on #1218, an edge-case angle over the new arithmetic and control flow. Three findings fixed, one declined. 1. The retry budget credited back this document's own share, on the reasoning that the retry replaces it. It does not: the original read and rewritten bodies stay reachable through `updates` while the write loop runs, so the re-read and its rewrite are allocated ON TOP of them. The bound could be exceeded by up to one document's share while the arithmetic still reported it satisfied. The budget is now the genuine headroom, `cap - retained`. 2. A concurrent edit that REMOVES the link left a body with no occurrences, which cascadeRetainedBytes still charged twice — once for the read and once for a rewritten copy that does not exist, because strings.Replace returns its input unchanged when there is nothing to replace. That could refuse an otherwise valid rename for memory the cascade never allocates. 3. The handler classified this error by PROSE before testing it by identity. The UNIQUE-constraint arm matches a substring, and the refusal error embeds the caller's title verbatim, so renaming a document to a title containing the words "UNIQUE constraint" came back as a 409 name collision — advice to pick a different name, for a rename that was refused for size and would fail identically under any name. Typed sentinel now tested first. DECLINED: unchecked int64 arithmetic in the projection. The multiplicands are derived from the length of a string already resident in memory, so overflowing int64 needs a single document body of roughly nine exabytes; and the accumulator returns as soon as it passes the cap, so it cannot run away either. Saturating arithmetic here would be guarding a state the machine cannot reach. Tests, three new, each mutation-verified: - RetryBudgetExcludesThisDocumentsOwnStrings — deliberately separate from the existing retry test, because that one catches the check being ABSENT and this one catches it being too GENEROUS. The grown body is sized to fall BETWEEN the two budgets; a body far over the cap cannot tell them apart. - ConcurrentEditThatRemovesTheLinkDoesNotRefuseTheRename — its first version sized the link-free body against the CAP rather than against the retry's real headroom, so the refusal it caught was correct behaviour and the test was wrong, not the code. Re-sized against the headroom: fits when charged once, does not when charged twice. - IsNotMisreportedAsATitleCollision — at the handler, since the defect is entirely in its classification order. Mutation matrix for this round: credit the share back -> only the tight-budget test fails; charge the no-op body twice -> only the link-removed test fails; order the substring arm first -> only the misclassification test fails. Gates: `go test ./...` under Postgres 17 EXIT=0; touched packages re-run after the lint fix EXIT=0; gofmt clean; `make lint` 0 issues. CI green on b09aca12. BUG-2798, BUG-2796 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * test(documents): close four ways the cascade-bound tests could pass for the wrong reason (BUG-2798) Codex round 4, aimed at the TESTS rather than the code. No production behaviour changes here; four instruments that were weaker than they read. 1. Every retained-byte case exceeded the cap under `max(read, rewritten)` as well as under `read + rewritten`, so none of them could tell the two arithmetics apart — and taking the larger would hold twice the cap. Added CountsBothStringsNotTheLargerOne: an ordinary same-length rename over content totalling ~60% of the cap, which the sum refuses and the max admits. Its preconditions assert both halves of that gap. 2. Nothing pinned the 255 itself. Every length case derived its inputs from MaxDocumentTitleRunes, so changing the constant to 512 left them all green. That is fine for arithmetic and wrong for this number: it is a product decision Dave ruled, and a silent change to it silently changes how much amplification the cheap door lets through. Deliberately NOT done for MaxRenameCascadeRetainedBytes, which is mine and carries a measured receipt that is expected to be re-measured. 3. The oversize tests asserted only THAT a rename is refused, never that it is refused BEFORE the amplified string is built — which is the entire point of the guard. Moving links.ReplaceTitle above it would have kept them green. Added RefusesBeforeBuildingTheRewrittenBody, measuring cumulative allocation with a ~20x margin: refusing costs the one body it had to scan, building first costs ~108 MB. Verified by mutation — with the guard moved after the rewrite it reports 110,748,144 bytes against a 52,428,800 ceiling. This filing warned that measuring memory to prove the ABSENCE of amplification is flaky by construction. That still holds for the shape it described, a peak-RSS floor. This is the opposite: a generous ceiling on a deterministic counter, with the two outcomes twenty times apart. 4. The "reports the projection" assertions checked for the words `maximum` and `bytes`, which a message saying "maximum bytes exceeded" would satisfy while telling a caller nothing. They now require the cap's actual value and a real byte count. Mutation matrix: charge only the larger string -> only CountsBothStrings fails; move the rewrite above the guard -> only RefusesBeforeBuilding fails. Seventeen mutations across four rounds, each detected by the test that should catch it. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): compose the 413 from typed fields instead of splicing err.Error() (BUG-2798) Codex round 5, on the side effects of a REFUSED rename. Side effects were otherwise clean — rollback removes versions, link rewrites, attachment stamps and the title change, and no activity row, SSE event or webhook is emitted — but the response body was built by appending err.Error() to a public sentence. That published whatever any layer had wrapped around the error on its way up. Today that is "update links: store: ", which is a call path clients have no business seeing; tomorrow it is whatever the next wrapper adds, with no decision point in between. The response is now composed from typed fields on a new store.RenameCascadeTooLargeError (NewTitle, Retained, Max), reached with errors.As. Unwrap keeps errors.Is(err, ErrRenameCascadeTooLarge) true, so every existing sentinel check is unaffected. Both FIGURES stay in the message, deliberately: Dave's day-63 ruling asked the refusal to state what it would hold and what the cap is, so "split the rename" is actionable advice rather than a shrug. The reviewer read those numbers as a content-size oracle; that framing does not survive the trust boundary — a rename requires `editor`, documents are readable at `viewer`, so the caller can already read every document the figure summarises and learns nothing from it. What they had no business receiving was the internal call path, and that is what changed. This is round 3's lesson applied in the other direction. There, prose was being used to CLASSIFY an error and should have been identity. Here, prose was being used to REPORT one and should have been data. The test now asserts both halves: the real byte counts are present (not merely the word "maximum"), and the strings "update links:" and "store:" are ABSENT. Mutation-verified — splicing err.Error() back in fails on both leaked prefixes. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * docs(documents): correct eight claims the prose made that the code does not (BUG-2798) Codex round 6, aimed at the comments rather than the code. Eight findings, all mine, all real, no behaviour changed. This is the failure mode my own trail keeps naming — code right, prose broader than the sweep, always in the same direction — so they are corrected individually rather than smoothed over. Stale after the round-1 rename: - models cited store.MaxRenameCascadeProjectedBytes, which no longer exists. - the constant's own hostile figure read 110,729,522; it is 110,729,520. - the HTTP test said each body is ~135 KB; the formula produces 264,790 bytes. Claims wider than what is true: - The round-trip test's header stated a biconditional over the whole validator. False: a 300-rune title round-trips perfectly and is still refused, for the unrelated reason that it is an amplification factor. The property is about the SYNTAX rule, over titles inside the length bound, and now says so. - The mirrored grammar/unescaper comment claimed that a TypeScript change would make this test start disagreeing. It cannot — they are static copies, and nothing in the repository fails when the two drift. Replaced with what the duplication actually buys and what it does not. - The cap's receipt used `items` measurements to conclude the guard "cannot fire on honest use" for DOCUMENTS, having itself noted that instance's documents table is empty. The proxy is reasonable and it is an assumption, not a measurement of the guarded path; the inference is now named, with the narrower claim that survives without it. - The 413's comment cited the image_too_large precedent as a bound on OUTPUT while this guard bounds retained read-plus-write. What carries across is the shape — a small request refused for what handling it would cost — not the quantity. - TestRenameCascade_RefusesTheSingleDocumentAttack described the 2 MiB / 110,729,520-byte shape it does not build; it sizes from the cap (1,271,000 bytes retaining 67,108,800). The full-strength shape is exercised by the allocation test, and the comment now points there instead of describing a body that is not in the function. One figure was replaced by measurement rather than corrected by arithmetic: the allocation test claimed a "~20x margin". Both sides are now measured and stated separately — refusing allocates 2,114,624 bytes, ~24.8x under the 52,428,800 ceiling; building the rewrite first allocates 110,748,144, ~2.1x OVER it, taken from running the test against the mutation rather than computed. The smaller margin is the binding one, since it is the gap a regression must cross to be caught, and saying "~20x" hid that. Gates: `go test` on the three touched packages under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): stop charging the cascade budget for rows the rewriter cannot touch (BUG-2798) Codex round 7. Third instance of one class, and the one I stopped short of when I extended the previous two. The SELECT that finds candidate linkers is a LIKE; the thing that rewrites them is links.ReplaceTitle. They do not agree on what counts as a match, so the SELECT returns a SUPERSET — and every row in the difference was charged to the caller's retained-byte budget and then handed a no-op UPDATE. Instances one and two were `%` and `_`, wildcards on both dialects, closed by the ESCAPE clause. This is the case half, and it splits the OTHER way from the backslash bug: SQLite's LIKE is ASCII case-insensitive by default while Postgres's is case-sensitive, so renaming `Alpha` scans every body holding `[[alpha]]` on SQLite only. ReplaceTitle is case-sensitive on both and will never touch them, so enough case-variant content could push an otherwise valid rename to a 413 — on one dialect, for content that was never in scope. The fix is to skip a row with no case-sensitive occurrence outright, which closes both halves: no budget is spent, and no pointless UPDATE is issued for a body the cascade was never going to change. The authority on what is a linker is the rewriter's own count, not the pattern that proposed the candidate. The test runs on BOTH dialects deliberately, unlike its Postgres-only backslash sibling: on Postgres it asserts the behaviour was already correct, which is what makes it a regression test rather than a SQLite quirk shim. It also asserts the case variants are left byte-identical — `[[alpha]]` is a different link, not a missed one. Mutation-verified: restoring the charge fails it with 33,554,790 bytes against the 33,554,432 cap. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. An earlier attempt at that gate died on host disk exhaustion, not on this diff — 373 stale go-tmp directories from crashed runs, cleared, re-run green. BUG-2798 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): report the whole operation's size when a retry is refused (BUG-2798) Codex round 8, on concurrency and the rest of the rename transaction. The advisory-lock lifetime, the CAS loop's termination, the ordering against attachment stamps and version writes, and the new skip's effect on the transaction's invariants all came back clean. One P2 stood. The retry path bounded correctly and REPORTED wrongly. It compared the re-read body against the headroom — right — and then named only that body in the error. Everything the scan counted is still held, so the operation's real size is the scan total plus the re-read, and a refusal could therefore say it would hold 16,777,200 bytes against a limit of 33,554,432: a refusal whose own figures do not justify it, which reads as a server bug rather than as advice you can act on. The compare-and-set is now handed the scan TOTAL instead of a pre-computed budget, so the same number both bounds and explains: refuse when scanTotal + grown exceeds the cap, and report scanTotal + grown. The test now asserts the refusal justifies itself — the figure reported must exceed the cap it cites — via the typed error added in round 5, which is what makes that property checkable at all rather than a string comparison. Mutation-verified: reporting the re-read alone fails it with exactly the 16,777,200-against-33,554,432 shape. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): refuse titles whose links the cascade cannot find; count retry buffers (BUG-2798) Codex round 9, asking what is MISSING rather than what is wrong. Three findings: two fixed, one already filed. ## Titles validated against the wrong layer The validator ACCEPTED `Alpha|Beta` and `Alpha\Beta`, and I defended that choice with a test. Both round-trip perfectly — the renderer reads each back as exactly the title it started from — and the first version of this fix banned them on vibes, so being shown that was a real correction. It was still the wrong call, for a reason that test could not see. A link to such a title can be STORED escaped, as `[[Alpha\|Beta]]`, and the rename cascade searches for the raw `[[Alpha|Beta]]` only. It does not find those links, so the rename succeeds and leaves them pointing at a title that no longer exists — silently, which is BUG-2796's defect wearing different syntax. So the property a title has to satisfy is stricter than the one I tested: not "the renderer reads it back", but "the renderer reads it back AND the cascade can find its links". Validating against the layer that DISPLAYS a title while the layer that MAINTAINS it disagrees is the same mistake as the unescaped LIKE, met from the other side — twice in one unit, which is the part worth noticing. `[` stays accepted: the cascade's search term matches it literally, so it passes the stricter property too. The two characters get their own test rather than a row in the round-trip table, because the table asserts a biconditional and these are refused for a reason that predicate deliberately does not model. That test asserts its own premise — each title must still round-trip — so if that ever stops being true it fails rather than passing for a new reason. ## Retry buffers were not counted rewriteLinkerCAS bounded each retry against the scan total plus THAT attempt. Earlier attempts' buffers become unreachable when expected/next are reassigned, but unreachable is not reclaimed, so a run of failures could hold several copies while the arithmetic counted one. Now accumulated across attempts, which is conservative — it counts garbage as if live — and errs toward refusing, which is the safe direction for a memory bound. The loop is capped at cascadeRewriteAttempts, so it cannot grow without end. ## Already filed Item renames remaining unbounded, and item titles lacking this validation, are real and out of this unit's scope: BUG-2804 and BUG-2805, filed after round 2 with the code verified rather than taken on report. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798, BUG-2796 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): validate a title only when the rename actually changes it (BUG-2798) Codex round 10, on back-compatibility. This one is a regression THIS fix introduced, not one it inherited, and it falsified a promise the fix makes about itself in three places. "Enforced at write time; existing titles stay valid until their next rename" is Dave's ruling, and it is repeated in the constant's doc comment and in two commit messages. Validating every SUPPLIED title broke it for the most ordinary shape of an edit there is: a client that PATCHes the whole object, title included, to change the content. Under that, a document with a legacy title became uneditable rather than merely un-renameable — the opposite of grandfathering. Grandfathering is not something you get by validating at write time. It is something you get by not validating a write that is not a rename. The check now fires only when the supplied title DIFFERS from the stored one, which is also exactly the test the store already applies before cascading, so the validation and the work it guards now agree on what counts as a rename. The regression test seeds its legacy document through the store, because the title it needs can no longer be created through the API — which is precisely the population the grandfathering clause exists for. Three legs: the echoed-title content edit succeeds, a title-less content PATCH succeeds, and renaming to another invalid title is still refused. The last is the control; without it, deleting the validation entirely would pass. Filed rather than folded: BUG-2806, existing documents whose links are stored in escaped form are still orphaned by a rename. Round 9 stopped NEW titles of that shape being created; it did not repair the ones already stored, and the asymmetry — the product refusing to create a shape it still mishandles — belongs in the record rather than in this PR, which is nine commits deep on a bound it has already outgrown. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): validate the rename under the lock, not against a pre-lock read (BUG-2798) Codex round 11. Round 10 moved title validation behind "only when the title actually changes", which is the right rule and was applied at the wrong place. The handler compares the supplied title against a document it read BEFORE the rename lock. UpdateDocument re-reads under the lock. Those can disagree: echo a legacy title back on a content edit while another request renames the document, and the handler sees "unchanged, skip validation" while the store sees a genuine rename — and writes the legacy title through with nothing having checked it. The rule is unchanged; the enforcement point moved to where the rename is actually decided. The store validates inside the transaction, on the same branch that triggers the cascade, and returns a typed InvalidDocumentTitleError carrying the reason. The handler keeps its pre-lock check, which is still worth having — it gives the common case a fast 400 without opening a transaction — and gains an arm that surfaces the store's refusal for the case its own check could not see. Grandfathering survives intact, because the store's check sits on the title-actually-changed branch, which is the same condition the handler uses. The test calls the store directly rather than reproducing the race: driving the interleaving would test the scheduler, while the property worth pinning is that the store refuses regardless of what a caller did. Its control leg is the grandfathering case — a content edit echoing the unchanged legacy title must still succeed — so validating everything here would fail it, which is round 10's regression restated as a guard. Mutation-verified: removing the store-side check leaves the handler tests green and fails this one with a nil error. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * perf(documents): stop counting every linker's occurrences twice (BUG-2798) Codex round 13, on what the guard costs the SUCCESS path rather than what it blocks on the failure path. The cascade counts occurrences because the size guard needs that number before it is willing to build anything. It then handed the same content to links.ReplaceTitle, whose strings.Replace with n < 0 counts it again. Every ordinary rename therefore paid a second full pass over every linking document for a number it already had. links.ReplaceTitleN takes the count the caller already computed. It is a separate function rather than an optional parameter because the obligation is real and silent when broken: passing a number that is too small does not error, it leaves later occurrences unrewritten, which on this path means links left pointing at a title that no longer exists. A name at the call site is cheaper than a comment nobody reads. NO measured speedup is claimed, and the doc comment says so. This removes one linear pass from a path that also allocates a full copy of the same content and issues a write per linker, so the saving is real but not obviously significant. It is here because doing the same work twice needs a reason and there was not one — not because a benchmark asked for it. The test asserts equivalence with ReplaceTitle across several shapes, including the new-title-embeds-old case, and its counterfactual leg asserts that an under-count visibly DIVERGES — if it did not, the caller's obligation would be imaginary and the API misleading. Round 13 also confirmed two things worth recording: no quadratic scan across linkers, and the ESCAPE clause does not materially change the query plan because the leading `%` already forced a content scan. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * docs(store): record the consequence the cascade cap necessarily has (BUG-2798) Codex round 14, on authorization and abuse. The refusal paths came back authorization-clean: reachable only after the workspace-access and `editor` checks, with viewers, guests, non-members and cross-workspace document IDs stopped first, and the reported byte figure exposing nothing an editor cannot already read. One consequence stands, and it is a property of having a cap at all rather than a defect in this one: once a workspace's documents linking a title exceed 32 MiB, that title can no longer be renamed, and any editor can put it in that state. Recorded in the constant's doc comment rather than fixed here, because the comparison that matters is with what it replaces. The same input previously took the server down for everyone; it now denies one operation to a role that can already delete every document in the workspace. Trading an unbounded OOM for a bounded, legible refusal is the point of the guard, not a gap in it. What IS missing is that the state has no exit but manual cleanup, with nothing telling an operator which documents to clean. That is a quota-and-recovery question rather than a cascade question, and it is filed as IDEA-2807 with three candidate shapes and an argument for the cheapest one — a state you can get out of is a different severity from one you cannot. The filing also says what is NOT established: no real workspace is known to approach the cap, and this fix's own receipt suggests none does, so it is a trap that exists rather than one anyone has fallen into. The adjacent concern the review raised — repeated near-cap renames contending for the rename lock and the connection pool — is noted there too, with the observation that the pre-existing behaviour was strictly worse, since each attempt was unbounded. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * revert(links): remove ReplaceTitleN — it optimised nothing (BUG-2798) Codex round 15 caught a claim of mine that was simply false. Reverting the functional half of c00606b0. I added ReplaceTitleN so the cascade could hand strings.Replace the occurrence count it had already computed, and wrote that this "removes one linear pass from a path that also allocates a full copy". It does not. strings.Replace calls Count UNCONDITIONALLY, before it looks at n: func Replace(s, old, new string, n int) string { if old == new || n == 0 { return s } // Compute number of replacements. if m := Count(s, old); m == 0 { return s } else if n < 0 || m < n { n = m } Read from this machine's GOROOT this turn, rather than recalled. Passing n constrains how many replacements are APPLIED; it does not skip the count. So the function bought nothing and cost something: a second way to do the same thing, carrying an obligation that fails SILENTLY when broken — an under-count leaves later occurrences unrewritten, which on this path means links pointing at a title that no longer exists. API surface with a silent failure mode and no payoff is worse than no API, so it goes rather than getting a corrected comment. The failure is the one my own trail keeps naming: I asserted a mechanism without reading it. What makes this instance worse than the earlier ones is that I wrote a careful hedge — "no measured speedup is claimed" — which reads as rigour while the sentence beside it stated the mechanism as fact. Declining to measure a claim is not the same as checking it, and the hedge made the unchecked claim look examined. The occurrence count stays where it is: the guard genuinely needs it before it will build anything, and computing it there is not redundant with anything the guard can avoid. Also declined this round, as already filed: renaming a legacy title with `|`, `\` or `]` leaves escaped links stale — that is BUG-2806, filed at round 10 with the mechanism verified in code. Gates: `go test ./...` under Postgres 17 EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN	2026-08-27 22:45:27 -04:00
xarmian	cade263fe5	fix(store): compare-and-set the wiki-link cascade's writes (BUG-2785) (#1213) ... * fix(store): compare-and-set the wiki-link cascade's writes (BUG-2785) A document rename cascades through every document linking the old title as a read-modify-write across two statements: SELECT each linker's content, rewrite the string in Go, UPDATE the row. A content edit to a linker committing between those two statements was silently overwritten — the cascade wrote the body it built from the version it read, with no error and no version row for the loss. Same lost-update shape as BUG-2770's activity-metadata merge, one table over, and fixed the same way: the UPDATE now carries `content = ?` with the body the cascade read, plus a bounded retry that re-reads the row and re-applies the rewrite. DIALECT SCOPE, which decides what the tests can prove. Reachable on POSTGRES only. SQLite's DSN sets `_txlock=immediate`, so UpdateDocument's db.Begin() takes the write lock at BEGIN and holds it across the whole read→write window; a concurrent edit cannot commit inside it and serializes on busy_timeout instead. On Postgres under READ COMMITTED each statement takes a fresh snapshot and the stale body wins. The CAS is therefore a no-op on SQLite by construction — the predicate always matches, because nobody else can have written. Two consequences, both acted on rather than noted: the new tests SKIP loudly on SQLite instead of passing for a reason unrelated to this fix, and the mutation matrix was run under Postgres, where removing the CAS leaves a SQLite suite entirely green. The retry rewrites the WINNER's body rather than replaying the original rewrite — replaying it would reintroduce exactly the text this bug loses. A mutation that replays instead is in the matrix. THE ZERO-ROW RESULT NEEDS A PROBE. The UPDATE now has two predicates that can each refuse it, and RowsAffected cannot say which did: `deleted_at IS NULL` (the linker was archived — a documented normal outcome, stop) or `content = ?` (a concurrent edit landed — re-read and retry). Treating them alike either retries forever against a deleted row or discards a live linker's rewrite, so a probe distinguishes them, as BUG-2770 needed for the same reason. The probe reads through tx, never the pool (BUG-2409). On retry exhaustion the RENAME fails rather than leaving one linker holding a title that no longer exists. The alternative — log and continue — was considered and rejected: it trades a loud retryable failure for a silent inconsistency, and a rename is atomic in intent. Exhausting three attempts needs three consecutive commits to the same linker inside one cascade. Adds afterLinkCascadeRead, the seam between the cascade's read and its writes. No existing seam reaches that gap: afterDocumentPreLockRead fires before the transaction, afterDocumentPreWrite before the renamed document's own update. That new seam also closes a gap a previous unit recorded as permanently open: the `deleted_at IS NULL` guard carried a note calling itself UNTESTED because reaching its window "would cost a fifth seam". This is that fifth seam, so the note is removed and replaced by a record of the closure, and the archived-linker test now drives exactly that window. Mutation matrix, run under Postgres — 5 mutants, 5 detected, including one that is literally the pre-fix code: M1 CAS predicate removed (the unfixed behaviour) → lost-update test M2 soft-delete probe arm removed → archived-linker test M3 retry budget cut to one attempt → lost-update test M4 retry replays the original rewrite → lost-update test M5 seam removed (control: tests must notice they never raced) → lost-update test Gates: gofmt clean, lint 0 issues, govulncheck clean, full suite green on SQLite (28 pkgs) and on Postgres 17 (28 pkgs, internal/store 346s vs 79s — the positive control that the PG legs ran rather than skipped). Not fixed here, filed as BUG-2795: cascadeTitleRename, the ITEM-side cascade, has the identical defect and no lock closes its window either. Its fix does not transfer — it rewrites by POSITION from item_wiki_links offsets, so a retry must re-derive positions from the winner's body and re-run replaceWikiLinks, which is a redesign of the retry unit rather than a predicate on an UPDATE. wiki_links.go's claim that it "matches the document rename behavior" is corrected to say which half no longer matches. * style(store): gofmt the CONVE-23 sweep comment (BUG-2785) Same failure as the previous unit, same cause, and worth naming rather than quietly fixing: gofmt wants a blank line between list items once one item grows a second paragraph, which the note about the previously- untested deleted_at guard made true. I re-ran build and the full Postgres suite after those comment edits but not lint, because the change was 'only a comment' — the exact reasoning the previous unit's fix commit warned about in writing, one unit earlier. The PR body's claim that all gates were re-run after the prose edits was false and has been corrected there too. The rule as remembered does not work. The mechanical form does: gofmt and lint are the last action before a push, comment-only changes included. * fix(links,server): stop a rename hanging the server, and report cascade contention honestly (BUG-2785) Three findings from Codex round 2 on this PR. The first is a server hang. 1. ReplaceTitle could never terminate. replaceAll looped "find old in result, splice new in" — re-searching the string it was building, including the text it had just inserted. When the NEW title contains the OLD link token it grows without bound. Measured, not argued: ReplaceTitle("x [[A]] y", "A", "A]] [[A") builds `[[A]] [[A]]`, which still contains `[[A]]`; a probe against the old implementation ran 3s without terminating before being killed. Document titles have no validation, so this is reachable from user input — and the caller is inside the rename transaction holding the workspace rename advisory lock (BUG-2778), so the hang would take every other rename in that workspace down with it while exhausting memory. strings.Replace with n = -1 has the semantics that were wanted: non-overlapping, left-to-right, over the input. Three-case regression test, all three of which fail against the old implementation, plus a control that catches a "fix" which terminates by doing nothing. Pre-existing, and folded in rather than filed: three lines against a server hang, and this PR's retry calls the helper again per attempt, which makes it reachable more often than before. 2. Retry exhaustion surfaced as an opaque 500. The rename rolls back cleanly and retrying can succeed, so "an internal error occurred" tells the caller the opposite of the truth. Adds the exported ErrLinkCascadeContention sentinel; the handler now answers 503 lock_contention with Retry-After, reusing the disposition BUG-2778 already established for 55P03/40P01. 3. That 503's message claimed the workspace was "busy with another rename". 55P03 there is just as likely to be an ordinary content edit holding the row, and the new arm is definitely one. It no longer names a cause the server has not established. Also closes the coverage gap round 2 named around this unit's own decision: exhaustion now has a test asserting the rename ROLLS BACK (target keeps its title) and that the concurrent editor's text survives that rollback. cascadeRewriteAttempts becomes a var so the test can force exhaustion at 1 rather than arranging three consecutive commits, which would need a per-attempt hook in production code — the divergence from its const sibling is noted where it lives. Records two limitations in the code rather than leaving "the cascade is safe now" to rot: the MIRROR direction is still open (a content writer that read before this transaction can commit afterwards and reinstate the old title — fixing it means giving ordinary content writes a CAS too), and delete-then-restore of a linker mid-rename brings back the old title. Both pre-existing, neither worsened here. Mutation matrix now 7 mutants, 7 detected, run under Postgres: M6 (%w -> %v, sentinel lost) and M7 (exhaustion swallowed) cover the new mechanisms. Gates re-run on the tree being pushed, tests included, after the final comment edit rather than before it: gofmt clean, lint 0 issues, SQLite 28/28, Postgres 17 28/28 (internal/store 345s vs 79s). gofmt caught an unformatted test file locally this time, which is the point. * fix(store,links): correct three prose claims and close the SQLite coverage gap (BUG-2785) Codex round 4, on SQLite semantics and prose accuracy. One finding says a bug I FILED is wrong; that is the important one. 1. BUG-2795's premise was false, and this PR repeated it in a comment. I filed that item claiming the item-side cascade has "the identical defect" and that "no lock closes its window either", on the strength of a grep for pg_advisory_xact_lock in items.go that turned up only the parent-link locks. It missed acquireWorkspaceSeqLock (items.go:2136), taken UNCONDITIONALLY by every UpdateItem — content-only edits included — immediately after Begin and long before cascadeTitleRename, and held to COMMIT. So two item updates in a workspace fully serialize on Postgres and an ordinary content edit CANNOT commit inside that cascade's window. The scenario I filed is not reachable. Not fully invalid: sweeping every `SET content` writer in internal/store finds RemapAttachmentReferencesInWorkspace, which rewrites items.content in its own transaction without that lock. So a real but far narrower window survives — attachment remap versus cascade, not user-edit versus cascade. BUG-2795 corrected on its trail and dropped to low; the comment here now states the lock, the one surviving writer, and stops claiming parity with the document cascade. I searched for the locks I expected rather than for what serializes that path, then wrote a sentence broader than the search. Same failure this PR's review has produced repeatedly. 2. "A title nobody should be able to write" was false — the document API validates doc_type and status, never the title. Correcting it surfaced a real second defect: renaming to `A]] [[A` now terminates (round 2's fix) but writes `[[A]] [[A]]`, two links to nothing. Filed as BUG-2796. The termination test deliberately still asserts the COUNT rather than the output, so it does not freeze today's broken rendering as intended behaviour. 3. The hang's blast-radius claim named the workspace rename advisory lock without qualifying the dialect. That lock is a no-op on SQLite, where the equivalent damage is the database-wide write lock the transaction already holds under BEGIN IMMEDIATE. Different mechanism, same outcome for everyone else. Also closes the last coverage gap round 2 named. The CAS predicate runs on SQLite in production and every concurrency test skips there, so TestUpdateDocument_CascadeRewritesEveryLinkOnBothDialects does not skip: two linkers, multiple links per body, plus a document that merely contains the word and must be left alone. Verified it earns its place — a mutant comparing against the rewritten body instead of the body that was read compiles, and dies to this test ON SQLITE, where nothing else would have caught it. Documents the parallel-test constraint on the now-mutable cascadeRewriteAttempts, with its boundary: no current t.Parallel test in internal/store reaches this cascade, but that is a fact about today's corpus rather than an invariant. Gates on the pushed tree: gofmt clean, lint 0 issues, govulncheck clean, SQLite 28/28, Postgres 17 28/28 (internal/store 349s vs 79s). * docs(store,links): point the cascade comment at the root cause, qualify a claim in its second location (BUG-2785) Codex round 5, probing cross-connection visibility and transaction boundaries. Two findings, both mine, neither changing behaviour. 1. The "one surviving writer" framing was too comfortable, twice over. RemapAttachmentReferencesInWorkspace is not a rare non-interactive writer: bundle import reaches it on an ordinary user-triggered import, and the workspace is ALREADY VISIBLE to its owner while it runs — store.ImportWorkspace commits the workspace row (with owner_id) in its own transaction before opening the one that inserts items, and the bundle handler runs the remap as Phase 3 afterwards. Both verified in code. So that writer races ordinary item edits, not just the rename cascade, and it is missing a guard outright rather than being an exotic pairing. Filed as BUG-2797, which covers the remap itself; BUG-2795 is now a consequence of it and says so on its trail. The comment here points at the root cause instead of implying the cascade's pairing is the whole story. 2. The dialect-unqualified advisory-lock claim survived in a SECOND location. Round 4 caught it in links.go and I fixed it there; the same sentence sat in the termination test's comment, and I never enumerated the sites. That is CONVE-23's verify half failing exactly as it warns: I fixed the instance I was shown rather than the population. Both now qualified, and a sweep for remaining unqualified copies leaves only BUG-2778's own comment, which already carries its no-op-on-SQLite note. Worth recording that this is the second time on BUG-2795 that a scope sentence of mine ran ahead of the sweep supporting it — first "no lock closes its window either" (acquireWorkspaceSeqLock did), now "not user triggered" (import is). Both found by a review round rather than by me. Gates on the pushed tree: gofmt clean, lint 0 issues, SQLite 28/28, Postgres 17 28/28 (internal/store 348s vs 79s).	2026-08-26 22:04:15 -04:00
xarmian	773222368c	fix(store): serialize document renames and stop reading the pool inside transactions (BUG-2778) (#1208) ... Two deadlocks, one in the database and one in the application. THE DATABASE ONE, which is what BUG-2778 was filed about. A document rename takes row locks in two stages inside one transaction: updateLinksInTx writes every OTHER document whose content links the old title, then the final UPDATE writes THIS document. Two concurrent renames of documents that link to each other therefore take the same two locks in opposite orders, and Postgres aborts one with SQLSTATE 40P01 — a 500 on an ordinary rename. A throwaway probe against the unfixed code deadlocked on 12 of 12 rounds; this is deterministic, not theoretical. The fix serializes renames per workspace with a dedicated advisory key (`pad:document-rename:<ws>`), taken before any row lock and whenever a title is supplied. No-op on SQLite, whose single writer cannot produce the cycle. `SET LOCAL lock_timeout = '5s'` bounds the wait, because a transaction that waits with a pool connection already in hand converts contention into pool exhaustion; the handler maps 55P03 and 40P01 to a retryable 503 rather than a generic 500. WHY NOT `ORDER BY id`, which is what I proposed when I FILED this from reading rather than from a repro: each transaction's cascade set is a single row, and the cycle is cascade-then-self, so ordering the cascade leaves it exactly as reachable. Run as a mutation, that fix fails the regression test. Reproducing the bug is what refuted my own diagnosis. THE APPLICATION ONE, found in review and larger than the filed bug. Seven production paths issued a read through the connection POOL from inside an open transaction. Under a saturated pool the second connection never arrives, so the transaction cannot finish and never releases what it holds — no SQLSTATE names this and no lock timeout breaks it. All seven now take their executor: the document and item slug scans, both version checks, the done-field lookup on the item update and move paths, the open-children guard's collection read, and the OAuth startup backfill's workspace lookup. Three of those were found only after asking for the true population rather than for a sample; two were INDIRECT (through GetCollection), which a grep for `s.db` inside transaction bodies cannot see. The instrument is a one-connection pool, which makes the hazard deterministic instead of load-dependent. ALSO FIXED, adjacent and found by the same reviews: - The rename decided from a PRE-LOCK snapshot: the lock made the cascade safe against another rename and then handed it a stale OLD TITLE. It now re-reads under the lock and decides from that row. - A concurrent soft-delete could commit mid-rename, leaving the cascade's rewrites behind while the caller was told not-found. The final UPDATE now carries `deleted_at IS NULL` with a checked row count, so the rename and its cascade land together or not at all. - Class sweep (CONVE-18) of the same unordered-scan-then-per-row-update shape: the attachment remap (items and comments) and the outbox user-ref scrub are now ordered. Scoped claim — it orders those per-row updates, not every lock in those transactions — and NOT reproduced, unlike the rename. 18 mutations aimed, 15 die. The three survivors are written down where they live: one is an equivalent mutant (the cascade is the first row-locking stage, so "lock before the cascade" and "lock at the top of the transaction" are the same order), one is a guard covering a window no seam can currently schedule, and one is a class-sweep fix with no repro.	2026-08-25 22:40:58 -04:00
xarmian	7680919dbd	fix(store): re-stamp attachment refs on item/document restore so a racing GC claim is refused (BUG-2629) (#1155) ... An attachment referenced only from soft-deleted content was reclaimable by the orphan GC: AttachmentReferenced scans LIVE rows only, so the archived reference is invisible to the sweep, and past the grace period the claim reclaims the blob. On restore the reference is live again and dangles. The reachable case is a never-attached upload (item_id NULL) referenced from a document — documents have no document_id column, so such a row is necessarily never-attached and the ClaimNeverAttachedAttachment predicate applies with nothing else standing in its way — or from an item's content where the attachment was uploaded unattached. RestoreItem and RestoreDocument now call stampAttachmentRefsTx before clearing deleted_at, inside the restoring transaction, per that helper's ORDERING contract: the stamp row-locks the attachment, so a GC claim racing the restore blocks until commit and re-evaluates last_referenced_at against the fresh stamp — refusing. RestoreItem stamps content + fields; RestoreDocument (previously a bare Exec) is wrapped in a transaction that reads the soft-deleted content + workspace and stamps content. This is prevention only: a blob already reclaimed before the restore is gone (the claim is irrevocable by design), and the restore-time stamp matches zero rows. Surfacing an already-dangling reference to the user on restore is tracked separately as IDEA-2646. Regression tests archive content holding the only reference, age the stamp past the claim's stale window, restore, then run the claim directly (the sweep's live scan would protect now-live content and pass for the wrong reason). All three legs fail on unfixed code. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V	2026-08-18 14:43:13 -04:00
xarmian	2c8ddffcb0	fix(store): cover documents and comment bodies in the attachment reference walks (BUG-2614, BUG-2615) (#1145) ... * fix(store): cover documents and comment bodies in the attachment reference walks (BUG-2614, BUG-2615) Two defects with one shape: a content surface that carries `pad-attachment:` references was missing from a walk meant to cover every such surface. Both were found by Codex during BUG-2415 and both predate it. BUG-2614 — the orphan GC could reclaim a live reference. AttachmentReferenced counted items and comments; documents.content was never scanned, and neither document write path stamped. An attachment referenced only from a document was therefore both invisible to the sweep's scan AND unprotected by the stamp that covers references landing mid-sweep. The filing asked whether the documents surface is dead enough to delete instead of widening the scan. Evidence says widen, and I am not making the deletion call inside a bug fix: /workspaces/{ws}/documents has full CRUD mounted and authenticated today (list/create/get/patch/delete plus restore, versions and activity), so a direct API consumer can still write one. It IS legacy — the route block says "v1 — will be replaced by items in Phase 2" and no first-party client reaches it (zero references in the web API client and in cmd/pad) — and production carries 4 document rows, all soft-deleted, none referencing an attachment, newest touched 2026-04-27. "Reachable but unused by us" is not dead, and the conservative fix is a few lines. Retiring the surface belongs with the Phase 2 migration, deliberately. CreateDocument had no transaction, so it gains one: the stamp has to commit atomically with the content carrying the reference or it cannot serialize against a concurrent claim, which is the whole point. UpdateDocument already had a transaction and only needed the call — and only when content is actually written, since a metadata-only PATCH neither adds nor keeps a reference and must not vouch for one. BUG-2615 — the bundle import's remap rewrote item content and fields but not comment bodies, so an imported comment kept the SOURCE workspace's ids: broken references in the destination, and the rehydrated rows they should point at left referenced by nothing. Bundles do carry comments (export.go exports them, ImportWorkspace re-inserts them); they carry no documents, so this stays scoped to comments. The remap also now stamps what the rewrites point AT. ImportWorkspace already stamps each comment body at insert, but the body still holds the source ids then and the remap runs later in the handler, so those stamps land on nothing that ends up referenced — leaving a fresh clone referenced only by text the transaction just wrote and carrying no stamp, which is exactly the shape the never-attached claim reclaims. The REWRITTEN TEXTS are passed rather than every id in the map, so a clone nothing references is not vouched for and does not survive an extra GC window. Seven negative controls, one mutation at a time, each failing exactly the test that covers it: the documents scan leg, each of the two stamps, the comment write-back (at store and end-to-end level), the remap stamp, and an over-broad stamp-the-whole-map variant that the precision test catches. Per the standing bar out of BUG-2301, every regression test here was RUN against the unfixed code and observed to fail — including the end-to-end bundle fixture the filing asked for, whose item deliberately carries no reference so that the items walk alone cannot rescue it. * fix(store): stamp before the remap's content writes, and state the caller precondition (BUG-2615) Codex round 2, two P1s. The first is mine and is a straight violation of the protocol I was mirroring: I stamped AFTER the item and comment UPDATEs. stampAttachmentRefsTx's own contract says to call it before the content statement, for two reasons that both bite here. On Postgres the stamp row-locks the attachment rows for the rest of the transaction, so a concurrent GC claim blocks and re-evaluates against the fresh stamp — stamping last instead lets a claim delete the target while the rewritten text is still uncommitted, after which the stamp matches zero rows and the transaction commits a dangling reference. And every other writer takes attachments before content rows, so writing content first inverts the lock order and deadlocks against them. The texts are known as soon as both scans finish, so the stamp simply moves up. The second — the scan-then-write over comments has no row lock and no old-value predicate, so a concurrent edit committed in between is clobbered — is real as a shape but not reachable at the only call site, and is NOT fixed here. The bundle import runs this against a workspace it has just created, which no other session can reach yet: there is no concurrent writer to lose an edit to, and no contention for the long transaction to hold up. The pre-existing items walk has the identical shape, so this is a property of the function rather than of the comment leg. Adding row locks or a compare-and-swap would be machinery for an unreachable window. What that argument does require is that the precondition stop being tribal knowledge, since it is about the CALLER and the next caller is exactly who would break it. It is now stated at the top of the function, where someone adding a second call site reads it, rather than in this message. Also declined, both pre-existing and neither introduced here: the one-transaction scan of the whole population (same reasoning — one caller, fresh workspace), and document slug allocation outside the create transaction, which predates the transaction existing at all and yields a spurious unique-violation rather than partial state. NOT COVERED BY A TEST, stated rather than implied: the stamp ORDERING. The existing guard asserts the stamp is present and fails without it, but it reads end state, so it cannot distinguish before-the-writes from after. Proving the order needs a concurrent-session Postgres instrument of the kind BUG-2409 used; that is not built here. The ordering rests on the reasoning above and on the contract documented at stampAttachmentRefsTx. * docs(store): make the scanned-surface set an explicit contract (BUG-2614) Codex round 3 P2. Both comments a maintainer reads still described the scan as covering items and comment bodies — AttachmentReferenced's doc, and the orphan-GC sweep's "Item content references the attachment" branch — so the change that added documents left the two artifacts that explain it stale. Same class as the sentinel comment on BUG-2301: the code was right and the text someone acts on was not. They now also say the thing neither said before, which is why this defect happened twice: the SET of scanned surfaces is the contract. Any surface that persists user-authored text containing a `pad-attachment:` token has to be listed there, and adding one without adding it here silently makes its references invisible to the GC. Comments (IDEA-1650) and documents (BUG-2614) were both found after the fact, which is the argument for writing the rule down rather than the two instances. Round 3's P1 — restore paths do not re-stamp, so a reference reclaimed while archived is dangling after restore — is filed as BUG-2629, not fixed here. It is pre-existing and uniform: RestoreItem does not stamp either, so fixing only RestoreDocument would leave the larger hole open while making documents inconsistently better-protected. The filing records the asymmetry that decides its priority: items are usually shielded by the claim's own item_id IS NULL predicate, while a document-referenced attachment has no document_id column to be shielded by and is always claimable. * docs(store): mark the unstamped rename cascades in place, pointing at BUG-2629 (BUG-2614) Codex raised the title-rename cascade's missing stamp in two separate rounds despite being told it was filed. Being raised twice is the signal that the disposition was only in a bug tracker and not where a reader of this code meets the problem — the same correction BUG-2301 ended on. Both sites now carry it: documents.go::updateLinksInTx and wiki_links.go::cascadeTitleRename, each naming BUG-2629, why it is not fixed here (uniform across both surfaces, so half-fixing makes them inconsistent), and why it is the weakest member of that family (the cascade rewrites link text in content whose references were already stamped and are still visible to the scan, so a genuinely new reference needs a title containing a pad-attachment token). Comments only.	2026-08-17 14:26:09 -04:00
xarmian	0fd5d0cdfb	fix: green up Go (PostgreSQL) CI (BUG-842) (#275) ... * fix(store): swap plainto_tsquery → websearch_to_tsquery for PG FTS (BUG-842) `TestListItems_FTS_HyphenatedSearchTerm/task-five` has been failing on every Go (PostgreSQL) CI run because `plainto_tsquery('english', 'task-five')` doesn't match the asciihword lexeme(s) the english parser produces for an indexed `task-five-distinctive`. The result is that every PG full-text search for hyphenated terms returns zero rows. `websearch_to_tsquery` (Postgres 11+) is purpose-built for arbitrary user input and tokenizes hyphenated terms the same way `to_tsvector` does for the indexed document, so the query intersects the index correctly. Swapped in three spots in the postgres dialect — FTSMatch, FTSSnippet, FTSRank — and updated the caller-side comments that referenced plainto_tsquery. SQLite path is unchanged: it goes through items_fts MATCH with sanitizeFTSQuery, never through these methods. * fix(server): drain background goroutines on Stop() (BUG-842) `TestAdminBillingStats_SidecarSidecarError_DegradesToLocalOnly` (and other server tests) have been flaking on the Go (PostgreSQL) CI runner with `TempDir RemoveAll cleanup: directory not empty`. Root cause: several request handlers spawned bare `go func() { ... }()` goroutines that touched the SQLite WAL DB after the test function returned. testServer's t.Cleanup closed the store but had no way to drain those goroutines first, so a fire-and-forget WAL write could re-create the `-wal`/`-shm` files between Close() and t.TempDir's RemoveAll. Add a Server.bg sync.WaitGroup, a Server.goAsync helper that wraps a WaitGroup-tracked goroutine, and a Server.Stop() that blocks until every goAsync closure has finished. Convert the four known fire-and-forget sites to goAsync: - middleware_auth.go (TouchUserActivity) - handlers_auth.go (password reset email) - handlers_cloud.go (stripe_processed_events pruning) - handlers_members.go (workspace invitation email) Wire `srv.Stop()` into both testServer (server_test.go) and newMetricsTestServer (metrics_auth_test.go) so cleanup order is Stop → Close → TempDir RemoveAll. Add TestServer_Stop_DrainsBackgroundGoroutines to pin the contract: a goAsync goroutine must block Stop until it returns. * fix(store): correct PG FTS hyphenation via OR-combined plainto_tsquery (BUG-842) The previous attempt swapped plainto_tsquery → websearch_to_tsquery, which was wrong: websearch_to_tsquery treats `-` as a NEGATION operator (Google-style), so `task-five` becomes `task & !five` and the search returns 0 rows for the same reason as before. This commit reverts the swap and applies the actual fix. PG's english parser indexes `task-five-distinctive` as `{task-five-distinct, task, five, distinct}` — the asciihword AND its parts. plainto_tsquery applied to the partial query `task-five` produces `task-fiv & task & five`: the stemmed asciihword for the PARTIAL query (`task-fiv`) is NOT in the vector, so the AND fails. Replacing the hyphen with a space makes plainto emit `task & five`, which DOES match — but doing that unconditionally breaks `BUG-842`- style queries: PG indexes the `-842` suffix as a negative-number lexeme, so `plainto_tsquery('BUG-842')` matches via `-842`, while `plainto_tsquery('BUG 842')` searches for `842` and misses. The fix ORs the two query variants together so the search vector is matched against either the raw user query OR its hyphen-as-space form. Both `task-five` (against `task-five-distinctive`) and `BUG-842` (against `BUG-842 fix the cleanup race`) hit. Verified locally against postgres:17-alpine via PAD_TEST_POSTGRES_URL — both 10x stress and race-detector runs are green. Surfaces: - dialect.go: FTSMatch / FTSSnippet / FTSRank now consume TWO placeholders each in the PG dialect. - items.go: listItemsFTS PG branch + SearchItems PG branch update args to pass (raw, sanitized) for every PG `?` placeholder. - search.go: SearchItems main / count / facets PG branches updated likewise. New sanitizePGFTSQuery helper alongside sanitizeFTSQuery. - documents.go: ListDocuments PG branch updated. Tests: - TestListItems_FTS_HyphenatedSearchTerm extended with a `BUG-842` case to pin the OR-combined logic — naive hyphen-stripping would silently regress this. - New TestSanitizePGFTSQuery unit test. * chore: gofmt 11 files with import-order issues (BUG-842 PR cleanup) The Go (SQLite) CI job has been failing on `main` (and every PR built against it) because golangci-lint flags 11 files whose third-party imports are intermixed with internal imports — the import-grouping rule that gofmt enforces. None of these were introduced by the BUG-842 PR; they're pre-existing on main. The PR can't go green without this cleanup, though, so it's bundled here. Pure mechanical change — `gofmt -w <files>` only re-orders import groups; no logic changes. Files touched: cmd/pad/configure.go cmd/pad/main.go internal/cli/format.go internal/server/handlers_admin_invitations.go internal/server/handlers_admin_users.go internal/server/handlers_grants.go internal/server/handlers_share_links.go internal/server/handlers_stars.go internal/server/middleware_auth.go internal/store/store.go internal/store/store_test.go After this commit `gofmt -l ./cmd ./internal` returns clean.	2026-04-28 16:21:43 -04:00
xarmian	7cda0d7896	feat: rebrand to Perpetual Software + new tagline (IDEA-832) (#273) ... Migrates from xarmian/pad to PerpetualSoftware/pad across the entire repo and updates the product subtitle to "Collaborate with your AI agents". Go module rename - go.mod: github.com/xarmian/pad → github.com/PerpetualSoftware/pad - All Go imports updated across cmd/pad, internal/{cli,server,store, models,collections,items,events,metrics,webhooks} (~130 files) - Test fixtures with the literal repo slug ("xarmian/pad" in JSON shapes, SSH/HTTPS git URL strings, workspace_context fixtures) also updated, including the secondary repo entry (xarmian/pad-web → PerpetualSoftware/pad-web — pad-web was also moved to the org per branch context) Docs / config - README badges, install instructions, brew tap, Docker image, source build path, sponsor link (sponsor link kept as personal @xarmian) - Subtitle: "Project management for developers and AI agents." → "Collaborate with your AI agents." (README, manifests, web layout meta, .goreleaser homebrew description) - CONTRIBUTING.md, SECURITY.md, skills/INSTALL.md - .goreleaser.yaml: homebrew_casks owner, GHCR image, release github owner, cosign cert-identity regex, comments - .github/workflows/release.yml: tap/release comments - deploy/k8s/deployment.yaml: container image - docs/deployment.md: clone URL - web/static/{site.webmanifest,manifest.json}: description - web/src/routes/+layout.svelte: meta description + og:description Brew tap path is PerpetualSoftware/tap/pad (CamelCase, matches GitHub user case). GHCR image is ghcr.io/perpetualsoftware/pad (lowercased per GHCR's URL normalization). CODEOWNERS @xarmian and FUNDING.yml github: xarmian intentionally retained — those are the personal maintainer / sponsor account, separate from the org repo. Verification: go build ./..., go test ./... (all pkgs pass), web build, and make install all clean (TASK-844, TASK-845).	2026-04-28 12:26:39 -04:00
xarmian	dcf7c1d58e	fix(store): apply Tag and Pinned filters in ListDocuments FTS branch (BUG-820) (#263) ... The non-FTS path in ListDocuments applies Tag and Pinned filters (lines 31-42), but when params.Query is non-empty the FTS branch rebuilds query and args from scratch and only re-applies Type and Status — Tag and Pinned were silently dropped. Result: `/documents?q=foo&tag=urgent` returned all docs matching foo regardless of tag, similarly for pinned. Documents-side analog of BUG-812 (which fixed the equivalent issue on the items FTS path). Fix: mirror the Tag (s.dialect.JSONArrayContains on d.tags) and Pinned (d.pinned = TRUE/FALSE) filter blocks into the FTS branch after the existing Type/Status blocks. Backend-only — handlers and DocumentListParams already plumb both params through. Tests: - TestListDocuments_FTS_TagFilter — two docs match the search; only one has the tag; assert exactly the tagged one returned. - TestListDocuments_FTS_PinnedFilter — covers both pinned=true and pinned=false branches, asserting each narrows correctly. Manual verification: with two docs `BUG820scratch alpha` (tagged "urgent", pinned) and `BUG820scratch beta` (untagged, unpinned): - ?q=BUG820scratch → 2 docs - ?q=BUG820scratch&tag=urgent → 1 doc (alpha) - ?q=BUG820scratch&pinned=true → 1 doc (alpha) - ?q=BUG820scratch&pinned=false → 1 doc (beta)	2026-04-27 12:38:50 -04:00
xarmian	068c208824	fix: sanitize SQLite FTS5 queries + whitespace guards (BUG-818) (#261) ... * fix(store): sanitize FTS5 queries in listItemsFTS and SearchItems (BUG-818) The sanitizeFTSQuery helper in internal/store/search.go wraps each whitespace-delimited token in double quotes so SQLite FTS5 treats specials (hyphens, AND/OR/NOT, parens) as literal characters rather than boolean operators. Store.Search already used it; Store.listItemsFTS and Store.SearchItems didn't, so any hyphen in `?search=` returned HTTP 500 with "no such column: <suffix>" — including issue refs like TASK-5, kebab-case slugs, dates, etc. Apply sanitizeFTSQuery at the SQLite arg-binding sites in both unfixed functions. Postgres branches stay unsanitized: plainto_tsquery accepts arbitrary input safely (matches the existing pattern in Store.Search). Tests: - TestListItems_FTS_HyphenatedSearchTerm — exercises the listItems path on multiple hyphenated queries via a table-driven sub-test. - TestSearchItems_HyphenatedQuery — same regression on the SearchItems path used by /api/v1/search. - TestSanitizeFTSQuery — direct unit test covering empty, whitespace- only, plain word, hyphenated phrase, multi-token, FTS5 boolean operators (AND/OR/NOT), parens, embedded quotes (stripped), surrounding whitespace, and unicode. Manual verification: previously-500 queries now return 200 with results: /items?search=match-me → HTTP 200, 2 items /items?search=TASK-5 → HTTP 200, 8 items /items?search=pad-cloud → HTTP 200, 103 items * fix(store): address Codex review on PR for BUG-818 Codex review caught two extensions to the original BUG-818 fix: 1. MEDIUM — Store.ListDocuments (internal/store/documents.go) had the same FTS5 boolean-parser vulnerability as Store.listItemsFTS and Store.SearchItems before the original commit. Hyphenated /documents?q= queries (e.g. ?q=release-notes-q2) returned HTTP 500 with "no such column" the same way. Apply sanitizeFTSQuery in the SQLite branch; leave Postgres unchanged. 2. LOW — Whitespace-only queries collapse to empty after FTS sanitization, and SQLite FTS5 errors on `MATCH ''` with "syntax error near \"\"". Add TrimSpace guards at the routing/entry points: - listItems: route to FTS only if TrimSpace(Search) != "" - SearchItems: short-circuit to empty results - ListDocuments: same routing guard - Store.Search: short-circuit to empty results Tests: - TestListDocuments_HyphenatedQuery — regression on the documents FTS path - TestFTS_WhitespaceOnlyQuery_DoesNotCrash — covers all 3 entry points (ListItems, SearchItems, ListDocuments) for spaces, tabs, mixed whitespace Manual verification (all 6 endpoints now HTTP 200): - /workspaces/{ws}/items?search=task-five - /workspaces/{ws}/items?search=<3 spaces> - /workspaces/{ws}/documents?q=release-notes - /workspaces/{ws}/documents?q=<3 spaces> - /search?q=task-5 - /search?q=<3 spaces>	2026-04-27 12:19:18 -04:00
xarmian	dd381e1066	chore: delete 5 unwired document handlers (TASK-769) (#252) ... * chore: delete 5 unwired document handlers (TASK-769) internal/server/handlers_documents.go had 5 dead HTTP handlers that were drafted as Documents-v1 extensions but never wired into the router (server.go:509 already labels Documents itself as "v1, will be replaced by items in Phase 2"): - handleQuickSave (POST /documents/quick-save) — title-based upsert - handleBulkRead (POST /documents/bulk-read) — multi-doc fetch by IDs - handleGetBacklinks (GET /documents/{id}/backlinks) - handleGetLinks (GET /documents/{id}/links) - handleGetContext (GET /documents/context?type=) Investigation confirmed zero consumers: - Not registered in setupRouter (`grep -n "QuickSave\|BulkRead\|Backlinks\|GetLinks\|GetContext" server.go` → empty). - Not used by the SvelteKit frontend (`web/src/`). - Not used by the CLI (`internal/cli/`). - Pre-launch repo, no fork or downstream that could be relying on them. Delete scope is intentionally limited to the HTTP handlers. The underlying `Store.QuickSave / BulkRead / GetBacklinks / GetLinks / GetContext` methods stay — they're tested at the store level (internal/store/store_test.go) and preserve optionality if Phase 2 work needs to revive any of these features. `models.QuickSave` stays for the same reason. After this lands, IDEA-732's lint catalog is fully cleared on main (staticcheck SA* + U1000 returns zero). TASK-771 (flip CI only-new-issues=false) becomes safe. Verified: - `go build ./...` clean - `go vet ./...` clean - `go test ./...` all pass - `staticcheck -checks "SA*,U1000" ./...` clean - All `import "strings"` etc. still used elsewhere in file Parent: PLAN-644. * chore: also delete now-test-only document store helpers (TASK-769) Codex round 1 on PR #252 flagged that the document-store helpers retained for "Phase 2 optionality" are now exclusively kept alive by their own store tests — Store.QuickSave, BulkRead, GetBacklinks, GetLinks, GetContext are not called by any production code path after the handler deletions in the previous commit. Same for the models.QuickSave struct. Pre-launch with no external consumers, optionality preservation has a real cost (dead code on main, ongoing test maintenance). When Phase 2 needs any of these capabilities it is cheaper to re-derive them against the Items model than to drag dead Documents-v1 plumbing forward. So delete them now. Removed: - internal/store/documents.go: QuickSave (38 lines), BulkRead (28), GetBacklinks (15), GetLinks (28), GetContext (41). - internal/models/document.go: QuickSave struct. - internal/store/store_test.go: TestQuickSave (38 lines), TestBulkRead (16), TestDocumentLinking (29), TestContext (23). Kept: - TestDocumentLinkRename — exercises UpdateDocument's internal link-rewriting path, not any of the deleted helpers. - GetDocumentByTitle — still used by TestDocumentLinkRename. - The full CRUD/restore handlers and their store methods — these are still wired into setupRouter and have their own coverage. Verified: - `go build ./...` clean - `go vet ./...` clean - `go test ./...` all pass (TestDocumentLinkRename and the wider doc CRUD/version/activity tests still cover the surviving paths). - `staticcheck -checks "SA*,U1000" ./...` clean - No new unused imports introduced (links package is still used by documents.go for ReplaceTitle in UpdateDocument). Parent: PLAN-644. * chore: drop GetDocumentByTitle and refactor TestDocumentLinkRename (TASK-769) Codex round 2 caught the chain — after deleting QuickSave/BulkRead/ GetBacklinks/GetLinks/GetContext, Store.GetDocumentByTitle was kept alive by exactly one test (TestDocumentLinkRename), which was re-fetching by title only because the test ignored the *Document already returned by createTestDoc. Use the createTestDoc return value instead, then drop GetDocumentByTitle from the store. Same idea, cleaner test, one fewer test-only API on the store. The rename behaviour (the actual thing under test) is unchanged. Verified: - `go build ./...` clean - `go test ./internal/store` and `./internal/server` pass - `staticcheck -checks "SA*,U1000" ./...` still clean Parent: PLAN-644. * chore: drop now-orphaned links.Extract (TASK-769) Codex round 3 caught the next link in the chain: after Store.GetLinks was deleted, links.Extract had no remaining callers — links.ReplaceTitle is the only Extract-package function still used (by UpdateDocument's rename rewrite). The linkPattern regex was only used by Extract. Drop linkPattern, the regexp import, and Extract itself. Leaves ReplaceTitle and its private string helpers (replaceAll, indexOf) intact. The cleanup chain ends here: ReplaceTitle is still wired into a live production path (Documents-v1 rename), and the supporting helpers have no other roles to inherit. Verified: - `go build ./...` clean - `go test ./internal/store` and `./internal/server` pass - `staticcheck -checks "SA*,U1000" ./...` clean Parent: PLAN-644.	2026-04-25 12:22:53 -04:00
xarmian	157ca4e88f	chore: bump Go toolchain to 1.26 (TASK-763) (#247) ... * chore: bump Go toolchain to 1.26 (TASK-763) Bump Go from 1.25 to 1.26 across all toolchain pins: - go.mod — go 1.25.0 → go 1.26.0 - Dockerfile — golang:1.25-alpine → golang:1.26-alpine - .github/workflows/ci.yml — three setup-go steps (Go, Go-Postgres, E2E jobs) - .github/workflows/release.yml — release pipeline No `toolchain` directive: the repo is pre-launch with no external contributors yet, so we set the floor where we want it (hard requirement). Verified locally before commit: - golangci-lint v2.11.4 builds and runs under Go 1.26.2 (pinned in CI) - golang:1.26-alpine and 1.26.2-alpine images present on Docker Hub - go build ./... clean - go vet ./... clean - go test ./... all pass Parent: PLAN-644 (OSS Repo Hygiene and Launch Polish). * chore: gofmt -w under Go 1.26 (TASK-763) Apply Go 1.26's gofmt to the codebase. ~41 files reformatted, all struct-tag whitespace realignment — no semantic changes. Verified: - gofmt -l ./cmd ./internal returns empty after - go build ./... still clean - go test ./... still passes (run before commit) Bundling the gofmt diff with the toolchain bump in the same PR because the formatting drift is a direct consequence of moving from 1.25 to 1.26; splitting them creates a mandatory two-PR ordering for no value. Parent: PLAN-644. * docs: bump documented Go floor to 1.26 (TASK-763) Match go.mod's hard 1.26.0 requirement in the source-build instructions. Caught by Codex review round 1 on PR #247. - README.md:158 — "Go 1.25+" → "Go 1.26+" - CONTRIBUTING.md:9 — "Go 1.25+" → "Go 1.26+"	2026-04-25 11:35:19 -04:00
xarmian	935b3d7b0e	fix: PG agent role reorder, JSONB tag filters, Redis credential leak ... - Rebind prepared statements in agent role and card reordering so PostgreSQL receives $1/$2 instead of ? placeholders - Add JSONArrayContains dialect method: SQLite uses LIKE, PostgreSQL uses jsonb @> operator — fixes tag filtering on JSONB columns - Redact Redis credentials from startup log: log addr+db only, not the full connection URL which may contain passwords	2026-04-06 00:20:20 +00:00
xarmian	fa3aee6561	fix: SQL injection in field filters, document search ranking, item search ordering ... - Sanitize field-filter keys from query params before interpolating into JSON path expressions — prevents SQL injection via crafted query parameter names (affects both SQLite and PostgreSQL) - Fix document search ORDER BY rank on PostgreSQL — the PG query path doesn't expose a `rank` column; use ts_rank() with DESC - Fix item search rank ordering in ListItems and SearchItems for PostgreSQL — ts_rank() needs DESC (higher = more relevant)	2026-04-05 23:59:53 +00:00
xarmian	55101e9680	fix: remaining PostgreSQL boolean/ranking issues from Codex review ... - Convert isDiff int→bool in version inserts (items.go, documents.go) using s.dialect.BoolToInt() for cross-driver compatibility - Fix boolean parameter writes in export.go ImportWorkspace (is_default, pinned, is_diff all used int literals) - Fix webhook CASE expression: return FALSE instead of 0 for PostgreSQL BOOLEAN active column - Fix search rank ordering: PostgreSQL ts_rank() uses DESC (higher = more relevant) vs SQLite bm25() ASC (more negative = better) - Fix pinned WHERE clauses: use TRUE/FALSE instead of 1/0 for PostgreSQL BOOLEAN compatibility	2026-04-05 20:55:54 +00:00
xarmian	63d5cb7b73	fix: address Codex review findings for PostgreSQL compatibility ... P1: Wrap store helper queries (uniqueSlug, uniqueSlugExcluding, backfillItemNumbers) with s.q() for placeholder rebinding. P1: Replace boolToInt() with s.dialect.BoolToInt() so pgx receives native booleans instead of 0/1 integers. P1: Change boolean scan variables from int to bool to match PostgreSQL's native boolean type. P1: Fix FTS table aliases in PostgreSQL search branches. P1: Make api_tokens.workspace_id nullable for user-scoped tokens. P2: Move eventBus.Close() before srv.Shutdown() so SSE handlers drain before the HTTP server shutdown deadline.	2026-04-05 20:45:38 +00:00
xarmian	a4a701367a	feat: add PostgreSQL support with dual-driver store layer (TASK-157) ... - Create Dialect abstraction for SQLite/PostgreSQL SQL differences (JSON ops, FTS, placeholders, datetime, aggregation) - Add Store.NewPostgres() constructor with connection pooling - Create consolidated PostgreSQL schema (pgmigrations/001_initial.sql) with tsvector FTS, JSONB columns, and GIN indexes - Refactor all store queries (~150) to use s.q() for placeholder rebinding - Replace hardcoded json_extract/FTS5/GROUP_CONCAT with dialect methods - Support PAD_DB_DRIVER=postgres + PAD_DATABASE_URL env vars - Keep SQLite as the default for local/self-hosted mode - Add dialect unit tests (rebind, SQLite, PostgreSQL)	2026-04-05 18:50:50 +00:00
xarmian	81579847c6	Initial release ... Pad — project management for developers and AI agents. Single Go binary with embedded SvelteKit web UI, SQLite storage, CLI, and Claude Code /pad skill integration. https://getpad.dev	2026-03-26 01:52:36 +00:00