mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-10 23:15:40 +00:00
fix(documents): count retained bytes, bound the retry path, escape the cascade's LIKE pattern (BUG-2798)
Codex round 1 on #1218. Three findings, all real, all fixed here. 1. The guard bounded projected OUTPUT, which bounds nothing when the new title is SHORTER than the old one. Renaming a 255-character title to a one-character title makes each 2 MiB linker project ~40 KiB while the cascade still retains its 2 MiB read for the compare-and-set, so hundreds of linkers exhaust memory while the counter reports well under the cap. The counter now sums RETAINED bytes — read plus written, both alive at once — so the cap is a statement about resident memory rather than about output. MaxRenameCascadeProjectedBytes becomes MaxRenameCascadeRetainedBytes and moves 16 -> 32 MiB, because the legitimate ceiling it clears doubles under the new metric (that instance's whole wiki-linking corpus retains ~20,154,952 bytes); the single-document attack retains 110,729,522, so it is still refused by 3.3x. 2. The compare-and-set's retry path bypassed the guard entirely. On contention it re-reads the linker and calls ReplaceTitle on whatever the winner wrote — a NEW input, bounded by nothing the scan had checked — so a content edit landing inside the cascade's window could grow a linker from harmless to enormous and walk the rename back into the amplification it would have been refused for. Each document's compare-and-set now carries the cap less what the other linkers hold, and re-checks the grown body against it. 3. The cascade's `content LIKE ?` search term went in unescaped, so a document TITLE decided how the pattern was read. `\` is the default LIKE escape character on Postgres and NOT on SQLite, so `[[Alpha\Beta]]` was searched for as itself on one dialect and as `[[AlphaBeta]]` on the other: linkers not found, cascade rewrites nothing, rename reports success, every link left stale. Silent and dialect-dependent. Codex named the backslash; `%` and `_` are the rest of the class (CONVE-18) — wildcards on both dialects, so a title carrying them selects documents that do not link it. An explicit `ESCAPE '\'` clause plus escapeLikePattern makes both dialects agree, rather than leaving SQLite correct by accident. Finding 3 also constrains finding 3 of the ORIGINAL fix: models' validator allows a lone backslash in a title on the grounds that both renderers handle it, which was true of rendering and false of cascading. That comment now records the dependency — allowing it is only correct while the cascade's pattern stays escaped. Tests, four new, each mutation-verified against the code it guards: - CountsRetainedBytesNotJustOutput — the shrinking rename. Asserts as a PRECONDITION that the projected-output total stays under the cap, so the test cannot pass for the old reason. - RetryRecheckesTheBudgetAgainstTheGrownBody — drives the real race through the afterLinkCascadeRead seam. POSTGRES ONLY and skipped loudly elsewhere: SQLite's BEGIN IMMEDIATE closes the window structurally, so a green run there would be a property of the DSN. - FindsLinkersWhoseTitleContainsABackslash — Postgres only, same reasoning inverted: SQLite is the dialect that was accidentally right. - DoesNotSpendTheBudgetOnDocumentsThatDoNotLinkTheTitle — `%` and `_`. Its first version asserted the decoy's content was untouched and passed against the unescaped pattern, because over-matched rows rewrite to themselves. The observable harm is that they spend the caller's budget, so that is what it now asserts. Mutation matrix for this round: output-only counter -> only the shrinking test fails; retry check removed -> only the retry test fails (PG); LIKE unescaped -> the budget legs fail on SQLite and the backslash test fails on PG. Gates: `go test ./...` under Postgres 17 EXIT=0; SQLite packages EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798, BUG-2796 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN
This commit is contained in:
@@ -138,7 +138,12 @@ const MaxDocumentTitleRunes = 255
|
||||
// "stored legacy titles that contain a literal `|`". Banning it would
|
||||
// refuse what that branch was written to support.
|
||||
// - a lone `\` not followed by `\`, `]` or `|` — passes the grammar as an
|
||||
// escape pair and survives the unescaper unchanged.
|
||||
// escape pair and survives the unescaper unchanged. Note this one depends
|
||||
// on store.escapeLikePattern: the rename cascade finds linking documents
|
||||
// with `content LIKE`, where Postgres reads `\` as an escape character, so
|
||||
// before that escaping landed a backslash title rendered fine and then
|
||||
// silently failed to cascade on one dialect. Allowing it here is only
|
||||
// correct while the cascade's pattern stays escaped.
|
||||
//
|
||||
// Boundary, stated rather than papered over: this is derived from the SHARED
|
||||
// stored-syntax path in markdown.ts. The legacy documents surface has no
|
||||
|
||||
@@ -109,8 +109,8 @@ func TestRenameCascadeTooLarge_IsPermanentShapedNotRetryable(t *testing.T) {
|
||||
// request cap, which is the point: the hostile input is cheap to deliver.
|
||||
const newTitleLen = 255
|
||||
const linkers = 3
|
||||
perDoc := (store.MaxRenameCascadeProjectedBytes / linkers) + (store.MaxRenameCascadeProjectedBytes / (linkers * 4))
|
||||
occurrences := perDoc / (5 + (newTitleLen - 1))
|
||||
perDoc := (store.MaxRenameCascadeRetainedBytes / linkers) + (store.MaxRenameCascadeRetainedBytes / (linkers * 4))
|
||||
occurrences := perDoc / (5 + 5 + (newTitleLen - 1))
|
||||
body := strings.Repeat("[[A]]", occurrences)
|
||||
|
||||
for i := 0; i < linkers; i++ {
|
||||
|
||||
+128
-45
@@ -564,13 +564,40 @@ func (s *Store) acquireWorkspaceDocumentRenameLock(tx *sql.Tx, workspaceID strin
|
||||
return nil
|
||||
}
|
||||
|
||||
// escapeLikePattern escapes the three characters that carry meaning inside a
|
||||
// LIKE pattern, for use with an explicit `ESCAPE '\'` clause.
|
||||
//
|
||||
// Without this the cascade's own search term is interpreted as a pattern, and
|
||||
// a document TITLE decides how (BUG-2798, codex round 1 P2 — plus the rest of
|
||||
// the class it was an instance of):
|
||||
//
|
||||
// - `_` and `%` are wildcards in BOTH dialects, so a title containing them
|
||||
// selects documents that do not link it. Those extra rows rewrite to
|
||||
// themselves, so the damage is not corruption — it is that the guard below
|
||||
// is computed from this result set, so an over-broad pattern spends a
|
||||
// caller's budget on rows that were never going to change.
|
||||
// - `\` is where the two dialects DISAGREE, which is the dangerous half.
|
||||
// Postgres LIKE treats backslash as the default escape character; SQLite
|
||||
// LIKE has no default escape character at all. So `[[Alpha\Beta]]` is
|
||||
// searched for as the literal it is on SQLite and as `[[AlphaBeta]]` on
|
||||
// Postgres — the linking documents are simply not found, the cascade
|
||||
// rewrites nothing, and the rename succeeds leaving every link stale. A
|
||||
// silent, dialect-dependent data defect.
|
||||
//
|
||||
// The explicit ESCAPE clause makes both dialects agree, rather than leaving
|
||||
// SQLite correct by accident and Postgres wrong by default.
|
||||
func escapeLikePattern(s string) string {
|
||||
r := strings.NewReplacer(`\`, `\\`, `%`, `\%`, `_`, `\_`)
|
||||
return r.Replace(s)
|
||||
}
|
||||
|
||||
func (s *Store) updateLinksInTx(tx *sql.Tx, workspaceID, oldTitle, newTitle string) error {
|
||||
// Find all documents in the workspace that contain [[oldTitle]]
|
||||
searchTerm := "[[" + oldTitle + "]]"
|
||||
rows, err := tx.Query(s.q(`
|
||||
SELECT id, content FROM documents
|
||||
WHERE workspace_id = ? AND deleted_at IS NULL AND content LIKE ?
|
||||
`), workspaceID, "%"+searchTerm+"%")
|
||||
WHERE workspace_id = ? AND deleted_at IS NULL AND content LIKE ? ESCAPE '\'
|
||||
`), workspaceID, "%"+escapeLikePattern(searchTerm)+"%")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -583,39 +610,48 @@ func (s *Store) updateLinksInTx(tx *sql.Tx, workspaceID, oldTitle, newTitle stri
|
||||
// the column handed us and never a normalized form of it.
|
||||
read string
|
||||
rewritten string
|
||||
// retained is what this row contributed to the running total, kept so
|
||||
// the compare-and-set below can be given ITS share of the budget when
|
||||
// a concurrent edit forces it to re-read and re-rewrite.
|
||||
retained int64
|
||||
}
|
||||
var updates []docUpdate
|
||||
var projected int64
|
||||
var retained int64
|
||||
for rows.Next() {
|
||||
var du docUpdate
|
||||
if err := rows.Scan(&du.id, &du.read); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Project this linker's rewritten size BEFORE building it, and refuse
|
||||
// on the running TOTAL across the linking set (BUG-2798).
|
||||
//
|
||||
// The quantity is exact rather than an estimate: strings.Replace
|
||||
// substitutes every non-overlapping occurrence, so the output is
|
||||
// len(read) + occurrences * (len(new) - len(old)) to the byte.
|
||||
// Project what this linker will make the cascade HOLD, before building
|
||||
// it, and refuse on the running TOTAL across the linking set
|
||||
// (BUG-2798).
|
||||
//
|
||||
// The total is the right thing to bound, and a per-document cap would
|
||||
// not be. Measured: with the title bound in place, one linker holding
|
||||
// the largest body a 2 MiB request can carry projects 108,632,370
|
||||
// bytes — 51.8x — and the cascade holds EVERY rewritten body in
|
||||
// `updates` before it writes any of them, so k linkers hold k times
|
||||
// that (measured linear at k = 1/2/4). A per-document cap of C still
|
||||
// admits k * C, which is the same unbounded shape one level up.
|
||||
// bytes of output — 51.8x — and the loop below holds EVERY rewritten
|
||||
// body in `updates` before it writes any of them, so k linkers hold k
|
||||
// times that (measured linear at k = 1/2/4). A per-document cap of C
|
||||
// still admits k * C, which is the same unbounded shape one level up.
|
||||
//
|
||||
// RETAINED bytes, not output bytes. An earlier version of this guard
|
||||
// summed only the projected output, which bounds nothing when the new
|
||||
// title is SHORTER than the old one: renaming a 255-character title to
|
||||
// a one-character title makes each 2 MiB linker project about 40 KiB
|
||||
// while the cascade still retains its 2 MiB read for the
|
||||
// compare-and-set, so hundreds of linkers exhaust memory while the
|
||||
// counter reports well under the cap (codex round 1 P1). Both strings
|
||||
// are alive at once, so both are counted.
|
||||
//
|
||||
// Refusing here rather than after the loop is what makes the bound
|
||||
// real: at the moment of refusal the process holds the linkers already
|
||||
// projected (under the cap by construction) plus this one row's body,
|
||||
// counted (under the cap by construction) plus this one row's body,
|
||||
// and none of the amplified output.
|
||||
occurrences := int64(strings.Count(du.read, searchTerm))
|
||||
projected += int64(len(du.read)) + occurrences*int64(len(newTitle)-len(oldTitle))
|
||||
if projected > MaxRenameCascadeProjectedBytes {
|
||||
return fmt.Errorf("%w: renaming to %q projects at least %d bytes across linked documents, maximum %d",
|
||||
ErrRenameCascadeTooLarge, newTitle, projected, MaxRenameCascadeProjectedBytes)
|
||||
du.retained = cascadeRetainedBytes(du.read, searchTerm, oldTitle, newTitle)
|
||||
retained += du.retained
|
||||
if retained > MaxRenameCascadeRetainedBytes {
|
||||
return newRenameCascadeTooLargeError(newTitle, retained)
|
||||
}
|
||||
|
||||
du.rewritten = links.ReplaceTitle(du.read, oldTitle, newTitle)
|
||||
@@ -636,13 +672,40 @@ func (s *Store) updateLinksInTx(tx *sql.Tx, workspaceID, oldTitle, newTitle stri
|
||||
}
|
||||
|
||||
for _, du := range updates {
|
||||
if err := s.rewriteLinkerCAS(tx, du.id, du.read, du.rewritten, oldTitle, newTitle); err != nil {
|
||||
// Budget handed to this document's compare-and-set: the cap, less
|
||||
// everything the OTHER documents are holding. A retry that re-reads a
|
||||
// body grown by a concurrent edit is checked against it, so the
|
||||
// aggregate bound survives the retry path as well as the scan (codex
|
||||
// round 1 P1 — the guard used to cover only the scan, and the retry
|
||||
// called ReplaceTitle on an unbounded re-read).
|
||||
budget := MaxRenameCascadeRetainedBytes - (retained - du.retained)
|
||||
if err := s.rewriteLinkerCAS(tx, du.id, du.read, du.rewritten, oldTitle, newTitle, searchTerm, budget); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// cascadeRetainedBytes is what one linking document makes the cascade hold:
|
||||
// the body it read (kept verbatim as the compare-and-set token) plus the body
|
||||
// it will write.
|
||||
//
|
||||
// Exact rather than an estimate. strings.Replace substitutes every
|
||||
// non-overlapping occurrence, so the rewritten length is
|
||||
// len(read) + occurrences * (len(new) - len(old)) to the byte, and this
|
||||
// function is the only place that arithmetic lives — the scan and the retry
|
||||
// path must not be allowed to drift apart on it.
|
||||
func cascadeRetainedBytes(read, searchTerm, oldTitle, newTitle string) int64 {
|
||||
occurrences := int64(strings.Count(read, searchTerm))
|
||||
rewritten := int64(len(read)) + occurrences*int64(len(newTitle)-len(oldTitle))
|
||||
return int64(len(read)) + rewritten
|
||||
}
|
||||
|
||||
func newRenameCascadeTooLargeError(newTitle string, retained int64) error {
|
||||
return fmt.Errorf("%w: renaming to %q would hold at least %d bytes of linked-document content, maximum %d",
|
||||
ErrRenameCascadeTooLarge, newTitle, retained, MaxRenameCascadeRetainedBytes)
|
||||
}
|
||||
|
||||
// ErrLinkCascadeContention reports that a rename's link cascade lost its
|
||||
// compare-and-set on the same linking document too many times in a row.
|
||||
//
|
||||
@@ -653,9 +716,8 @@ func (s *Store) updateLinksInTx(tx *sql.Tx, workspaceID, oldTitle, newTitle stri
|
||||
// opposite of the truth here (codex round 2 on BUG-2785).
|
||||
var ErrLinkCascadeContention = errors.New("store: link cascade lost the compare-and-set")
|
||||
|
||||
// ErrRenameCascadeTooLarge reports that a rename was refused because the work
|
||||
// it projects across linking documents exceeds
|
||||
// MaxRenameCascadeProjectedBytes.
|
||||
// ErrRenameCascadeTooLarge reports that a rename was refused because the
|
||||
// linked-document content it would hold exceeds MaxRenameCascadeRetainedBytes.
|
||||
//
|
||||
// Deliberately NOT in ErrLinkCascadeContention's family, and the distinction
|
||||
// is the caller-visible one: contention means "someone else got there first,
|
||||
@@ -665,37 +727,47 @@ var ErrLinkCascadeContention = errors.New("store: link cascade lost the compare-
|
||||
// the projection so the caller can see what it asked for. Blurring the two
|
||||
// vocabularies would tell a client to retry forever (BUG-2798, lead ruling
|
||||
// day-63).
|
||||
var ErrRenameCascadeTooLarge = errors.New("store: rename cascade exceeds the projected-output bound")
|
||||
var ErrRenameCascadeTooLarge = errors.New("store: rename cascade exceeds the retained-content bound")
|
||||
|
||||
// MaxRenameCascadeProjectedBytes bounds the TOTAL bytes a single rename may
|
||||
// project across every document linking the renamed title.
|
||||
// MaxRenameCascadeRetainedBytes bounds the TOTAL linked-document content a
|
||||
// single rename may hold in memory: for every linking document, the body read
|
||||
// plus the body written.
|
||||
//
|
||||
// 16 MiB, and the basis is measured rather than picked:
|
||||
// RETAINED rather than merely projected-output, because output alone is not
|
||||
// the resource. A rename to a SHORTER title projects less output than its
|
||||
// input while still holding every read body for the compare-and-set — so an
|
||||
// output-only counter reports ~40 KiB per 2 MiB linker and bounds nothing in
|
||||
// that direction (codex round 1). Counting both strings makes the cap a
|
||||
// statement about resident memory, which is what actually runs out.
|
||||
//
|
||||
// 32 MiB, and both bounds of the gap are measured rather than picked:
|
||||
//
|
||||
// - Legitimate ceiling. In this development instance's database — a mature
|
||||
// workspace set, 206 MB on disk — the ENTIRE corpus of wiki-linking
|
||||
// content is 2,949 items totalling 10,077,476 bytes (largest single body
|
||||
// 86,147 bytes). That is the absolute ceiling on any conceivable single
|
||||
// cascade there: it assumes every wiki-linking document links the one
|
||||
// title being renamed, which no real workspace does. 16 MiB is above
|
||||
// 86,147 bytes). A cascade over all of it would retain read + rewritten,
|
||||
// so ~20,154,952 bytes. That is the absolute ceiling on any conceivable
|
||||
// single cascade there: it assumes every wiki-linking document links the
|
||||
// one title being renamed, which no real workspace does. 32 MiB is ~1.6x
|
||||
// that impossible worst case, so the guard cannot fire on honest use.
|
||||
// (Measured on `items`, the live surface; the `documents` table in that
|
||||
// instance is empty, which is why the proxy — the two carry the same kind
|
||||
// of content through the same kind of cascade.)
|
||||
// (Measured on `items`, the live surface; that instance's `documents`
|
||||
// table is empty, which is why the proxy — the two carry the same kind of
|
||||
// content through the same kind of cascade.)
|
||||
// - Hostile floor. A single linking document holding the largest body a
|
||||
// 2 MiB request can carry projects 108,632,370 bytes once the title bound
|
||||
// is in place — 6.5x this cap — so the attack is refused at k = 1 and
|
||||
// 2 MiB request can carry retains 110,729,522 bytes once the title bound
|
||||
// is in place — 3.3x this cap — so the attack is refused at k = 1 and
|
||||
// every k above it, rather than at some threshold count of documents.
|
||||
// - Cost of the bound itself. The cascade holds read and rewritten bodies
|
||||
// concurrently, so the cap is a promise about resident memory: at most
|
||||
// ~2x this per in-flight rename, which is a bounded, budgetable number
|
||||
// for a server that previously had none.
|
||||
//
|
||||
// The gap between the two figures is deliberate and wide: a cap has to be far
|
||||
// enough above real use that nobody meets it by accident, and far enough
|
||||
// below the hazard that meeting it costs nothing. 16 MiB is ~1.6x the former
|
||||
// and ~0.15x the latter.
|
||||
const MaxRenameCascadeProjectedBytes = 16 << 20
|
||||
// The gap is deliberate and wide: a cap has to be far enough above real use
|
||||
// that nobody meets it by accident, and far enough below the hazard that
|
||||
// meeting it costs nothing.
|
||||
//
|
||||
// What it does NOT cover, stated so the next reader does not over-read it:
|
||||
// this bounds ONE rename's linked-document content, not concurrent renames (N
|
||||
// of them may each hold up to this), and not the base cost of a workspace
|
||||
// whose linking documents are legitimately large — a cascade under the cap
|
||||
// still allocates whatever it holds.
|
||||
const MaxRenameCascadeRetainedBytes = 32 << 20
|
||||
|
||||
// cascadeRewriteAttempts bounds rewriteLinkerCAS's retry loop.
|
||||
//
|
||||
@@ -789,7 +861,7 @@ var cascadeRewriteAttempts = 3
|
||||
// Both are pre-existing and neither is made worse here. They are recorded
|
||||
// because the next reader's question is "is the cascade correct now", and the
|
||||
// honest answer is "for the direction this bug named".
|
||||
func (s *Store) rewriteLinkerCAS(tx *sql.Tx, id, read, rewritten, oldTitle, newTitle string) error {
|
||||
func (s *Store) rewriteLinkerCAS(tx *sql.Tx, id, read, rewritten, oldTitle, newTitle, searchTerm string, budget int64) error {
|
||||
expected := read
|
||||
next := rewritten
|
||||
for attempt := 0; attempt < cascadeRewriteAttempts; attempt++ {
|
||||
@@ -821,6 +893,17 @@ func (s *Store) rewriteLinkerCAS(tx *sql.Tx, id, read, rewritten, oldTitle, newT
|
||||
return err
|
||||
}
|
||||
|
||||
// The re-read body is a NEW input, supplied by whoever won the race,
|
||||
// and it is bounded by nothing this cascade has already checked. Its
|
||||
// budget is the cap less what the other linkers are holding, so the
|
||||
// aggregate bound holds across retries too — without this, an editor
|
||||
// could grow a linker between the scan and the retry and walk the
|
||||
// rename straight back into the amplification it was refused for
|
||||
// (BUG-2798, codex round 1 P1).
|
||||
if grown := cascadeRetainedBytes(current, searchTerm, oldTitle, newTitle); grown > budget {
|
||||
return newRenameCascadeTooLargeError(newTitle, grown)
|
||||
}
|
||||
|
||||
// A concurrent edit won. Rewrite ITS body rather than ours: replaying
|
||||
// the original rewrite would reintroduce the very content this bug is
|
||||
// about losing. If that edit already removed the link, ReplaceTitle is
|
||||
|
||||
@@ -3,6 +3,7 @@ package store
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/PerpetualSoftware/pad/internal/models"
|
||||
@@ -11,30 +12,33 @@ import (
|
||||
// BUG-2798. A document rename rewrites `[[oldTitle]]` → `[[newTitle]]` in
|
||||
// every linking document, and the cascade holds every rewritten body in memory
|
||||
// before it writes any of them. Neither the title length nor the number of
|
||||
// linking documents was bounded, so one rename could project more output than
|
||||
// the process could hold — measured at 20,000x on the filing, and still 51.8x
|
||||
// per document after the title bound alone.
|
||||
// linking documents was bounded, so one rename could hold more content than the
|
||||
// process could carry — measured at 20,000x amplification on the filing, and
|
||||
// still 51.8x per document after the title bound alone.
|
||||
//
|
||||
// The title bound (models.MaxDocumentTitleRunes) is the cheap door. This file
|
||||
// covers the wall: the cascade refuses when its projected TOTAL exceeds
|
||||
// MaxRenameCascadeProjectedBytes.
|
||||
// covers the wall: the cascade refuses when the TOTAL it would retain across
|
||||
// the linking set — every read body plus every written body — exceeds
|
||||
// MaxRenameCascadeRetainedBytes.
|
||||
|
||||
// linkerBody returns a body containing exactly n `[[A]]` occurrences, and the
|
||||
// number of bytes renaming "A" to a title of length newLen would project for
|
||||
// it: len(content) + occurrences * (len(new) - len(old)).
|
||||
// number of bytes the cascade would RETAIN for it when renaming "A" to a title
|
||||
// of length newLen: the body it reads plus the body it writes.
|
||||
//
|
||||
// Exact, not an estimate — strings.Replace substitutes every non-overlapping
|
||||
// occurrence, so this is the output size to the byte.
|
||||
// Deliberately computed here rather than by calling cascadeRetainedBytes — a
|
||||
// test that reuses the implementation's arithmetic cannot catch that
|
||||
// arithmetic being wrong.
|
||||
func linkerBody(n, newLen int) (string, int) {
|
||||
body := strings.Repeat("[[A]]", n)
|
||||
return body, len(body) + n*(newLen-1)
|
||||
rewritten := len(body) + n*(newLen-1)
|
||||
return body, len(body) + rewritten
|
||||
}
|
||||
|
||||
// TestRenameCascade_RefusesOnProjectedTOTAL_NotPerDocument is the load-bearing
|
||||
// test, and its shape is the finding it encodes: a per-document cap would not
|
||||
// close this bug.
|
||||
//
|
||||
// Every linking document here projects comfortably UNDER the cap on its own.
|
||||
// Every linking document here retains comfortably UNDER the cap on its own.
|
||||
// Only the total is over. A guard that tested each document in isolation would
|
||||
// admit all three, allocate the sum, and pass a test that merely asserted "a
|
||||
// huge single document is refused" — which is why this test asserts the
|
||||
@@ -47,20 +51,20 @@ func TestRenameCascade_RefusesOnProjectedTOTAL_NotPerDocument(t *testing.T) {
|
||||
// Size each linker so that linkers-1 of them fit under the cap and all of
|
||||
// them do not. Derived from the cap rather than hardcoded, so the test
|
||||
// keeps discriminating if the cap moves.
|
||||
perDocTarget := (MaxRenameCascadeProjectedBytes / linkers) + (MaxRenameCascadeProjectedBytes / (linkers * 4))
|
||||
occurrences := perDocTarget / (5 + (newTitleLen - 1))
|
||||
body, perDocProjected := linkerBody(occurrences, newTitleLen)
|
||||
perDocTarget := (MaxRenameCascadeRetainedBytes / linkers) + (MaxRenameCascadeRetainedBytes / (linkers * 4))
|
||||
occurrences := perDocTarget / (5 + 5 + (newTitleLen - 1))
|
||||
body, perDocRetained := linkerBody(occurrences, newTitleLen)
|
||||
|
||||
// Preconditions — these are what make the test discriminate. If either
|
||||
// fails the test is no longer testing what its name says.
|
||||
if perDocProjected >= MaxRenameCascadeProjectedBytes {
|
||||
t.Fatalf("precondition: per-document projection %d must be UNDER the cap %d, "+
|
||||
if perDocRetained >= MaxRenameCascadeRetainedBytes {
|
||||
t.Fatalf("precondition: per-document retention %d must be UNDER the cap %d, "+
|
||||
"otherwise a per-document guard would also pass this test",
|
||||
perDocProjected, MaxRenameCascadeProjectedBytes)
|
||||
perDocRetained, MaxRenameCascadeRetainedBytes)
|
||||
}
|
||||
if total := perDocProjected * linkers; total <= MaxRenameCascadeProjectedBytes {
|
||||
t.Fatalf("precondition: total projection %d must EXCEED the cap %d",
|
||||
total, MaxRenameCascadeProjectedBytes)
|
||||
if total := perDocRetained * linkers; total <= MaxRenameCascadeRetainedBytes {
|
||||
t.Fatalf("precondition: total retention %d must EXCEED the cap %d",
|
||||
total, MaxRenameCascadeRetainedBytes)
|
||||
}
|
||||
|
||||
s := testStore(t)
|
||||
@@ -83,9 +87,9 @@ func TestRenameCascade_RefusesOnProjectedTOTAL_NotPerDocument(t *testing.T) {
|
||||
}
|
||||
|
||||
// 2. Refused with the projection in the message. The only actionable
|
||||
// information for a caller is what was projected against what is
|
||||
// allowed; an error that says "too large" and nothing else sends them
|
||||
// guessing.
|
||||
// information for a caller is what the rename would hold against what
|
||||
// is allowed; an error that says "too large" and nothing else sends
|
||||
// them guessing.
|
||||
if msg := err.Error(); !strings.Contains(msg, "maximum") || !strings.Contains(msg, "bytes") {
|
||||
t.Errorf("error message lacks the projection: %q", msg)
|
||||
}
|
||||
@@ -142,19 +146,19 @@ func TestRenameCascade_AllowsAnOrdinaryRename(t *testing.T) {
|
||||
|
||||
// TestRenameCascade_RefusesTheSingleDocumentAttack covers the k=1 case
|
||||
// directly: one linking document holding the largest body a 2 MiB request can
|
||||
// carry still projects 108,632,370 bytes once the title bound is in place
|
||||
// (measured), which is 6.5x the cap. The attack is refused at every k, not
|
||||
// only at a threshold count of documents.
|
||||
// carry still retains 110,729,522 bytes once the title bound is in place
|
||||
// (108,632,370 written plus the 2,097,152 read), which is 3.3x the cap. The
|
||||
// attack is refused at every k, not only at a threshold count of documents.
|
||||
//
|
||||
// Kept separate from the total-versus-per-document test because it is the one
|
||||
// case a per-document guard WOULD catch — asserting both makes it explicit
|
||||
// that the total guard is a superset, not a replacement of unclear scope.
|
||||
func TestRenameCascade_RefusesTheSingleDocumentAttack(t *testing.T) {
|
||||
const newTitleLen = 255
|
||||
occurrences := (MaxRenameCascadeProjectedBytes / (5 + (newTitleLen - 1))) * 2 // 2x the cap
|
||||
body, projected := linkerBody(occurrences, newTitleLen)
|
||||
if projected <= MaxRenameCascadeProjectedBytes {
|
||||
t.Fatalf("precondition: single-document projection %d must exceed the cap %d", projected, MaxRenameCascadeProjectedBytes)
|
||||
occurrences := (MaxRenameCascadeRetainedBytes / (5 + 5 + (newTitleLen - 1))) * 2 // 2x the cap
|
||||
body, retained := linkerBody(occurrences, newTitleLen)
|
||||
if retained <= MaxRenameCascadeRetainedBytes {
|
||||
t.Fatalf("precondition: single-document retention %d must exceed the cap %d", retained, MaxRenameCascadeRetainedBytes)
|
||||
}
|
||||
|
||||
s := testStore(t)
|
||||
@@ -168,3 +172,214 @@ func TestRenameCascade_RefusesTheSingleDocumentAttack(t *testing.T) {
|
||||
t.Fatalf("rename: got %v, want ErrRenameCascadeTooLarge", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRenameCascade_CountsRetainedBytesNotJustOutput pins codex round 1's P1
|
||||
// against the guard that shipped in the first commit, which summed only the
|
||||
// PROJECTED OUTPUT.
|
||||
//
|
||||
// The counterexample is a rename to a SHORTER title. Every linker here holds a
|
||||
// large body, but the rewrite shrinks it, so an output-only counter reports a
|
||||
// small number while the cascade still retains every read body for its
|
||||
// compare-and-set. Under the old guard the total below reported far under the
|
||||
// cap and the rename proceeded; the retained-bytes guard refuses it.
|
||||
//
|
||||
// This is why the direction of the rename matters and why the constant is
|
||||
// named for retention rather than for projection.
|
||||
func TestRenameCascade_CountsRetainedBytesNotJustOutput(t *testing.T) {
|
||||
// Old title long, new title short — the shrinking direction.
|
||||
oldTitle := strings.Repeat("O", 200)
|
||||
newTitle := "n"
|
||||
|
||||
// Each linker is ~2 MiB of `[[<200-char title>]]`, the shape a single 2 MiB
|
||||
// request can deliver.
|
||||
occurrencesPerDoc := (2 << 20) / (len(oldTitle) + 4)
|
||||
body := strings.Repeat("[["+oldTitle+"]]", occurrencesPerDoc)
|
||||
|
||||
// Enough linkers that the RETAINED total is over the cap while the
|
||||
// projected OUTPUT total stays under it. That gap is the finding.
|
||||
linkers := (MaxRenameCascadeRetainedBytes / len(body)) + 2
|
||||
|
||||
outputTotal := 0
|
||||
retainedTotal := 0
|
||||
for i := 0; i < linkers; i++ {
|
||||
rewritten := len(body) + occurrencesPerDoc*(len(newTitle)-len(oldTitle))
|
||||
outputTotal += rewritten
|
||||
retainedTotal += len(body) + rewritten
|
||||
}
|
||||
if outputTotal > MaxRenameCascadeRetainedBytes {
|
||||
t.Fatalf("precondition: projected OUTPUT total %d must stay UNDER the cap %d, "+
|
||||
"otherwise an output-only guard would also refuse this and the test proves nothing",
|
||||
outputTotal, MaxRenameCascadeRetainedBytes)
|
||||
}
|
||||
if retainedTotal <= MaxRenameCascadeRetainedBytes {
|
||||
t.Fatalf("precondition: RETAINED total %d must exceed the cap %d", retainedTotal, MaxRenameCascadeRetainedBytes)
|
||||
}
|
||||
|
||||
s := testStore(t)
|
||||
ws := createTestWorkspace(t, s, "CascadeShrinking")
|
||||
target := createTestDoc(t, s, ws.ID, oldTitle, "the document being renamed")
|
||||
for i := 0; i < linkers; i++ {
|
||||
createTestDoc(t, s, ws.ID, "Linker"+string(rune('a'+i)), body)
|
||||
}
|
||||
|
||||
if _, err := s.UpdateDocument(target.ID, models.DocumentUpdate{Title: &newTitle}); !errors.Is(err, ErrRenameCascadeTooLarge) {
|
||||
t.Fatalf("shrinking rename: got %v, want ErrRenameCascadeTooLarge — an output-only guard admits this", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRenameCascade_FindsLinkersWhoseTitleContainsABackslash pins codex round
|
||||
// 1's P2. POSTGRES ONLY, and skipped loudly elsewhere rather than passing
|
||||
// quietly — the defect is a DIALECT DIVERGENCE, and SQLite is the dialect that
|
||||
// was accidentally right.
|
||||
//
|
||||
// The cascade finds linkers with `content LIKE ?`. Postgres LIKE treats
|
||||
// backslash as the default escape character; SQLite LIKE has no default escape
|
||||
// character at all. So an unescaped search term for a title containing `\` was
|
||||
// searched for as the literal it is on SQLite, and as a DIFFERENT literal on
|
||||
// Postgres — `[[Alpha\Beta]]` became `[[AlphaBeta]]`, the linking documents
|
||||
// were not found, the cascade rewrote nothing, and the rename reported success
|
||||
// leaving every link pointing at a title that no longer exists.
|
||||
//
|
||||
// A green run on SQLite is therefore a property of the DSN, not evidence about
|
||||
// this fix, which is why this skips instead.
|
||||
func TestRenameCascade_FindsLinkersWhoseTitleContainsABackslash(t *testing.T) {
|
||||
s := testStore(t)
|
||||
if s.dialect.Driver() != DriverPostgres {
|
||||
t.Skip("asserts a Postgres LIKE-escape property; SQLite LIKE has no default escape character, so the unescaped pattern is accidentally correct there")
|
||||
}
|
||||
|
||||
ws := createTestWorkspace(t, s, "CascadeLikeBackslash")
|
||||
title := `Alpha\Beta`
|
||||
target := createTestDoc(t, s, ws.ID, title, "the document being renamed")
|
||||
linker := createTestDoc(t, s, ws.ID, "RealLinker", "see [["+title+"]] here")
|
||||
|
||||
newTitle := "Renamed"
|
||||
if _, err := s.UpdateDocument(target.ID, models.DocumentUpdate{Title: &newTitle}); err != nil {
|
||||
t.Fatalf("rename: %v", err)
|
||||
}
|
||||
|
||||
got, err := s.GetDocument(linker.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("read back linker: %v", err)
|
||||
}
|
||||
if want := "see [[Renamed]] here"; got.Content != want {
|
||||
t.Errorf("the linker was not rewritten — the cascade's LIKE pattern did not find it:\n got: %q\nwant: %q",
|
||||
got.Content, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRenameCascade_DoesNotSpendTheBudgetOnDocumentsThatDoNotLinkTheTitle is
|
||||
// the rest of the class codex's backslash finding was an instance of
|
||||
// (CONVE-18): `%` and `_` are LIKE wildcards in BOTH dialects, so an unescaped
|
||||
// search term for a title containing them selects documents that do not link
|
||||
// it.
|
||||
//
|
||||
// Over-matching cannot be caught by asserting content — the extra rows rewrite
|
||||
// to themselves, because ReplaceTitle looks for the literal. It is observable
|
||||
// through the guard, which is computed from this result set: unrelated
|
||||
// documents inflate the retained total, and a rename that fits the cap is
|
||||
// refused because of content it was never going to touch. That is the harm,
|
||||
// and it is what this asserts.
|
||||
//
|
||||
// The first version of this test asserted the decoy's content was untouched
|
||||
// and passed against the unescaped pattern — a vacuous green. Recorded here
|
||||
// because the fix was to find the observable consequence, not to trust the
|
||||
// mechanism.
|
||||
func TestRenameCascade_DoesNotSpendTheBudgetOnDocumentsThatDoNotLinkTheTitle(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
title string
|
||||
decoy string // matches the title read as a PATTERN, not as a literal
|
||||
}{
|
||||
{"percent", `Alpha%Beta`, `[[AlphaXYZBeta]]`},
|
||||
{"underscore", `Alpha_Beta`, `[[AlphaZBeta]]`},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s := testStore(t)
|
||||
ws := createTestWorkspace(t, s, "CascadeLike"+tc.name)
|
||||
target := createTestDoc(t, s, ws.ID, tc.title, "the document being renamed")
|
||||
linker := createTestDoc(t, s, ws.ID, "RealLinker", "see [["+tc.title+"]] here")
|
||||
|
||||
// Decoys big enough that INCLUDING them blows the cap, while the
|
||||
// real linker alone is negligible. With the pattern escaped they
|
||||
// are not selected and the rename is comfortably under budget.
|
||||
decoyBody := strings.Repeat("x", 1<<20) + " " + tc.decoy
|
||||
decoys := (MaxRenameCascadeRetainedBytes / len(decoyBody)) + 2
|
||||
for i := 0; i < decoys; i++ {
|
||||
createTestDoc(t, s, ws.ID, "Decoy"+string(rune('a'+i)), decoyBody)
|
||||
}
|
||||
|
||||
newTitle := "Renamed"
|
||||
if _, err := s.UpdateDocument(target.ID, models.DocumentUpdate{Title: &newTitle}); err != nil {
|
||||
t.Fatalf("a rename well under the cap was refused because of documents that do not link it: %v", err)
|
||||
}
|
||||
|
||||
got, err := s.GetDocument(linker.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("read back linker: %v", err)
|
||||
}
|
||||
if want := "see [[Renamed]] here"; got.Content != want {
|
||||
t.Errorf("the real linker was not rewritten:\n got: %q\nwant: %q", got.Content, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRenameCascade_RetryRecheckesTheBudgetAgainstTheGrownBody pins codex round
|
||||
// 1's other P1: the guard used to cover only the cascade's SCAN, while the
|
||||
// compare-and-set's retry path re-read a linker's body and called ReplaceTitle
|
||||
// on it with no bound at all.
|
||||
//
|
||||
// The re-read body is a NEW input supplied by whoever won the race, so a
|
||||
// content edit landing inside the cascade's window could grow a linker from
|
||||
// harmless to enormous and walk the rename straight back into the
|
||||
// amplification it would have been refused for.
|
||||
//
|
||||
// POSTGRES ONLY, for the same structural reason as
|
||||
// TestUpdateDocument_CascadeDoesNotOverwriteConcurrentEdit: SQLite's
|
||||
// `_txlock=immediate` DSN takes the write lock at BEGIN and holds it across the
|
||||
// cascade's whole read→write window, so a concurrent edit cannot commit inside
|
||||
// it. A green run there would be a property of the DSN, not evidence about this
|
||||
// guard.
|
||||
func TestRenameCascade_RetryRecheckesTheBudgetAgainstTheGrownBody(t *testing.T) {
|
||||
s := testStore(t)
|
||||
if s.dialect.Driver() != DriverPostgres {
|
||||
t.Skip("needs a concurrent edit to commit inside the cascade's read→write window; SQLite's BEGIN IMMEDIATE closes it structurally")
|
||||
}
|
||||
|
||||
ws := createTestWorkspace(t, s, "CascadeRetryBudget")
|
||||
target := createTestDoc(t, s, ws.ID, "A", "the document being renamed")
|
||||
|
||||
// Small at scan time — the cascade counts a few bytes and proceeds.
|
||||
linker := createTestDoc(t, s, ws.ID, "Linker", "before [[A]] after")
|
||||
|
||||
// The winner's body is over the cap on its own, so the retry's re-read is
|
||||
// the first and only place this can be caught.
|
||||
newTitleLen := 255
|
||||
occurrences := (MaxRenameCascadeRetainedBytes / (5 + 5 + (newTitleLen - 1))) * 2
|
||||
grownBody, grownRetained := linkerBody(occurrences, newTitleLen)
|
||||
if grownRetained <= MaxRenameCascadeRetainedBytes {
|
||||
t.Fatalf("precondition: the grown body's retention %d must exceed the cap %d", grownRetained, MaxRenameCascadeRetainedBytes)
|
||||
}
|
||||
|
||||
var once sync.Once
|
||||
var editErr error
|
||||
s.afterLinkCascadeRead = func(string) {
|
||||
once.Do(func() {
|
||||
// Content-only, so it takes no rename lock and can commit inside
|
||||
// the cascade's window (BUG-2785's seam, same mechanism).
|
||||
_, editErr = s.UpdateDocument(linker.ID, models.DocumentUpdate{Content: &grownBody})
|
||||
})
|
||||
}
|
||||
defer func() { s.afterLinkCascadeRead = nil }()
|
||||
|
||||
newTitle := strings.Repeat("T", newTitleLen)
|
||||
_, err := s.UpdateDocument(target.ID, models.DocumentUpdate{Title: &newTitle})
|
||||
|
||||
if editErr != nil {
|
||||
t.Fatalf("the concurrent edit itself failed, so this run never exercised the retry path: %v", editErr)
|
||||
}
|
||||
if !errors.Is(err, ErrRenameCascadeTooLarge) {
|
||||
t.Fatalf("rename: got %v, want ErrRenameCascadeTooLarge — the retry re-read an unbounded body", err)
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user