7 Commits

Author	SHA1	Message	Date
xarmian	b67e78343a	test(materialize): fire the idle timer mid-job on purpose, not by wall-clock luck (BUG-3435) (#1827) ... TestSupervisorIdleTimerDuringJob staged 'the idle timer fires while a job runs' with a 300ms timer, a 200ms pause and a 400ms job. On a loaded macOS runner the pause stretched past the timer, the worker went idle BETWEEN the jobs, and the test failed with '2 spawns'. Widening the pause to 320ms reproduces that exact failure 3/3. Now nothing fires on its own (a one-hour idle). A new script command, await:<path>, holds the job inside the worker until the test releases it; the test fires the timer itself (back-dating lastJob, Reset(0)) once the job has provably started, and waits on a new test hook, onIdleBusy, which idleStop calls when it finds the slot taken. Then it releases the job, checks one spawn and that the job's end re-armed the timer on the same worker, and fires once more to see the worker stop. Mutants: idleStop proceeding without the slot check (red 10/10), and armIdle not recording the job's end (red). 30/30 green with every CPU saturated, and under -race. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS	2026-10-06 09:25:18 -04:00
xarmian	f0c65fe08c	chore(web): adopt Tiptap 3.31.4 and regenerate the materializer corpus (TASK-3421) (#1811) ... * chore(web): adopt Tiptap 3.31.4 and regenerate the materializer corpus (TASK-3421) 3.31.4's TableCell/TableHeader renderHTML stop rendering a default colspan="1" / rowspan="1". parseHTML is unchanged, so a stored body carrying them parses to the same document. Replaying all 50 old corpus op-logs through the 3.31.4 bundle reproduces the old output exactly once those attributes are removed (and fails on 28 cases without removing them). Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * test(web): pin that a pre-3.31.4 table body still dedupes on open (TASK-3421) Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS	2026-10-05 15:52:54 -04:00
xarmian	cc2194552b	test(materialize): two CI reds were test instruments, not the product (TASK-2198) (#1694) ... main was red at cb99879d, and #1693 was red. Both are test defects, measured. 1. TestMaterializeRecoveryEndToEnd, "nothing_pending, want applied" (main). dequeue clears an item's queued mark before its pass runs, so the test's 20ms sweep re-enqueues an item whose slow first pass is still running. The second pass correctly finds nothing pending, and the fixture kept only the LAST result per item. Reproduced without PostgreSQL: a 150ms materializer records [applied nothing_pending]. testServer is a private SQLite per test even in the PG job, so there is no collision. The fixture now keeps every result, and waitApplied waits for an apply. A new leg pins that the second pass writes nothing, and fails with the old reading's exact CI string. Production is unchanged: the second pass is one read. 2. TestSupervisorRealWorkerParity, "not ready within 1m0s" under -race (#1693). The race-built test binary as a real worker took 23.4s idle and 49.6s at 2x CPU oversubscription to its first job (measured), against the harness's default 60s start timeout. Real-worker harnesses under the race detector now get 5m. The shipped binary is not race-built. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC	2026-09-30 13:13:13 -04:00
xarmian	cb99879d53	feat(materialize): recover unsaved editor sessions into items.content (TASK-2198 U4) (#1683) ... * feat(materialize): recover unsaved editor sessions into items.content in the background (TASK-2198 U4) Wires the U3 supervisor in. A collab room reclaimed at the end of its grace TTL with content-bearing rows above the flush watermark (T3), or a dormant pending item found by a one-minute sweep (T1), is queued; one worker goroutine materializes the item's whole content-bearing op-log with the whole-workspace link index and lands it through store.MaterializeFlush, under the collab item lock with no room open. The write is a flush: cursor == MAX(op-log id), rows pending, no set-aside rows, checked in the write's own transaction; watermark to the cursor only; seq bumped; one version row attributed to system/recovery; the op-log is never pruned. PAD_MATERIALIZE=off disables it entirely. A per-item in-memory failure budget (3 failures, doubling backoff, reset by a new op-log row) keeps untrusted input from being retried in a loop. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(store): recovery watermark is the cursor, not MAX at UPDATE time (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(store): a second recovery still writes its own version row (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * docs(mcp): op-log recovery bounds the stale-body window; tool surface v0.60 (TASK-2198 U4) content_state applied_pending_flush now clears on its own after the item's room closes, unless recovery fails or PAD_MATERIALIZE=off. Every sentence that called the window unbounded, or said nothing server-side moves op-log content into items.content, is corrected (CLAUDE.md, instructions.md, README, docs/deployment.md, docs/backup.md, code comments). BEHAVIOR bump to 0.60; goldens regenerated. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): count ErrNoMemoryCap silently against the per-item budget (TASK-2198 U4) U3 now refuses every job with ErrNoMemoryCap when no memory cap can be established, and warns once itself. The recovery worker counts it like any failure but logs it at debug, not per item. Its log lines go through a logger seam so the test can tell. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * style: gofmt (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * feat(materialize): one log line per budget exhaustion; count set-aside skips on the sweep line (TASK-2198 U4) An item that uses up its failure budget logs exactly one WARN naming the item, the failure count and the last error kind (DEBUG when the kind is no_memory_cap, which the supervisor already warned about once), and logs again only after op-log growth resets the budget and it is exhausted anew. The schema-version skip goes through the same line. Items skipped for set-aside rows are counted and reported on a new single sweep summary line, written only when it has news; never per item. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): keyset-page the recovery sweep so no candidate starves (TASK-2198 U4, codex r1) The sweep read the oldest 200 candidates from the start every tick, and the room check and the failure budget filtered them afterwards. Once 200 sat in backoff or exhausted, every sweep returned the same 200 and the 201st was never queued: filtering above a bound is starvation. Each tick now reads one page strictly after an in-memory keyset cursor (newest op-log created_at, item id) and wraps to the start after a short page, so every candidate is offered within ceil(N/limit)+1 ticks, whatever sorts ahead of it. Keyset over NOT IN: no parameter-count limit, and it also covers the room check, which an exclusion list would not. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(materialize): the sweep cursor wraps; docs for the exhaustion line, the set-aside count and paging (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * docs(deployment): reflow the recovery budget paragraph (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(materialize): recovery strips the follows-title marker as a tab's save does (TASK-2198 U4, BUG-3315) Since #1688 a tab loads a link that follows its target's title with the follows-title marker in the link's title attribute, so the marker lives in Y.Docs that recovery replays. Three fixture cases (a follows-title link; the same link after a rename; a URL edited to an external address) are generated from documents that really hold the marker, and the existing 8 are unchanged. A new leg recovers each through the production path (whole-workspace index) and through the runner with a restricted and an empty index, and asserts no 'pad-follows-title' reaches the body. It goes red in all 7 ways when the bundle's marker strips are disabled. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(store): a recovery version row carries #1691's history data (TASK-2198 U4) Rebased onto PLAN-2348 U2 (#1691), which adds user_id, line counts and a create marker to version rows. Recovery writes its row through the same UpdateItem insert, so it gets them automatically: no user_id (a system row), not a create, and line counts equal to diff.LineCounts of the stale and recovered bodies. The leg pins that. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC	2026-09-30 11:03:07 -04:00
xarmian	d78ff9bdc5	feat(materialize): worker supervisor with per-OS memory cap (TASK-2198 U3) (#1682) ... * feat(materialize): parent-side worker supervisor with per-OS memory cap (TASK-2198 U3) The Supervisor runs materializer jobs in ONE long-lived child process (`pad __materialize-worker`), started lazily on the first job and fed jobs serially. A job past PAD_MATERIALIZE_TIMEOUT (default 2s) kills the child (goja's Interrupt, sent as timeout_ms a tenth earlier, is only the soft layer); a dead child is respawned on the next job after an exponential backoff (1s doubling, capped at 5 min, reset by a child that lived a minute). Every stop is logged with its reason (deadline, memory, exit, protocol, start, canceled, closed); every start logs the effective timeout, memory limit and cap mechanism. The child's stderr goes to the log, pid-tagged and bounded, never near the protocol. A job whose child died is never retried here. Memory cap (PAD_MATERIALIZE_MEM_LIMIT, default 2GiB, bytes or KiB/MiB/GiB, clamped to 1536MiB-16GiB): - Linux: RLIMIT_AS set on the child by prlimit before its first byte of input; the child only runs its own startup uncapped. - Windows: Job Object with JOB_OBJECT_LIMIT_PROCESS_MEMORY; the child is created suspended, assigned, then resumed: no uncapped window. - macOS: parent-side RSS watchdog every 100ms (proc_info syscall, ps fallback). - Other: deadline kill only. Not wired into the server yet (U4). Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * ci(materialize): run the supervisor on the real macOS and Windows caps (TASK-2198 U3) native-smoke now builds the real materializer bundle (as the go job does) and runs the supervisor, memory-cap and deadline tests on macos-latest (proc_info RSS watchdog) and windows-latest (suspended start + Job Object). PAD_MATERIALIZE_TEST_NO_SKIP turns the cap tests' skips into failures there, so a cap that stops running cannot pass as skipped. Tests: a real Windows processGone (OpenProcess + GetExitCodeProcess), RSS samplers for darwin (the production proc_info path) and Windows (K32GetProcessMemoryInfo), and an allocation helper that writes noise into every byte so macOS memory compression cannot hide it from the watchdog. The worker-loop tests share one loaded Runner through serveWorker (RunWorker's loop, split out); TestWorkerRoundTrip still goes through RunWorker end to end. go test -race -count=3 on the package: 714s -> 413s. docs: the materializer section says the worker and its knobs are inert until U4 wires it in. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): a worker never outlives its supervising process (TASK-2198 U3, codex r1) If the server died mid-job (SIGKILL, OOM kill, crash) its worker ran on with no deadline kill and, on macOS, no memory watchdog. Two layers: - Linux: Pdeathsig = SIGKILL. The fork runs on a fresh goroutine (startUnlocked), never locked to its thread, so it cannot land on a thread the runtime later retires (golang/go#27505); Go's forkExec re-checks getppid if the parent is already gone. - Every Unix (the only layer on macOS): the worker command calls ExitWhenOrphaned before the bundle loads, polling its parent pid every 200ms against PAD_MATERIALIZE_PARENT_PID and exiting (status 70) when it changes. It writes nothing. - Windows was already covered by the Job Object's KILL_ON_JOB_CLOSE. TestSupervisorOrphanedWorkerExits kills a helper process that supervises a worker hung in a job and requires the worker gone within 1s: 0s on Linux, 105ms with the ppid watch alone; with both layers removed the worker outlives its parent by 5s. It matches native-smoke's TestSupervisor.* selection, so it runs on macOS and Windows too. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): relative memory cap, refuse when uncappable, GOMAXPROCS=2; nix-safe ps test (TASK-2198 U3) The absolute Linux RLIMIT_AS failed on CI: a 64 MiB allocation died under a 1600 MiB cap, because the runtime there reserved more address space than on the machine the floor was measured on. The cap is now RELATIVE on every OS: once the worker has loaded and before it is sent any job, its usage is read as a baseline and the cap is set to baseline + PAD_MATERIALIZE_MEM_LIMIT — address space on Linux (/proc/<pid>/stat vsize, prlimit, read back), committed memory on Windows (the Job Object's per-process limit, set after the suspended start and load), resident memory on macOS (the watchdog's threshold). Every start logs baseline, limit and effective cap. Lead ruling, REFUSE: if the baseline probe fails or is implausible (0, or above MaxPlausibleBaseline = 8 GiB, ~4.8x the largest measured 1704 MiB), onBaselineProbeFailure kills the worker without sending it the job and fails the job with ErrNoMemoryCap; one WARN until a probe succeeds again. An OS with no cap mechanism never starts a worker and refuses every job with ErrNoMemoryCap, one WARN naming runtime.GOOS. The worker runs with GOMAXPROCS=2: measured loaded-worker baseline 1357 MiB vs 1597 MiB at this host's default (8) and 1704 MiB at 64. MinMemLimit drops from 1536 MiB (an artefact of the absolute cap) to 256 MiB: the corpus grows the worker by at most +10 MiB of address space and +18 MiB resident, and a 4,000-deep nested document by +129..194 MiB. A cgo worker's "pthread_create failed" at the cap is now classified as a memory death. Nix: TestParsePSRSS parses fixed strings only; the live `ps` leg is TestPSRSSLive and skips when there is no `ps` (the Nix sandbox has none; ps is only macOS's fallback), unless PAD_MATERIALIZE_TEST_NO_SKIP is set. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): Windows kills on the job's memory notification; steady thread set before the baseline (TASK-2198 U3) Windows (CI on c1866957): the Job Object refused the 1 GiB commit (VirtualAlloc errno=1455), but the worker did not die and the job ran to its deadline. The cap no longer depends on the runtime choosing to crash: the job is associated with an I/O completion port, and JOB_OBJECT_MSG_PROCESS_MEMORY_LIMIT for the worker's pid kills it at once (ErrMemoryLimit). errno=1455 on stderr is a second signal. On every OS a cap mechanism that sees the limit hit calls onLimit, and the kill is reported as "reached its <mechanism> cap". Linux (CI's cgo test binary at GOMAXPROCS=64): threads created after the baseline ate the growth budget. The worker now brings its thread set to steady state before readiness (warmThreads: an allocating goroutine per P, then a GC). Measured on a cgo build: threads added after the baseline are 0 at GOMAXPROCS=2 with or without it; at 64, +3 threads and +272 MiB of address space during the corpus without it, +0 and +0 MiB with it. 256 MiB stays the floor (largest growth at GOMAXPROCS=2: +193 MiB, a 4,000-deep nested document; the corpus adds +0 MiB). The cap test's second leg no longer uses GOMAXPROCS=64 (which also changed thread creation) to stand in for a bigger host: the helper inflates its own baseline before readiness, with never-touched address space on Linux, committed memory on Windows and resident memory on macOS. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * feat(materialize): the worker stops after an idle timeout; the next job restarts it (TASK-2198 U3) Dave's ruling on the worker process: GO, with an idle exit. A loaded worker holds ~200 MB resident, which an install that rarely materializes should not pay for all day. PAD_MATERIALIZE_IDLE_TIMEOUT: default 5m, clamped to 30s-24h, 0 = never. After each job the supervisor (re)arms a timer. When it fires, the stop runs only while holding the job slot, so it never takes a worker with a job in flight. It re-checks the idle time, forgets the worker WITHOUT recording a death (backoff is untouched), and closes the worker's stdin so its read loop ends with status 0, with a kill after a grace if it does not go. The stop is logged at INFO, reason=idle. A job that arrives meanwhile waits for the slot, finds no worker, and starts a new one with no backoff wait. Legs: the idle exit after the timeout (INFO, status 0, no WARN); the next job respawns and succeeds with no wait; an idle stop leaves the backoff streak alone (crash, idle, crash waits 2s, not 4s); a job arriving during the stop runs on a new worker; a timer that fires mid-job does not stop that worker; NeverIdle arms no timer; env parsing and clamps. Mutations, each red on its own: recording a death (3 legs), logging as a kill (1), not holding the slot (1), never arming (3). Docs: the knob beside the other two, and `idle` in the stop reasons. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): Close does not return while an idle-stopping worker still runs (TASK-2198 U3, codex r1) An idle stop forgets the worker before it has exited, so a Close in that window found no worker and returned with it still alive. The supervisor now tracks the worker an idle stop is ending, and Close kills and reaps it. A leg pins it, and goes red without the fix. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): an idle stop that Close ended says so (TASK-2198 U3, codex r2) The idle stop stored its kill reason unconditionally, so it overwrote the 'closed' of a Close that had already killed the worker, and its stop line said idle. It now claims the reason only if none is set, and the line reports reason=closed. The Close-during-idle-stop leg asserts it. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC	2026-09-30 09:52:49 -04:00
xarmian	1e16d2228b	fix(web): link text with markdown/HTML-significant characters survives a save (BUG-3315 U1) (#1686) ... A wiki link's display text was mangled on its way through the editor, and the first save of any item linking such a title pinned the mangled text as a display override, with no rename involved (a census found 76 stored overrides of this shape among 107 that differ from their target's title). Load: escapeMarkdownLinkText escaped only brackets and backslashes, so the html-enabled parser read the title as markdown. `*real*` became emphasis, `<topic>` was dropped as a tag, `&gt;` was decoded. It now backslash-escapes \ [ ] * _ ~ ` < > &, all ASCII punctuation CommonMark lets a backslash escape. Save: unescapeMarkdownLinkText undid only bracket and backslash escapes, while the serializer also backslash-escapes other punctuation and writes < > & as entities. It now undoes both, in one pass. One residue is intrinsic and pinned by a test: the serializer writes a literal `&gt;` and a literal `>` as the same bytes, so a title containing entity-shaped text still becomes an override (0 of 9,620 live titles do). The materializer corpus: two expectations change (a backtick and `<b>` in link text, both this fix), taken from the goja bundle's output. A full regeneration was not used because the generator is nondeterministic: with unchanged code it rewrites all 50 cases' rows and 1 expected value. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC	2026-09-30 07:19:04 -04:00
xarmian	fa2755e183	feat(materialize): headless op-log materializer bundle, goja runner and worker (TASK-2198 U2) (#1678) ... * feat(materialize): headless op-log materializer bundle, goja runner and worker (TASK-2198 U2) Builds the editor's own JS (Yjs -> y-tiptap -> Tiptap schema -> tiptap-markdown -> the tab's flush pipeline) into one IIFE, embeds it, and runs it in goja. Not wired into the server yet (U3 supervisor, U4 write path). - web/src/lib/collab/materializer/: bundle entry (extension list mirrors Editor.svelte's collab branch via the U1 shared modules), op-log replay mirroring wsProvider's MESSAGE_SYNC arm, linkedom DOM shim and polyfills from the spike, and a vendored patch of prosemirror-markdown MarkdownSerializerState.atBlank (quadratic under goja; equivalent). - web/scripts/build-materializer.mjs (esbuild) -> web/build-materializer/, run by `npm run build`; embedded as pad.MaterializerJS. - internal/materialize: Runner (Go-native atob/btoa/TextEncoder/TextDecoder/ getRandomValues, schema-version check, ctx interrupt with a clean VM after), and RunWorker's length-prefixed JSON protocol behind the hidden `pad __materialize-worker` command. - Parity: vitest drives two real Editor.svelte mounts and compares materialize() with the tab's flush on 50 seeded cases; the same cases are committed as testdata/corpus.json and replayed in goja (TestCorpusParity). - CI Go jobs, release tests, Docker and Nix produce the bundle; linkedom and esbuild are new devDependencies. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(materialize): pin the atBlank patch's equivalence under V8 too (TASK-2198 U2, codex r1) Codex r1 claimed upstream's /(^|\n)$/ is true for a trailing CR in JS. It is not (checked in V8: false for "\r" and "a\u2028", as the patch returns), but the only equivalence test ran under goja and the patch header cited a V8-side test that did not exist. This adds it, with the line-terminator edge cases, and it fails when the patch checks CR instead of LF. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * build(nix): vendorHash for goja (TASK-2198 U2) Taken from the Nix job's recompute on c7fa1931 (specified sha256-iEqv…, got sha256-P1bQ…), so main does not depend on the heal job after merge. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC	2026-09-30 02:50:10 -04:00