mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-10-26 06:19:49 +00:00
main
15 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xarmian
|
08a5f1c7d2 |
fix(editor): a block marker after a hard break stays text (BUG-3584) (#2060)
* fix(editor): a block marker after a hard break stays text (BUG-3584) prosemirror-markdown escapes a leading -, +, *, >, # or 1. only at a block start, but a hard break starts a new line too, so 'First line' + hard break + '- not a list item' was stored as 'First line\\\n- not a list item' and read back as a paragraph plus a bullet list. PadText treats a line start as a block start for that escape, restoring the flag after. Editor and materializer share PadText (htmlProseExtensions), so parity holds. Claude-Session: https://claude.ai/code/session_01FAVXA5mJ5BNhKmv95jxQkQ * test(materialize): corpus covers block markers after a hard break (BUG-3584) The BUG-3583 corpus block gains lines starting '- ', '# ' and '> ' after hard breaks (one block for both, so adding it did not reshuffle the seeded cases); a vantage probe pins that the corpus reaches the escaped form. corpus.json regenerated; go test ./internal/materialize passes on the rebuilt bundle. Claude-Session: https://claude.ai/code/session_01FAVXA5mJ5BNhKmv95jxQkQ * fix(editor): every block marker after a hard break stays text, and nothing else changes (BUG-3584) The first fix (atBlockStart for the line) missed `1)`, indented bullets, thematic breaks and setext underlines, and escaped markers that open no block. PadText now writes, at any line start, the escaped form of exactly the markers that would open a block (lineStartMarker): a bullet or ordered marker followed by a space/tab/end, `>`, `#{1,6}` followed by a space/tab/ end, or a line of only -, =, _ or *. `#tag`, `-5`, `1.5` keep their form. At a true block start prosemirror-markdown's escape stays in charge except for what it missed (`1)` and an indented bullet/quote, both lists before). Measured against main's serializer over 28 strings at a block start and after a hard break: every difference is one of those bug fixes. Corpus regenerated (the first version's `\-item` escape is gone); go test ./internal/materialize passes on the rebuilt bundle; 22 unit tests. Claude-Session: https://claude.ai/code/session_01FAVXA5mJ5BNhKmv95jxQkQ |
||
|
xarmian
|
8d910dc20a |
fix(editor): bold, italic and strike always serialize to markdown that reads back (BUG-3583) (#2057)
* fix(editor): bold, italic and strike always serialize to markdown that reads back (BUG-3583) CommonMark lets a `**` / `*` / `~~` run open only where it is left-flanking and close only where it is right-flanking. A mark whose text starts with punctuation after a letter, or ends with punctuation before a letter, has no delimiter that can do both. Measured on the headless editor with the editor's own parser: every such case lost its mark on reload. tiptap-markdown slides an unflankable delimiter inward (the punctuation silently loses the mark), and skips that for a mark at output offset 0, so a bold run starting a block went out as `**Hello.**World` and read back as literal asterisks. flankingMarks.ts replaces bold, italic and strike (and re-adds code and underline, keeping StarterKit's mark order) in the body editor, the materializer (htmlProseExtensions, so parity holds) and the comment editor: - each run is written with its delimiter and, when it closes, tested against the characters actually around it with markdown-it's scanDelims; a run that cannot flank is rewritten to <strong>/<em>/<s>, which parse back to the same mark (html: true) and render through renderMarkdown's allow-list; - tiptap-markdown's sliding is switched off for these marks only (a guarded patch on prosemirror-markdown's base markString); - a textblock whose output could still mis-pair (a run ending in punctuation before a letter, or merged same-character runs) is parsed back and compared per character; on a mismatch it is re-rendered with the HTML form. Code needs nothing (measured). Fuzz (6000 paragraphs of letters, punctuation and spaces under overlapping marks, links and hard breaks, in paragraphs, quotes and list items): marks lost 0 (old serializer 211-228 per 1500); text altered 0-1 (old 163-172, the rest a pre-existing line-start escaping gap); output changed where the old one already round-tripped: 3 per 1500, all one whitespace placement. Serialization time on 500 paragraphs: equal to main for ordinary prose; the extra parse runs only on the textblocks that can fail. Corpus: a block with the failing shapes is added to the materializer parity generator; corpus.json regenerated; Go materialize tests pass on the new bundle. New e2e: typed bold "Hello." + "World" is stored as <strong>Hello.</strong>World; on main it is stored as **Hello.**World. (A reload of an open tab does not show the defect on main: it re-seeds from the op-log. The stored body is what readers and a pruned editor see.) Claude-Session: https://claude.ai/code/session_01FAVXA5mJ5BNhKmv95jxQkQ * fix(web): backlink snippets drop inline tags; every reader of an HTML mark pinned (BUG-3583) The lead's check before GO: the fix stores runs markdown delimiters cannot express as inline HTML, so every other reader of items.content must show it as formatting or plain text, never literal tags. - Share pages and their collection views: renderMarkdown's allow-list renders <strong>/<em>/<s> (e2e). - The editor seeding from the body (after an op-log prune): bold (e2e). - Search palette snippets: already stripped tags (e2e pins it). - Backlink snippets showed the raw slice, so `<strong>` appeared literally, and the slice could cut a tag in two (`…g>Hello.`). Tags and cut fragments are now dropped. Red on main in the new e2e, green here. - Email/notification templates carry no item content; the workspace and artifact exports hand out the raw body, where inline HTML is valid CommonMark/GFM; nothing syncs bodies to GitHub (read from the code). Claude-Session: https://claude.ai/code/session_01FAVXA5mJ5BNhKmv95jxQkQ |
||
|
xarmian
|
78ec772226 |
fix(collab): recovery refuses an incomplete op-log replay, and a failed compaction snapshot keeps the log (TASK-3574 (1) and (3)) (#2045)
From the day-91 outside review's op-log durability claims, both measured
and ruled by the lead.
(1) Recovery (materializeWith) serialized an op-log whose replay left
structs PENDING, because they referenced a row the log did not hold. It
returned a truncated but nonblank body ("alpha" for "alpha beta gamma"),
which the store's only guard (blank output over a nonblank body) let
through: the recovery flush stored it and moved the watermark over content
it never saw. Compaction (snapshotWith) always refused the same input.
- The bundle now refuses it too ("pending structs").
- The Runner and the supervisor map that to a typed
materialize.ErrIncompleteLog (wrapped with ErrJobFailed through the
supervisor).
- Recovery treats it as deterministic: it exhausts at once with kind
incomplete_log, so the existing WARN exhaustion line and
pad_materialize_giveup_total{kind="incomplete_log"} are the refusal's log
and metric. The body and watermark do not move, and there is no retry
until the op-log changes.
- The GC sweep already leaves such an item's rows alone: it prunes only
fully flushed logs. A test pins it.
(3) With compaction on (PAD_OPLOG_COMPACT, default off), a failed snapshot
left the item to the sweep, which deleted its log and with it the Yjs
identity compaction exists to keep. A failed (or unreadable) snapshot now
KEEPS the log and retries with in-memory backoff: 10 minutes, doubling,
capped at 24h; a restart retries once. Success clears the backoff, and
entries for items no longer listed are dropped. The test that asserted the
delete is flipped to assert keep, back off, then compact (lead ruling).
Tests: the bundle refuses the incomplete log while the complete one
materializes; recovery leaves the body, seq and state untouched, gives up
once with incomplete_log, and the sweep does not list the item; the
compaction keep/backoff/retry test. Counterfactuals: without the bundle's
refusal, the bundle returns "alpha" and recovery APPLIES the truncated body;
without the keep, all 27 rows are deleted.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
|
||
|
xarmian
|
92305fba4d |
fix(web): marks never sit on inline atoms, so a tab's editor equals its Y.Doc (BUG-3568) (#2038)
ProseMirror lets a mark sit on an inline atom (a hard break, an inline attachment image or chip). Bolding, striking or linking a range that held one marked it, and markdown or HTML parsing marked it when the source wrapped it (**a\ b**). y-tiptap never writes marks on non-text inline nodes to the Y.Doc, so that tab's editor held a document its own Y.Doc did not. It flushed one body while every other tab, a reload and the materializer derived another. This is the ~0.7% tab divergence materializerParity found. Measured: both Y.Docs identical, and only the marking tab's ProseMirror doc differs; a one-tab repro with no relay reproduces it 4 of 4. InlineAtomNoMarks (extensions/inlineAtomNoMarks.ts) is an appendTransaction plugin that strips marks from inline non-text nodes in the same update. It fires only on AddMarkStep, AddNodeMarkStep, or a replace whose slice holds a marked atom, so typing pays nothing. It is registered in Editor.svelte and the headless materializer. No node spec or persisted shape changes, so no SCHEMA_VERSION question; NodeSpec.marks cannot do this, since it governs a node's content (measured: it left the marks in place). One-time normalisation: a body holding a mark across a hard break (**a\ b**) flushes as **a**\ **b**, the form every Y-derived door already produced, so this touches only bodies that already flip-flopped. It is stable from then on. materializerParity: MAX_TAB_DISAGREEMENTS 2 -> 0. The corpus is regenerated; seed-300922 is now a case. The generator is not deterministic run to run (Yjs client ids), so a regen also rewrites a couple of other cases' expectations; each corpus is self-consistent. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
1eecf710c8 |
fix(web): angle-bracketed prose survives the editor: <Button>, Promise<T>, comments and unknown tags are text, not dropped markup (BUG-3557) (#2035)
The editor parsed markdown with html enabled, so `<Name>` in prose was a tag the schema could not hold: dropped on the next flush (one keystroke, or an agent's write applied by an open tab, which answered 200 while the text vanished). `<Table>` became an empty table that was then removed. htmlProse.ts derives the allowlist from the schema's own parse rules, so the HTML the editor itself writes (tables with spans, HTML-only marks) still reads back: - an inline tag passes only if written lower case, matching an inline node's or a mark's rule, and closing one that passed; anything else is literal text; - an HTML block passes only if every tag in it matches a rule; otherwise it is kept whole as an HtmlBlock (stored as a ```html fence); - comments are visible text. PadText escapes `<` only where the parse would read a tag, so `Promise<T>` round-trips byte for byte. Both the editor and the headless materializer take the same extensions, right after StarterKit. materializerParity now skips and counts cases where the two live tabs converge to different documents (BUG-3568, pre-existing: 2 of 300 seeds on main), at most 2 of 50, and still requires the materializer to match one of them; such cases stay out of the Go corpus (lead ruling). Claude-Session: https://claude.ai/code/session_01FAVXA5mJ5BNhKmv95jxQkQ |
||
|
xarmian
|
23c8e4f811 |
feat(collab): the GC compacts dormant op-logs and a covered resume merges (TASK-3531 U3-U5) (#1985)
* feat(collab): the GC tick compacts dormant op-logs and Join admits a covered resume (TASK-3531 U3-U5) The second half of TASK-3531, behind PAD_OPLOG_COMPACT=on (default off). U3: with a compactor set, each op-log GC tick first compacts every dormant, flushed op-log into one Yjs snapshot (#1984's primitives): the snapshot is built on the materializer worker WITHOUT the item lock, then swapped in under the lock with no room open, and CompactItemOpLog re-checks the log is the one it read. The sweep then runs as PruneSweepKeeping and keeps compacted logs. Anything not compacted (a failed job, a refused swap, set-aside rows, another schema) is left to the sweep, which deletes it as before. cmd/pad builds the materializer before the GC starts and hands it over as the compactor only when PAD_OPLOG_COMPACT=on and the materializer is enabled. U4: Join admits a resume cursor the snapshot covers (CompactedResumeCovers, which requires the snapshot row to exist) instead of force_refreshing it; the tab replays the snapshot and its own edits merge. A failed check refreshes. Telemetry for the default flip (lead ruling): pad_oplog_compactions_total {outcome}, pad_oplog_compaction_snapshot_bytes_total against pad_oplog_compaction_markdown_bytes_total, pad_collab_compacted_resumes_total. U5: server tests (compact then keep, the no-compactor and failed-job deletes, a room opened mid-job blocks the swap, a covered resume admitted vs refreshed) and an opt-in e2e (a tab asleep across the sweep merges, or with compaction off is handed its text back). Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(server): cap compaction jobs per GC tick and keep the deferred items (TASK-3531, codex) Compaction shares the materializer's single job slot with op-log recovery, which carries unflushed edits, so a dormant backlog must not hold it for a whole pass. At most 25 snapshot jobs a tick; items past the cap are kept from that tick's sweep (deleting them would lose their chance) and compacted by a later tick. The opt-in e2e's mode=off timeout now names the likely cause. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * test(materialize): updates built on pre-compaction structs merge onto the snapshot (TASK-3531) The merge property in CI, not only in the opt-in e2e (lead): over the corpus, a snapshot of the first k rows followed by the rows after k materializes to the full document (150 cases). Those later rows depend on the prefix's structs (lone, 114 of them are pending), so this is the dependency resolution a deleted-and-reseeded log fails. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
1a80389341 |
feat(collab): compact a dormant op-log into one Yjs snapshot (TASK-3531 U1+U2) (#1984)
* feat(collab): compact a dormant op-log into one Yjs snapshot (TASK-3531 U1+U2) The first half of TASK-3531. Nothing calls it yet: the sweep (U3) and Join (U4) come in the second PR, behind PAD_OPLOG_COMPACT (default off). U1, the materializer: snapshot() replays an op-log and returns its markdown plus the WHOLE document as one y-protocols Update frame. It refuses a frame whose replay alone does not give the same Yjs snapshot and markdown, and an op-log that leaves structs pending. A row that does not replay at all contributes nothing (here or in any tab) and is compacted away. Runner.Snapshot, a `mode` on the worker protocol, and Supervisor.Snapshot share Materialize's job slot, caps and deadline. U2, the store: items.yjs_compacted_through (the highest op-log id a snapshot covers) and items.yjs_snapshot_op_id (the snapshot row). CompactItemOpLog swaps the rows for the snapshot in one transaction iff the log is unchanged since it was read, still dormant and fully flushed, and moves the flush watermark to the snapshot. CompactedResumeCovers admits a cursor only while the snapshot row exists, so a path that deletes the log can never leave the columns admitting a stale tab; the three that delete the whole log also clear them. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * docs(store): CompactItemOpLog's no-append contract, and why the tx is not the exclusion (TASK-3531, codex) Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
4ed3b5eca2 |
fix(materialize): a job's deadline scales with its op-log (BUG-3521) (#1946)
* fix(materialize): a job's deadline scales with its op-log (BUG-3521) Op-log recovery could not finish a large document: the real op-log of an 86 KB plan (278 content-bearing rows, 470 KiB) took 1.36 s median in-process on an idle box and 1.77 s on one core, against a flat 2 s hard / 1.8 s soft deadline. It failed every attempt and exhausted its budget, leaving items.content stale for the CLI, MCP, search and exports until someone reopened the item. Each job's hard deadline is now the configured base (PAD_MATERIALIZE_TIMEOUT, 2 s) plus 20 ms per KiB of its rows, never below the base and at most MaxTimeout (60 s): ~5x the measured single-core rate, so that document gets ~11 s. The supervisor computes it from the job, so every caller gets it. The soft deadline stays at 90%, and recovery runs off the request path in its own worker. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(materialize): the worker-start line states the deadline rule, not only its base (BUG-3521, codex r1) Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
b67e78343a |
test(materialize): fire the idle timer mid-job on purpose, not by wall-clock luck (BUG-3435) (#1827)
TestSupervisorIdleTimerDuringJob staged 'the idle timer fires while a job runs' with a 300ms timer, a 200ms pause and a 400ms job. On a loaded macOS runner the pause stretched past the timer, the worker went idle BETWEEN the jobs, and the test failed with '2 spawns'. Widening the pause to 320ms reproduces that exact failure 3/3. Now nothing fires on its own (a one-hour idle). A new script command, await:<path>, holds the job inside the worker until the test releases it; the test fires the timer itself (back-dating lastJob, Reset(0)) once the job has provably started, and waits on a new test hook, onIdleBusy, which idleStop calls when it finds the slot taken. Then it releases the job, checks one spawn and that the job's end re-armed the timer on the same worker, and fires once more to see the worker stop. Mutants: idleStop proceeding without the slot check (red 10/10), and armIdle not recording the job's end (red). 30/30 green with every CPU saturated, and under -race. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
f0c65fe08c |
chore(web): adopt Tiptap 3.31.4 and regenerate the materializer corpus (TASK-3421) (#1811)
* chore(web): adopt Tiptap 3.31.4 and regenerate the materializer corpus (TASK-3421) 3.31.4's TableCell/TableHeader renderHTML stop rendering a default colspan="1" / rowspan="1". parseHTML is unchanged, so a stored body carrying them parses to the same document. Replaying all 50 old corpus op-logs through the 3.31.4 bundle reproduces the old output exactly once those attributes are removed (and fails on 28 cases without removing them). Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * test(web): pin that a pre-3.31.4 table body still dedupes on open (TASK-3421) Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
cc2194552b |
test(materialize): two CI reds were test instruments, not the product (TASK-2198) (#1694)
main was red at
|
||
|
xarmian
|
cb99879d53 |
feat(materialize): recover unsaved editor sessions into items.content (TASK-2198 U4) (#1683)
* feat(materialize): recover unsaved editor sessions into items.content in the background (TASK-2198 U4) Wires the U3 supervisor in. A collab room reclaimed at the end of its grace TTL with content-bearing rows above the flush watermark (T3), or a dormant pending item found by a one-minute sweep (T1), is queued; one worker goroutine materializes the item's whole content-bearing op-log with the whole-workspace link index and lands it through store.MaterializeFlush, under the collab item lock with no room open. The write is a flush: cursor == MAX(op-log id), rows pending, no set-aside rows, checked in the write's own transaction; watermark to the cursor only; seq bumped; one version row attributed to system/recovery; the op-log is never pruned. PAD_MATERIALIZE=off disables it entirely. A per-item in-memory failure budget (3 failures, doubling backoff, reset by a new op-log row) keeps untrusted input from being retried in a loop. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(store): recovery watermark is the cursor, not MAX at UPDATE time (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(store): a second recovery still writes its own version row (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * docs(mcp): op-log recovery bounds the stale-body window; tool surface v0.60 (TASK-2198 U4) content_state applied_pending_flush now clears on its own after the item's room closes, unless recovery fails or PAD_MATERIALIZE=off. Every sentence that called the window unbounded, or said nothing server-side moves op-log content into items.content, is corrected (CLAUDE.md, instructions.md, README, docs/deployment.md, docs/backup.md, code comments). BEHAVIOR bump to 0.60; goldens regenerated. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): count ErrNoMemoryCap silently against the per-item budget (TASK-2198 U4) U3 now refuses every job with ErrNoMemoryCap when no memory cap can be established, and warns once itself. The recovery worker counts it like any failure but logs it at debug, not per item. Its log lines go through a logger seam so the test can tell. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * style: gofmt (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * feat(materialize): one log line per budget exhaustion; count set-aside skips on the sweep line (TASK-2198 U4) An item that uses up its failure budget logs exactly one WARN naming the item, the failure count and the last error kind (DEBUG when the kind is no_memory_cap, which the supervisor already warned about once), and logs again only after op-log growth resets the budget and it is exhausted anew. The schema-version skip goes through the same line. Items skipped for set-aside rows are counted and reported on a new single sweep summary line, written only when it has news; never per item. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): keyset-page the recovery sweep so no candidate starves (TASK-2198 U4, codex r1) The sweep read the oldest 200 candidates from the start every tick, and the room check and the failure budget filtered them afterwards. Once 200 sat in backoff or exhausted, every sweep returned the same 200 and the 201st was never queued: filtering above a bound is starvation. Each tick now reads one page strictly after an in-memory keyset cursor (newest op-log created_at, item id) and wraps to the start after a short page, so every candidate is offered within ceil(N/limit)+1 ticks, whatever sorts ahead of it. Keyset over NOT IN: no parameter-count limit, and it also covers the room check, which an exclusion list would not. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(materialize): the sweep cursor wraps; docs for the exhaustion line, the set-aside count and paging (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * docs(deployment): reflow the recovery budget paragraph (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(materialize): recovery strips the follows-title marker as a tab's save does (TASK-2198 U4, BUG-3315) Since #1688 a tab loads a link that follows its target's title with the follows-title marker in the link's title attribute, so the marker lives in Y.Docs that recovery replays. Three fixture cases (a follows-title link; the same link after a rename; a URL edited to an external address) are generated from documents that really hold the marker, and the existing 8 are unchanged. A new leg recovers each through the production path (whole-workspace index) and through the runner with a restricted and an empty index, and asserts no 'pad-follows-title' reaches the body. It goes red in all 7 ways when the bundle's marker strips are disabled. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(store): a recovery version row carries #1691's history data (TASK-2198 U4) Rebased onto PLAN-2348 U2 (#1691), which adds user_id, line counts and a create marker to version rows. Recovery writes its row through the same UpdateItem insert, so it gets them automatically: no user_id (a system row), not a create, and line counts equal to diff.LineCounts of the stale and recovered bodies. The leg pins that. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC |
||
|
xarmian
|
d78ff9bdc5 |
feat(materialize): worker supervisor with per-OS memory cap (TASK-2198 U3) (#1682)
* feat(materialize): parent-side worker supervisor with per-OS memory cap (TASK-2198 U3) The Supervisor runs materializer jobs in ONE long-lived child process (`pad __materialize-worker`), started lazily on the first job and fed jobs serially. A job past PAD_MATERIALIZE_TIMEOUT (default 2s) kills the child (goja's Interrupt, sent as timeout_ms a tenth earlier, is only the soft layer); a dead child is respawned on the next job after an exponential backoff (1s doubling, capped at 5 min, reset by a child that lived a minute). Every stop is logged with its reason (deadline, memory, exit, protocol, start, canceled, closed); every start logs the effective timeout, memory limit and cap mechanism. The child's stderr goes to the log, pid-tagged and bounded, never near the protocol. A job whose child died is never retried here. Memory cap (PAD_MATERIALIZE_MEM_LIMIT, default 2GiB, bytes or KiB/MiB/GiB, clamped to 1536MiB-16GiB): - Linux: RLIMIT_AS set on the child by prlimit before its first byte of input; the child only runs its own startup uncapped. - Windows: Job Object with JOB_OBJECT_LIMIT_PROCESS_MEMORY; the child is created suspended, assigned, then resumed: no uncapped window. - macOS: parent-side RSS watchdog every 100ms (proc_info syscall, ps fallback). - Other: deadline kill only. Not wired into the server yet (U4). Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * ci(materialize): run the supervisor on the real macOS and Windows caps (TASK-2198 U3) native-smoke now builds the real materializer bundle (as the go job does) and runs the supervisor, memory-cap and deadline tests on macos-latest (proc_info RSS watchdog) and windows-latest (suspended start + Job Object). PAD_MATERIALIZE_TEST_NO_SKIP turns the cap tests' skips into failures there, so a cap that stops running cannot pass as skipped. Tests: a real Windows processGone (OpenProcess + GetExitCodeProcess), RSS samplers for darwin (the production proc_info path) and Windows (K32GetProcessMemoryInfo), and an allocation helper that writes noise into every byte so macOS memory compression cannot hide it from the watchdog. The worker-loop tests share one loaded Runner through serveWorker (RunWorker's loop, split out); TestWorkerRoundTrip still goes through RunWorker end to end. go test -race -count=3 on the package: 714s -> 413s. docs: the materializer section says the worker and its knobs are inert until U4 wires it in. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): a worker never outlives its supervising process (TASK-2198 U3, codex r1) If the server died mid-job (SIGKILL, OOM kill, crash) its worker ran on with no deadline kill and, on macOS, no memory watchdog. Two layers: - Linux: Pdeathsig = SIGKILL. The fork runs on a fresh goroutine (startUnlocked), never locked to its thread, so it cannot land on a thread the runtime later retires (golang/go#27505); Go's forkExec re-checks getppid if the parent is already gone. - Every Unix (the only layer on macOS): the worker command calls ExitWhenOrphaned before the bundle loads, polling its parent pid every 200ms against PAD_MATERIALIZE_PARENT_PID and exiting (status 70) when it changes. It writes nothing. - Windows was already covered by the Job Object's KILL_ON_JOB_CLOSE. TestSupervisorOrphanedWorkerExits kills a helper process that supervises a worker hung in a job and requires the worker gone within 1s: 0s on Linux, 105ms with the ppid watch alone; with both layers removed the worker outlives its parent by 5s. It matches native-smoke's TestSupervisor.* selection, so it runs on macOS and Windows too. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): relative memory cap, refuse when uncappable, GOMAXPROCS=2; nix-safe ps test (TASK-2198 U3) The absolute Linux RLIMIT_AS failed on CI: a 64 MiB allocation died under a 1600 MiB cap, because the runtime there reserved more address space than on the machine the floor was measured on. The cap is now RELATIVE on every OS: once the worker has loaded and before it is sent any job, its usage is read as a baseline and the cap is set to baseline + PAD_MATERIALIZE_MEM_LIMIT — address space on Linux (/proc/<pid>/stat vsize, prlimit, read back), committed memory on Windows (the Job Object's per-process limit, set after the suspended start and load), resident memory on macOS (the watchdog's threshold). Every start logs baseline, limit and effective cap. Lead ruling, REFUSE: if the baseline probe fails or is implausible (0, or above MaxPlausibleBaseline = 8 GiB, ~4.8x the largest measured 1704 MiB), onBaselineProbeFailure kills the worker without sending it the job and fails the job with ErrNoMemoryCap; one WARN until a probe succeeds again. An OS with no cap mechanism never starts a worker and refuses every job with ErrNoMemoryCap, one WARN naming runtime.GOOS. The worker runs with GOMAXPROCS=2: measured loaded-worker baseline 1357 MiB vs 1597 MiB at this host's default (8) and 1704 MiB at 64. MinMemLimit drops from 1536 MiB (an artefact of the absolute cap) to 256 MiB: the corpus grows the worker by at most +10 MiB of address space and +18 MiB resident, and a 4,000-deep nested document by +129..194 MiB. A cgo worker's "pthread_create failed" at the cap is now classified as a memory death. Nix: TestParsePSRSS parses fixed strings only; the live `ps` leg is TestPSRSSLive and skips when there is no `ps` (the Nix sandbox has none; ps is only macOS's fallback), unless PAD_MATERIALIZE_TEST_NO_SKIP is set. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): Windows kills on the job's memory notification; steady thread set before the baseline (TASK-2198 U3) Windows (CI on c1866957): the Job Object refused the 1 GiB commit (VirtualAlloc errno=1455), but the worker did not die and the job ran to its deadline. The cap no longer depends on the runtime choosing to crash: the job is associated with an I/O completion port, and JOB_OBJECT_MSG_PROCESS_MEMORY_LIMIT for the worker's pid kills it at once (ErrMemoryLimit). errno=1455 on stderr is a second signal. On every OS a cap mechanism that sees the limit hit calls onLimit, and the kill is reported as "reached its <mechanism> cap". Linux (CI's cgo test binary at GOMAXPROCS=64): threads created after the baseline ate the growth budget. The worker now brings its thread set to steady state before readiness (warmThreads: an allocating goroutine per P, then a GC). Measured on a cgo build: threads added after the baseline are 0 at GOMAXPROCS=2 with or without it; at 64, +3 threads and +272 MiB of address space during the corpus without it, +0 and +0 MiB with it. 256 MiB stays the floor (largest growth at GOMAXPROCS=2: +193 MiB, a 4,000-deep nested document; the corpus adds +0 MiB). The cap test's second leg no longer uses GOMAXPROCS=64 (which also changed thread creation) to stand in for a bigger host: the helper inflates its own baseline before readiness, with never-touched address space on Linux, committed memory on Windows and resident memory on macOS. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * feat(materialize): the worker stops after an idle timeout; the next job restarts it (TASK-2198 U3) Dave's ruling on the worker process: GO, with an idle exit. A loaded worker holds ~200 MB resident, which an install that rarely materializes should not pay for all day. PAD_MATERIALIZE_IDLE_TIMEOUT: default 5m, clamped to 30s-24h, 0 = never. After each job the supervisor (re)arms a timer. When it fires, the stop runs only while holding the job slot, so it never takes a worker with a job in flight. It re-checks the idle time, forgets the worker WITHOUT recording a death (backoff is untouched), and closes the worker's stdin so its read loop ends with status 0, with a kill after a grace if it does not go. The stop is logged at INFO, reason=idle. A job that arrives meanwhile waits for the slot, finds no worker, and starts a new one with no backoff wait. Legs: the idle exit after the timeout (INFO, status 0, no WARN); the next job respawns and succeeds with no wait; an idle stop leaves the backoff streak alone (crash, idle, crash waits 2s, not 4s); a job arriving during the stop runs on a new worker; a timer that fires mid-job does not stop that worker; NeverIdle arms no timer; env parsing and clamps. Mutations, each red on its own: recording a death (3 legs), logging as a kill (1), not holding the slot (1), never arming (3). Docs: the knob beside the other two, and `idle` in the stop reasons. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): Close does not return while an idle-stopping worker still runs (TASK-2198 U3, codex r1) An idle stop forgets the worker before it has exited, so a Close in that window found no worker and returned with it still alive. The supervisor now tracks the worker an idle stop is ending, and Close kills and reaps it. A leg pins it, and goes red without the fix. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): an idle stop that Close ended says so (TASK-2198 U3, codex r2) The idle stop stored its kill reason unconditionally, so it overwrote the 'closed' of a Close that had already killed the worker, and its stop line said idle. It now claims the reason only if none is set, and the line reports reason=closed. The Close-during-idle-stop leg asserts it. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC |
||
|
xarmian
|
1e16d2228b |
fix(web): link text with markdown/HTML-significant characters survives a save (BUG-3315 U1) (#1686)
A wiki link's display text was mangled on its way through the editor, and the first save of any item linking such a title pinned the mangled text as a display override, with no rename involved (a census found 76 stored overrides of this shape among 107 that differ from their target's title). Load: escapeMarkdownLinkText escaped only brackets and backslashes, so the html-enabled parser read the title as markdown. `*real*` became emphasis, `<topic>` was dropped as a tag, `>` was decoded. It now backslash-escapes \ [ ] * _ ~ ` < > &, all ASCII punctuation CommonMark lets a backslash escape. Save: unescapeMarkdownLinkText undid only bracket and backslash escapes, while the serializer also backslash-escapes other punctuation and writes < > & as entities. It now undoes both, in one pass. One residue is intrinsic and pinned by a test: the serializer writes a literal `>` and a literal `>` as the same bytes, so a title containing entity-shaped text still becomes an override (0 of 9,620 live titles do). The materializer corpus: two expectations change (a backtick and `<b>` in link text, both this fix), taken from the goja bundle's output. A full regeneration was not used because the generator is nondeterministic: with unchanged code it rewrites all 50 cases' rows and 1 expected value. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC |
||
|
xarmian
|
fa2755e183 |
feat(materialize): headless op-log materializer bundle, goja runner and worker (TASK-2198 U2) (#1678)
* feat(materialize): headless op-log materializer bundle, goja runner and worker (TASK-2198 U2)
Builds the editor's own JS (Yjs -> y-tiptap -> Tiptap schema -> tiptap-markdown
-> the tab's flush pipeline) into one IIFE, embeds it, and runs it in goja. Not
wired into the server yet (U3 supervisor, U4 write path).
- web/src/lib/collab/materializer/: bundle entry (extension list mirrors
Editor.svelte's collab branch via the U1 shared modules), op-log replay
mirroring wsProvider's MESSAGE_SYNC arm, linkedom DOM shim and polyfills
from the spike, and a vendored patch of prosemirror-markdown
MarkdownSerializerState.atBlank (quadratic under goja; equivalent).
- web/scripts/build-materializer.mjs (esbuild) -> web/build-materializer/,
run by `npm run build`; embedded as pad.MaterializerJS.
- internal/materialize: Runner (Go-native atob/btoa/TextEncoder/TextDecoder/
getRandomValues, schema-version check, ctx interrupt with a clean VM after),
and RunWorker's length-prefixed JSON protocol behind the hidden
`pad __materialize-worker` command.
- Parity: vitest drives two real Editor.svelte mounts and compares
materialize() with the tab's flush on 50 seeded cases; the same cases are
committed as testdata/corpus.json and replayed in goja (TestCorpusParity).
- CI Go jobs, release tests, Docker and Nix produce the bundle; linkedom and
esbuild are new devDependencies.
Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC
* test(materialize): pin the atBlank patch's equivalence under V8 too (TASK-2198 U2, codex r1)
Codex r1 claimed upstream's /(^|\n)$/ is true for a trailing CR in JS. It is
not (checked in V8: false for "\r" and "a\u2028", as the patch returns),
but the only equivalence test ran under goja and the patch header cited a
V8-side test that did not exist. This adds it, with the line-terminator edge
cases, and it fails when the patch checks CR instead of LF.
Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC
* build(nix): vendorHash for goja (TASK-2198 U2)
Taken from the Nix job's recompute on
|