From the day-91 outside review's op-log durability claims, both measured
and ruled by the lead.
(1) Recovery (materializeWith) serialized an op-log whose replay left
structs PENDING, because they referenced a row the log did not hold. It
returned a truncated but nonblank body ("alpha" for "alpha beta gamma"),
which the store's only guard (blank output over a nonblank body) let
through: the recovery flush stored it and moved the watermark over content
it never saw. Compaction (snapshotWith) always refused the same input.
- The bundle now refuses it too ("pending structs").
- The Runner and the supervisor map that to a typed
materialize.ErrIncompleteLog (wrapped with ErrJobFailed through the
supervisor).
- Recovery treats it as deterministic: it exhausts at once with kind
incomplete_log, so the existing WARN exhaustion line and
pad_materialize_giveup_total{kind="incomplete_log"} are the refusal's log
and metric. The body and watermark do not move, and there is no retry
until the op-log changes.
- The GC sweep already leaves such an item's rows alone: it prunes only
fully flushed logs. A test pins it.
(3) With compaction on (PAD_OPLOG_COMPACT, default off), a failed snapshot
left the item to the sweep, which deleted its log and with it the Yjs
identity compaction exists to keep. A failed (or unreadable) snapshot now
KEEPS the log and retries with in-memory backoff: 10 minutes, doubling,
capped at 24h; a restart retries once. Success clears the backoff, and
entries for items no longer listed are dropped. The test that asserted the
delete is flipped to assert keep, back off, then compact (lead ruling).
Tests: the bundle refuses the incomplete log while the complete one
materializes; recovery leaves the body, seq and state untouched, gives up
once with incomplete_log, and the sweep does not list the item; the
compaction keep/backoff/retry test. Counterfactuals: without the bundle's
refusal, the bundle returns "alpha" and recovery APPLIES the truncated body;
without the keep, all 27 rows are deleted.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* feat(collab): compact a dormant op-log into one Yjs snapshot (TASK-3531 U1+U2)
The first half of TASK-3531. Nothing calls it yet: the sweep (U3) and
Join (U4) come in the second PR, behind PAD_OPLOG_COMPACT (default off).
U1, the materializer: snapshot() replays an op-log and returns its
markdown plus the WHOLE document as one y-protocols Update frame. It
refuses a frame whose replay alone does not give the same Yjs snapshot
and markdown, and an op-log that leaves structs pending. A row that does
not replay at all contributes nothing (here or in any tab) and is
compacted away. Runner.Snapshot, a `mode` on the worker protocol, and
Supervisor.Snapshot share Materialize's job slot, caps and deadline.
U2, the store: items.yjs_compacted_through (the highest op-log id a
snapshot covers) and items.yjs_snapshot_op_id (the snapshot row).
CompactItemOpLog swaps the rows for the snapshot in one transaction iff
the log is unchanged since it was read, still dormant and fully flushed,
and moves the flush watermark to the snapshot. CompactedResumeCovers
admits a cursor only while the snapshot row exists, so a path that
deletes the log can never leave the columns admitting a stale tab; the
three that delete the whole log also clear them.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* docs(store): CompactItemOpLog's no-append contract, and why the tx is not the exclusion (TASK-3531, codex)
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* feat(materialize): headless op-log materializer bundle, goja runner and worker (TASK-2198 U2)
Builds the editor's own JS (Yjs -> y-tiptap -> Tiptap schema -> tiptap-markdown
-> the tab's flush pipeline) into one IIFE, embeds it, and runs it in goja. Not
wired into the server yet (U3 supervisor, U4 write path).
- web/src/lib/collab/materializer/: bundle entry (extension list mirrors
Editor.svelte's collab branch via the U1 shared modules), op-log replay
mirroring wsProvider's MESSAGE_SYNC arm, linkedom DOM shim and polyfills
from the spike, and a vendored patch of prosemirror-markdown
MarkdownSerializerState.atBlank (quadratic under goja; equivalent).
- web/scripts/build-materializer.mjs (esbuild) -> web/build-materializer/,
run by `npm run build`; embedded as pad.MaterializerJS.
- internal/materialize: Runner (Go-native atob/btoa/TextEncoder/TextDecoder/
getRandomValues, schema-version check, ctx interrupt with a clean VM after),
and RunWorker's length-prefixed JSON protocol behind the hidden
`pad __materialize-worker` command.
- Parity: vitest drives two real Editor.svelte mounts and compares
materialize() with the tab's flush on 50 seeded cases; the same cases are
committed as testdata/corpus.json and replayed in goja (TestCorpusParity).
- CI Go jobs, release tests, Docker and Nix produce the bundle; linkedom and
esbuild are new devDependencies.
Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC
* test(materialize): pin the atBlank patch's equivalence under V8 too (TASK-2198 U2, codex r1)
Codex r1 claimed upstream's /(^|\n)$/ is true for a trailing CR in JS. It is
not (checked in V8: false for "\r" and "a\u2028", as the patch returns),
but the only equivalence test ran under goja and the patch header cited a
V8-side test that did not exist. This adds it, with the line-terminator edge
cases, and it fails when the patch checks CR instead of LF.
Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC
* build(nix): vendorHash for goja (TASK-2198 U2)
Taken from the Nix job's recompute on c7fa1931 (specified sha256-iEqv…,
got sha256-P1bQ…), so main does not depend on the heal job after merge.
Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC
xarmian