mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-11 21:39:01 +00:00
c4d429d14c94de941ac761333cdac0a33895a05f
112 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xarmian
|
b437cc582d |
feat: item reminders — the fire-at-an-instant primitive, and one overdue rule for all four surfaces (IDEA-2641, closes #1010) (#1244)
* feat(store): item reminders — the fire-at-an-instant primitive (IDEA-2641) Adds the storage, the scheduler tick, and the canonical event for one-shot item reminders (GitHub #1010). Nothing in Pad acted at a target time before this: a due_date makes an item show up as overdue once somebody asks the dashboard, so "revisit TASK-X on the 1st" had to live in an external cron. A TABLE, NOT A SCHEMA-FIELD ANNOTATION. The design sketch proposed marking schema date fields with a `reminds: true` key on models.FieldDef; recon overturned it. Such a key does not survive an ordinary collection edit, two independent ways: the web editor destructures each field into an EditableField and rebuilds a fresh definition key-by-key on save, so unknown keys are dropped (`pattern` and `unique_scope` survive only because two lines were hand-added for them), and models.CollectionSchema has fixed fields with no catch-all, so any Go unmarshal+marshal round-trip strips unknown properties — the hazard retargetRelationFieldsTx mutates raw JSON to avoid. Both failures are silent and both disarm a whole collection's reminders at once. It is the same defect class that moved traits out of the schema column in TASK-2657. The table also gives the lifecycle a home. A reminder is armed, then fired, then acknowledged, and a re-arm returns it to armed — per-reminder state a field definition has nowhere to keep. remind_at is an RFC3339 UTC instant, deliberately not a `date` schema value: those admit both YYYY-MM-DD and full RFC3339 and are compared against the SERVER'S LOCAL calendar day. A fire-at time cannot carry that ambiguity. The remaining timezone question for due_date is filed separately. Firing is one transaction per reminder carrying BOTH the fired_at write and the outbox insert. That pairing is the point: a fired_at committed without its event is a reminder that silently notifies nobody and can never be retried, because the row has left the armed set; an event without fired_at fires every tick forever. The UPDATE's own `fired_at IS NULL` predicate is the arbiter, so two instances ticking at once produce exactly one winner. item.reminder_due is admitted to the closed events/1 set as v1.2, with a new PayloadReminder family and no SSE name. The subject is the REMINDER, not the item: two reminders can be armed on one item, so an item-subject event could not say which fired, and the reminder id is what an acknowledgement addresses. A new payload family rather than reusing the item snapshot for the same reason — a snapshot would validate and still not answer the only question the event exists to answer. No SSE name in v1 because the poll surface is the contract; adding one later is additive, removing one is not. Ack is explicit and nothing else acks. An item reaching a terminal status deliberately does NOT ack: that would make every status write a reminder mutation, and it would silently consume a reminder set to fire after the work was done. * feat(server): reminder surfaces, and one shared overdue rule for all four Second half of IDEA-2641: the HTTP surface, the scheduler tick's wiring, and the fix for the finding that justified the unit — `ready` / `next` did no date handling at all. OVERDUE NOW HAS ONE IMPLEMENTATION. It used to live inline in the dashboard's attention loop, which meant `pad project stale` inherited it (it filters that very list) and the recommendation surface never saw it. So a deadline reached the two surfaces that REPORT on work and never the one an agent PULLS from. overdue.go is now the only place that decides, and all four call it. Two behaviour changes fall out, both deliberate: - An overdue item bypasses the orphan branch's high/critical priority gate. That gate was where a deadline quietly stopped: a low-priority item three weeks late was reported by `stale` and never suggested by `next`. - Overdue sorts above in-progress. The list is capped at three, so a rank below in-progress would not merely order the deadline lower — on any workspace with three things in flight it would keep an overdue item off the surface entirely, which is indistinguishable from not shipping this. The server-local-today comparison is UNCHANGED and known to be wrong for multi-timezone deployments; it is filed as its own item with the cloud case stated. Changing what "overdue" means on every existing instance inside a change about where the rule LIVES is the kind of behaviour change nobody reviews. Fired reminders reach `next` / `ready` two ways, from one filtered list: PendingReminders is the addressable form (it carries the id an ack needs), and a prepended suggestion is the rendered form. They are prepended AFTER the cap rather than entered as ranking candidates — a reminder is not a task competing on priority, and whether it appeared should not depend on how busy the workspace is. Terminal-item reminders are FILTERED from the surface, never acked. Acking on terminal status would couple every status write to reminder state and would consume a reminder armed to fire after the work was done. The row stays exactly as the user left it; the distinction is observable, and asserted. Three guard tests caught this change and each was answered rather than silenced: - The request-body reader guard was right: the handlers now go through decodeJSON, inheriting the NUL refusal and the size cap. - The canonical-events guard was right: item.reminder_due is admitted to the duplicated contract table as SPEC-3 v1.7, with the reminder subject kind and the new payload family. SPEC-3's own text owes the same amendment. - The NUL census asked for a decision on eight new columns. None carries caller text: ids and FKs are server-generated, four are the server clock, and remind_at is now re-parsed and re-formatted in the STORE as well as at the edge — so the stored value is always machine-produced from a parsed time and no caller bytes reach the column. The doc comment that used to say "the caller normalizes" protected nothing. Regenerating the baseline also found that GEN_NUL_BASELINE=1, which the test's own instructions name, was never implemented — the flag did nothing, so the documented path was hand-editing the file. Implemented, so the next reader gets the mechanism the instructions promise. * test(reminders): the lifecycle, the four surfaces, and 22 killed mutants Every test here was designed against a specific mutation and the mutation was RUN. A green suite proves nothing about a suite nobody tried to break, and three of the mutants I first wrote were not experiments at all. Store (10 mutants, all killed): candidate predicate <= flipped to >=; the event emission lifted out of the fire transaction; the fire UPDATE's `fired_at IS NULL` arbiter removed; the RowsAffected check ignored; re-arm clearing fired_at but not acked_at; ack losing `fired_at IS NOT NULL`; the poll surface losing `acked_at IS NULL`; normalizeRemindAt no longer refusing; it dropping .UTC(); GetReminder losing its workspace scope. Surfaces (12, all killed): the priority gate no longer bypassing on overdue; the sort no longer ranking overdue first; attention leaving the shared helper; the reason losing its OVERDUE prefix; the comparison flipped to >; terminal items no longer skipped; terminal reminders no longer filtered; the filter ACKING instead of hiding; reminders appended instead of prepended; the tick running on a far-future clock; ack answering 200 for an unfired reminder; parseRemindAt accepting a bare date. THREE MUTANTS DID NOT COUNT ON THE FIRST PASS and were rewritten. Two failed to compile (`if false` orphaned a variable; deleting a parse orphaned an import) and one had an anchor matching two call sites. A non-compiling mutant emits zero FAIL lines and reads exactly like a surviving one — it invents a hole that is not there — so the harness reports BUILD-FAIL and ANCHOR-BAD as outcomes distinct from SURVIVED. It also restores files from an in-memory copy rather than `git checkout`, which would delete uncommitted work in the tree. ONE MUTANT GENUINELY SURVIVED and the test was at fault, not the mutant: appending rather than prepending reminder suggestions was undetectable because the fixture had a single item, so the reminder sat at index 0 either way. The fixture now fills the three-item cap with in-progress work, where an appended reminder lands fourth and vanishes. Faithful mutant, weak test — checked in that order. The same lesson shapes the four-surface fixture: it is a LOW-priority open orphan, because that is the case the old code handled worst. A high-priority task would have made the ready/next leg pass against the unfixed tree, which is a green that measures nothing. Negative controls throughout: a future deadline is not overdue and does not reach the gate bypass; a tick with nothing due fires nothing; a completed item is neither overdue nor suggested. Without them a helper that reported every date, or a tick that fired everything, would satisfy every positive leg. The lead's pin is asserted in both directions: a fired reminder on a done item is ABSENT from the surface and PRESENT and still unacknowledged in the table. Asserting only the absence would pass against an implementation that consumed the row, which is the behaviour the pin exists to forbid. * feat(mcp): pad_item.remind + ack-reminder, ToolSurfaceVersion 0.28 An agent that can RECEIVE a reminder but not set one has half the primitive. The poll surface is pad_project.next / ready, both long exposed, so reminders already reached agents — what was missing is the other half: deferring a piece of work is exactly the moment an agent knows when it wants to be asked again, and it had no way to say so. Two additive actions, two optional params. Nothing existing moved, so a v0.27 consumer enumerating neither is unaffected — the v0.13 / v0.11 / v0.8 disposition, which likewise wired existing CLI verbs onto the catalog. remind_at REFUSES a bare date rather than reading it as midnight. Worth stating because the `date` schema type accepts YYYY-MM-DD and a caller will reasonably try it here: a bare date names a 24-hour span, and choosing an hour inside it would fire at a time nobody picked. Re-arm and disarm stay CLI-only. Both address a reminder by an id the agent would have to list first, and no listing action exists on this surface — a door with no handle. Adding them later is additive. Five guards had to be taught, and each was answered on its merits rather than excluded: the HTTP parity test (route mappers added, so the actions work on the remote transport rather than being advertised and unrouted), the read-only catalog's cmdhelp fixture and expected cmdPath map, the field- conflict classifier (remind_at / reminder_id are NOT field writers — a reminder is a row in its own table addressed by its own id, so listing them as classified sources would have pointed detectFieldConflicts at something that is not a field source), and the instructions.md / README action tables. That machinery is why the version bump is safe to make now, and it earned its keep on this change: every one of the five failed on the first build after the catalog entry landed. CONVE-23 sweep for prose this falsifies: - SPEC-3 (DOC-2653) amended to v1.7 in the room, recording item.reminder_due with its new subject kind and payload family — the first canonical event with no user mutation behind it, since a scheduler tick produces it. - CLAUDE.md gains the reminder routes, the CLI verbs, and the v0.28 entry. It was also stale at 0.26 with NO v0.27 entry at all: the 0.27 unit swept instructions.md and README.md and missed this file. Both added. - skills/pad/SKILL.md gains the verbs and a routing entry, including the two things an agent will get wrong — the time is an instant, so ask for a time of day rather than picking one, and finishing the item does not acknowledge the reminder. * fix(reminders): codex round 1 — four findings, all real, all with a pin Round 1 found four defects and refuted none of them. Each fix carries a test that fails against the code as it was, and each of those was mutation-checked. **P1 — pending reminders bypassed item-level visibility.** Every other dashboard section reads `allItems`, which the store already scoped to the caller's collections AND their granted item ids. The pending-reminder list is a direct workspace-wide query and inherited none of that, so a guest holding a grant on ONE item could read the refs and titles of every other item in the collection through its reminders — an item-level leak wearing a notification's clothes. Now filtered with the same `isItemVisibleToGuest` call the sibling sections use. The test's two items share a COLLECTION on purpose: a collection-level filter was already applied, so separate collections would have made it pass against the unfixed code. **P1 — soft-deleted items could starve the queue permanently.** Candidate selection ignored `deleted_at`, and `fireOneReminder` rolls back when it finds the item gone — which leaves the reminder ARMED and therefore a candidate again on the next pass. Candidates are ordered oldest-first and bounded by a limit, so enough archived reminders fill every batch and no live reminder ever fires. Silent, too: the tick reports zero fired and looks idle. Excluded in the candidate query rather than skipped downstream, so those rows never occupy a slot; the reminders themselves are kept, so restoring an item restores its reminder with it — asserted, because a fix that reaped them would pass the starvation test alone. **P2 — the pass stopped at the first failing reminder.** The per-reminder transaction exists precisely so one unfireable row cannot hold back the rest, and `return fired, err` made that comment false — with candidates oldest-first, one persistently broken old reminder blocks every newer one forever. Now continues and joins the errors, so a pass that fired seven and failed three reports both halves rather than reading as clean. The loop is split behind an injected seam because a real mid-transaction failure is not reachable from outside: the database refuses the corrupt rows that would cause one (verified — invalid JSON in items.fields is rejected by the schema). **P2 — suggestions dropped the reminder id.** The docs tell an agent to acknowledge what it sees in next/ready, and the payload carried no handle: a stateless poller could read the reminder and had no way to retire it, so it would be shown the same item forever. `DashboardSuggestion` now carries `reminder_id` (omitempty), `pad project next` prints the exact ack command, and the test acks with the id the surface handed out rather than merely checking the field is populated — a wrong-but-present id satisfies equality with itself. Four mutants, four killed; one was rewritten first because its anchor matched two call sites and was therefore not an experiment. * fix(reminders): codex round 2 — four findings, all real **`--rearm` was unusable.** `ExactArgs(1)` forced an item ref that the rearm branch then ignored, so the flag could not be reached without supplying a ref that was silently discarded. Now `MaximumNArgs(1)`, with each mode checked explicitly: a ref is required to arm, and a ref supplied ALONGSIDE `--rearm` is refused rather than ignored — it names an item the reminder may not even belong to, and quietly dropping it is how a user learns nothing about the reminder they just moved. **`unremind --format json` emitted plain text**, breaking the parseable-output contract every sibling command honours. **The MCP `ref` param did not list `remind`.** Agents read that flat description to decide what to send, so an action missing from it is an invalid call waiting to happen. It now also says what `ack-reminder` takes instead, and why: a reminder is addressed by its own id because an item can carry several. **Fractional seconds fired early.** `time.Parse` accepts `09:00:00.900Z` and `Format(RFC3339)` drops the fraction, so it was stored as `09:00:00Z` and fired 900ms BEFORE the moment the caller named — silently, having rewritten their value on the way in. Seconds are genuinely the stored resolution (the column is compared as a string against a whole-second clock, and the tick runs every 30s), so the only question was which way to resolve it, and truncation resolved it the wrong way. `NormalizeInstant` now rounds UP: at most a second of lateness, in exchange for a guarantee that can be stated — a reminder never fires before the instant it was set for. Late is a reminder; early is a wrong answer. Whole seconds round-trip exactly, which is asserted, because an implementation that added a second unconditionally would otherwise pass. Three mutants for this round, three killed (round-up→truncate, round-up→unconditional-add, MaximumNArgs→ExactArgs). Thirty across the unit. Two fixes carry no dedicated test and it is worth being explicit rather than implying coverage: the `--format json` branch on `unremind` is a one-line output change with no server-free way to drive it, and the MCP `ref` description is prose the drift tests do not read — they assert an action is DOCUMENTED, not that a param's sentence lists it. * docs(reminders): the ack id is on the surface an agent polls, not only on the arm response CONVE-23 follow-through on the round-1 fix. Both agent-facing docs told a caller to acknowledge a reminder with the id "returned when you armed it" — true, and useless to the caller that matters: a poller reading next/ready never armed anything. The suggestion now carries reminder_id and `pad project next` prints the exact ack command, so the docs say that instead. The prose was written before the fix existed, which is exactly the case CONVE-23 is about: a change that makes an instruction stale without touching the file the instruction lives in. * test(reminders): bind the tick LOOP to the work, not just the pass (CONVE-19) Every other test in this file calls runReminderTick directly. That vouches for the component and says nothing about whether anything ever calls it — a tick that is never started is indistinguishable, from those tests, from one that is. It is the convention's exact case, and the failure I recorded on my own identity doc three times in one unit: I test the component and not the binding. Driven through the injectable tick channel so the assertion pins a SPECIFIC pass instead of racing a 30-second ticker, and polled to a bounded deadline so a loop that never runs FAILS rather than hanging the suite. Mutant: drop `s.runReminderTick()` from the select and this goes red while every direct-call test stays green. Killed. The idempotence leg exists because a second Start spawning a second loop would leave one running after Stop, making the BUG-842 drain invariant false for this sweeper specifically — the one property a copied lifecycle is most likely to get right by accident and least likely to be checked. The cmd/pad call site (cmd_server.go, alongside StartTokenReaper) stays verified by inspection: a source-scanning guard for it would be an instrument asserting facts about source, which is code with an adversary and not worth it for one line that sits in the middle of five identical neighbours. * fix(reminders): codex round 3 — a deferred reminder fired anyway, and the poll surface was unbounded **A re-arm mid-pass did not stop the fire.** The candidate scan selects an id; before the UPDATE runs, a `--rearm` can move that reminder into the future. Re-arm clears `fired_at`, so a predicate checking only `fired_at IS NULL` still matched — the pass fired a reminder the user had just deferred and emitted its event. The re-arm cannot undo that: it can clear the mark, but the event is already on the outbox and at-least-once means a consumer has seen it. The fire UPDATE now revalidates `remind_at <= nowTS` against the SAME nowTS the candidate scan used. Same-value deliberately: the arbiter and the scan must agree about when this pass is, or a reminder could pass one and fail the other for no reason but clock drift inside a single pass. **The poll surface was unbounded.** Every fired-and-unacknowledged reminder was loaded and turned into a suggestion prepended to a list that is otherwise capped at three, so a workspace with five hundred unacknowledged reminders returned five hundred suggestions — in the dashboard response, the hottest read in the product, growing until somebody acknowledged them. Two bounds, because they are two different guarantees: the query takes a window (default 50, oldest-fired first, so it holds what has waited longest), and the prepended suggestions are capped at 5 so `suggested_next` stays a recommendation rather than a second inbox. The full set stays addressable in `pending_reminders`. Truncation is REPORTED as a boolean, not a count. A count would have to be post-visibility-filter to be true for the caller reading it, and the store cannot compute that — the filter runs per item, above. "There are more than you can see here" is the strongest claim the data supports, so it is the one made. Four mutants; two killed outright, two survived and were run down under CONVE-28: - **Uncapped suggestions survived because the fixture had ONE reminder** — capped and uncapped are the same list at n=1. That is the SECOND time a single-item fixture hid a count-or-order property in this file. Fixture now arms eight; it also asserts all eight remain in `pending_reminders`, so the cap is pinned to the recommendation and not to the data. - **Removing the SQL LIMIT survived, correctly, and the test comment now says so.** The Go slice cap bounds the PAYLOAD; the SQL LIMIT bounds the DATABASE'S work. Only the first is observable at this level — with the LIMIT gone the response is still bounded, while the query silently goes back to materialising every pending row before discarding most of them. That is a memory and I/O property with no assertion available here, so it is stated as a coverage boundary rather than papered over with a green that would not have measured it. * docs(reminders): the fire predicate arbitrates against two actors, not one CONVE-23 inside the file the round-3 fix touched. The comment described the UPDATE as an arbiter for concurrent TICKS, which is what it was written for and is why I did not re-read it when asked whether a user edit could race the pass. It now says what it actually defends against, and names the general shape: an arbiter is only an arbiter with respect to the writers it can see. * fix(reminders): codex round 4 — the round-3 bound recreated the round-1 starvation Round 3 bounded the poll surface. Round 4 caught what that bound did: the query took the first N rows and the dashboard then discarded the ones it could not show — hidden items, unauthorised items, completed items — so N such rows hide a visible reminder behind them indefinitely, with no continuation to reach it. That is the SAME defect I had removed from the fire path one round earlier, reintroduced in the read path within the hour. The general form is worth stating because I clearly did not hold it: **a bounded window is only safe when the discarding happens BEFORE the bound.** Filtering above a limit is a starvation every time, and it does not matter what the filter is for. Two halves, because the two filters are not the same kind of thing: **Visibility is now scoped IN SQL**, using the same collection-id / item-id sets every other dashboard section gets through `allItems` — the same three-way shape as ItemListParams, where holding both collection grants and item grants is an OR. Invisible rows no longer occupy the window at all, which is strictly better than filtering them out afterwards and is what the sibling sections have always done. **Terminality is paged**, because SQL cannot evaluate it — a collection's schema defines which statuses are terminal. The collector refills from the next page when a page comes back short, bounded by a max scan so a workspace full of completed items cannot turn a dashboard read into a table scan. The bound is 10x the window: the common shape fills on the first page, and the pathological shape terminates in a fixed number of indexed reads. Stopping at the scan bound reports truncation, which is honest — there may be more, and we did not look. The empty-scope case is a THIRD state that reads like the second: nil CollectionIDs means unrestricted, a non-nil EMPTY slice means this caller sees no collections. Without an explicit guard they collapse, because the switch matches none of its cases at length zero and adds no clause at all — so "nothing visible" would return the whole workspace. Three mutants, one survived: the empty-scope guard, because no dashboard-level test produces that state (callers that would are refused earlier by workspace access). Faithful mutant, missing test — it now has a direct one, with a sanity leg so a build returning nothing cannot pass it by accident. A guard for a state nothing exercises is exactly the one that rots. * fix(reminders): codex round 5 — the MCP action I shipped did not work over stdio **P1: local stdio MCP `remind` was unusable.** cmdhelp derives positionals by regex from a command's `Use` string, and `<instant>` inside `remind <ref> --remind-at <instant>` matched — it became a second REQUIRED positional, so dispatch failed with `missing required argument "instant"`. The action was advertised on a transport where it could not run. **The MCP catalog's own tests did not catch it, and the reason is the finding.** That suite builds its cmdhelp document BY HAND: I wrote `Args: mkArgs("ref")` in it, so the fixture agreed with what I meant rather than with what the CLI says. Five parity and drift tests passed against a document I authored to match my own intention — the "a test that agrees with whatever the table says is not a test of the table" shape, which the canonical-events test warns about in its own comment two packages away. The new test reads the REAL command tree via cmdhelp.Build, which is the only thing in this repo that can disagree with me about what the CLI declares. **P2: `pad project ready` withheld the ack handle** that `next` prints. Showing a fired reminder on the surface an agent polls while withholding the id it needs to retire it means the same entry comes back on every poll, forever. **P2: suggestions asserted a collection they did not have.** The orphan branch admits ANY collection — its own comment claimed it gated on tasks "mirroring the active-plan branch", and that comment was simply false — while the output hardcoded `Collection: "tasks"` and the reason said "Open task". Pre-existing for high-priority items since BUG-1082; my overdue bypass widened it to any overdue item, which is how it surfaced. Fixed by carrying the item's REAL collection rather than by narrowing the branch: narrowing would silently drop the non-task items this has surfaced for a year, and the defect is the mislabelling, not the inclusion. The false comment is replaced with what the code actually does. The first version of that test used an overdue IDEA and SKIPPED — ideas use `new`, and the branch requires `open` or an active status, so it never became a candidate. A test that cannot fire is a failed reconstruction, not a pass; the fixture is now a bug-like collection whose vocabulary contains `open`, which is the population the defect can actually reach. Three mutants, three killed. Forty-one across the unit. * fix(reminders): codex round 6 — reminders fired from soft-deleted workspaces **P1, and the only defect in this unit whose consequence leaves the process.** Workspace soft-delete deliberately keeps items for the 30-day restore window, so the candidate query's filter on the ITEM's deleted_at found nothing wrong — and the tick kept firing, emitting outbound webhook events for a workspace whose owner had deleted it, possibly while deleting their account. Both queries now join workspaces and require `w.deleted_at IS NULL`. Nothing is destroyed: a restored workspace resumes firing, which the test asserts, because "stops firing" and "is destroyed" are very different answers to someone who restores a workspace and only one of them is right. That test first failed for the WRONG REASON and the fixture was at fault: it counted every outbox row in the workspace, and item creation writes its own, so the assertion was satisfiable by the fixture itself and discriminated nothing. Scoped to the reminder event type. **`Use: "remind <ref>"` declared a requirement the command contradicts.** cmdhelp derives the machine-readable arg spec from that string, and `--rearm` takes no ref — so the published contract said "required" for something optional. The requirement is CONDITIONAL, which cmdhelp cannot express, so the honest declaration is `[ref]` plus the explicit check that names both call shapes. The round-5 test grew a `required` column, which is what makes this observable at all: asserting only the arg NAMES would have passed. **The pad_item tool description omitted both new actions.** The params were declared and the actions dispatched, but the prose an agent reads to decide what a tool can do did not mention them — discoverable only by someone who already knew to look. It now describes both, including the two things an agent gets wrong: remind_at is an instant, and nothing but an explicit ack retires a fired reminder. Three mutants, three killed. Forty-four across the unit. * fix(reminders): codex round 7 — one predicate for the scan and the arbiter Third instance of one class, so this fixes the SHAPE rather than the instance. The class: the candidate scan filters on something the fire transaction does not revalidate, so a change committed between them fires a reminder that no longer qualifies. Round 3 was a re-armed instant. Round 1's soft-deleted item was the same thing caught from the other side. Round 7 is a workspace deleted between the scan and the fire — the round-6 fix added the condition to the SCAN only, and the arbiter went on not knowing about it. Fixing those one at a time is what let the third happen. `reminderFireable` is now a single string that both sites reference: the scan asks it and the fire UPDATE re-asks it, so they cannot disagree, and a fourth condition is one edit in one place rather than two edits someone has to remember are paired. Written as a correlated EXISTS on item_reminders.item_id rather than a JOIN precisely so the identical text is valid in both a SELECT and an UPDATE, and the scan drops its table alias so the two uses are the same characters. What deliberately stays outside it: `fired_at IS NULL` and `remind_at <= ?` live on the reminder row itself, are already spelled identically at both sites, and folding them in would need a parameter order the shared form cannot express. Said in the comment so the omission reads as a decision. Both directions are now tested at the arbiter — a workspace deleted mid-pass and an item deleted mid-pass — because the item case previously relied on the item load coming back nil, and someone simplifying the EXISTS down to the workspace check alone would otherwise still see green. Three mutants, three killed: the arbiter dropping the shared predicate, and the predicate dropping each of its two halves. Forty-seven across the unit. * fix(reminders): codex round 8 — workspace export silently dropped every reminder WorkspaceExport is a hand-maintained field list, so a new table joins it only if someone remembers. Reminders did not: a backup/restore, or a SQLite→Postgres migration via `pad db migrate-to-pg`, dropped every pending reminder with nothing in the destination to show anything had gone. The line that list has always drawn is item-scoped workspace CONTENT (comments, links, versions — exported) versus per-user state (stars, watches — not). A reminder has no user column and hangs off an item, which puts it on the exported side. Stating the rule rather than just adding the field, because the next person adding a table needs to know which side they are on. LIFECYCLE MARKS ARE CARRIED, not reset. A fired-and-unacknowledged reminder is still owed to whoever armed it, so it arrives pending; an armed one whose instant has passed fires once on the destination's first tick, which is what would have happened had the workspace never moved. Re-arming everything on import would invent a schedule the user did not set. NULL rather than empty string for the unset marks — the lifecycle is defined by NULL-ness, and "" would make a never-fired reminder read as fired at "". TestMigratedTablesCoversTheExport caught the second half, which I would have missed: `pad db migrate-to-pg`'s NUL preflight decides what to REFUSE on from MigratedTables, so a table the migration copies and the preflight does not know about is a gap in exactly the guard that exists to prevent one. Added there too, with the reason it can never actually fire — every column is machine-produced, so it is listed for coverage rather than expectation — and the "six tables" prose it falsified is now seven. Two mutants, two killed: export dropping the block, and import discarding the marks. Forty-nine across the unit. * test(reminders): state the fire-path invariant and pin it from the invariant The lead's read on why rounds 4 and 7 were the same class: the fire path had no stated invariant, so each fix defended an instance. This states it, and derives the pin from the paragraph rather than from the bug history. THE INVARIANT: the candidate scan is a hint and may be assumed to prove nothing. Every condition that made a row a candidate is re-asserted inside the transaction that marks it fired, in the same statement that does the marking, so checking and writing are one atomic act. Worded as "the scan proves nothing" rather than as a list on purpose — a list invites the next person to add a condition to the scan and stop, which is exactly what happened four times here. TestFirePathInvariant is the pin: one table, one row per scan-side condition, each invalidating that condition in the window between the scan and the fire and asserting the same three things — nothing fires, no event leaves, the reminder is not consumed. The earlier per-defect tests are folded in as rows; they said the same thing one instance at a time, which is how four of these shipped. Adding a fifth condition to the scan without a row here should feel like an omission. It carries a positive control, because four cases that all assert nothing happens would pass against a build that never fires at all. The matrix immediately falsified a claim in the paragraph I had just written. I wrote that the item load inside the transaction is "for the payload, not for the check"; removing the item half of reminderFireable alone changes no observable behaviour, because the load then returns nil and the deferred rollback undoes the write. Item liveness is defended TWICE and a single-mutant experiment cannot say which guard is carrying it — removing both is what kills the test. Both are kept, the predicate is named as primary (the row never matches, so no write happens at all), and the asymmetry is stated: workspace liveness has no second line, which is why dropping ITS half does fail the pin. Six mutants: five singles plus the pair. Five killed alone; the item single survives by design and is documented as such rather than left as an unexplained green. Fifty-five across the unit. * fix(reminders): codex round 9 — one legacy row could hide every reminder **P1: items.item_number is NULLABLE and I scanned it into an int.** Migration 006 added the column to existing rows, so a pre-numbering item still carries NULL — and scanning NULL into an int fails the Scan, which fails the QUERY, which degrades the whole pending-reminder section. One old row, and the feature is dark for everyone in that workspace. ListWatchesForUser, which this query was modelled on, uses sql.NullInt64 for exactly this column. I copied its shape and dropped the part that handles the column's actual nullability — the same way of being wrong as the round-5 cmdhelp fixture: borrowing a form without borrowing what it knows. The legacy row now carries no ref rather than a fabricated "PREFIX-0", which would name a different item. **P1: export shipped reminders that import could only discard.** The items section filters on deleted_at IS NULL, so a soft-deleted item is not in the bundle and its reminder can never be reunited with it. My comment claimed the item_links rationale — round-trip the raw graph so a restore reunites them — which is true for links and false here, because links keep soft-deleted endpoints in the bundle and items do not. A link is a row ABOUT two items; a reminder whose item is absent is a dangling schedule. **P2: import wrote remind_at raw.** Import is a writer, and a bundle is not necessarily one this server produced — hand-edited, or from another instance. A local offset or a bare date would land in the one column every comparison downstream treats as a UTC instant, firing early, late, or never. It now normalizes like every other door. An unparseable value is SKIPPED with a warning rather than failing the restore, matching the lenient import-side precedent already in this file, and the raw value's LENGTH is logged rather than its content. Three mutants, three killed; two needed rewriting because the single-line form did not compile — reverting the nullable scan also requires reverting the render, and dropping the normalization orphans a variable. PROCESS FAULT, recorded because it makes this round's findings weaker than they look: I edited the tree while this review was reading it — committed the invariant work and ran five mutation experiments, which write and restore source, over the same files. A review binds to the tree it read and I moved it underneath. Every finding above was re-verified against the current tree before being acted on, and the next round runs with no concurrent edits. * fix(reminders): codex round 10 — one orphaned item aborted a whole restore An ORPHANED item — one whose collection is missing from the bundle — still gets an itemMap entry. It has to: the entry is written before the skip because parent resolution inside the same loop reads the map for items it has not reached yet. So `itemMap[x] != ""` is satisfied by an id that names no row, and inserting a foreign key to it fails (SQLite enforces FKs here via the DSN's `_pragma=foreign_keys(on)`; Postgres always does). The pre-existing mapping is the sharp edge. The aggravating half was mine: this loop treated a failed reminder insert as FATAL, where item_links and item_versions both skip, so one orphaned item carrying a reminder rolled back an entire 900-item workspace restore. A reminder is the least critical thing in a bundle and it had the strictest failure handling in the file. Both halves fixed: the loop gates on items that actually landed, and a failed insert warns and skips like its siblings. TWO GUARDS THAT ONLY DIE TOGETHER, and this is measured rather than assumed. Reverting either alone leaves the test green — with the map gate restored the skip survives the FK failure, and with the fatal return restored the gate means the insert never fails. Removing both is what fails it. They are kept as a pair because they defend the same failure at different depths (prevent the bad write / survive a bad write arriving some other way), and the pair is recorded in the code so a future reader does not delete one as dead after watching its mutant survive. Second time this shape appeared today; the first was item liveness on the fire path. The bundle in the test is hand-built, because ExportWorkspace cannot produce an orphan — which is the reason it needed a test. That shape only arrives from a hand-edited or foreign bundle, and surviving those is what import is for. Three mutants: two singles that survive by design, plus the pair that kills. Sixty-one across the unit. * fix(reminders): codex round 11 — four contract slips, one of them another unit's **suggested_next returned up to eight entries against a cap of three.** Round 3 prepended reminders PAST the list's own cap, reasoning they should not compete for slots. Every consumer — the web dashboard, `pad project next`, `pad project ready` — is written for three. Worse, it silently falsified a decision recorded elsewhere: BootstrapDashboard deliberately has no suggested_next_overflow_count BECAUSE this list is capped at three upstream, and its comment names raising that cap as the moment to add one. My change made another unit's reasoning wrong in a file I never opened. The combined list is now trimmed back to three, reminders still leading — a reminder can push a task suggestion out, which is the right way round, and the full set stays addressable in pending_reminders. My first version of that trim used `limit`, which is REASSIGNED above to len(candidates) — so on a workspace whose only entries are reminders it would have truncated to zero, killing precisely the case the surface exists for. Caught by reading the surrounding lines before running anything; it has its own test now. **pending_reminders was uncapped in the bootstrap projection.** BootstrapDashboard embeds *DashboardResponse, so every new field joins the boot payload automatically — here, a window of up to 50, which is the budget PLAN-1410 spent a unit trimming. Capped at 5 with an overflow count, under its own constant rather than borrowing bootstrapAttentionCap: they answer different questions and a future change to one must not silently move the other. **Truncation was reported from the wrong question.** The collector used the store's `more` flag, which answers "is there another PAGE", not "did I read all of THIS one" — so a window filling part way through the final page reported that the caller had seen everything while unread rows sat behind the fill point. The paging bounds are now injectable so the case is testable at all: building it with a window of 50 needs ~75 rows in a specific pattern, with a window of 3 it is four. **Import accepted acked-without-fired**, which is not one of the lifecycle's three states. Such a row fires, is excluded from the pending surface because it is already acked, and can never be acknowledged because AckReminder requires acked_at IS NULL — an event emitted into permanent invisibility. The acknowledgement is dropped and the schedule kept, since an ack of something that never fired means nothing. Five mutants, five killed (one rewritten — removing the flag orphans a variable). Sixty-six across the unit. * fix(reminders): codex round 12 — a read is not a hold; scope the arm; ack from the ack Four P2s from round 12 (two independent runs, both landing on the same line of the fire path), each closed at the layer where it lives: - fireOneReminder pins the item and workspace rows FOR NO KEY UPDATE on Postgres before the arbiter UPDATE. reminderFireable re-asserted liveness at the predicate's instant and nothing held it to the commit instant; under READ COMMITTED an archival could commit in between and the event left the process about a deleted resource. Same idiom and same lock strength as CreateAttachmentForLiveItem; SQLite is excluded by its BEGIN IMMEDIATE, not skipped for convenience. Two PG-only pins verify "blocked" in pg_stat_activity, not by elapsed time; the pin-removed mutant fails both. - CreateReminder asserts "live item of THIS workspace" in the INSERT's own SELECT and returns ErrReminderItemGone otherwise. The table had an FK and no same-workspace constraint; a mismatched pair fed another workspace's title to this one's dashboard and webhooks. Handler maps it to 404. - AckReminder matches every fired row (COALESCE keeps the first ack, updated_at moves only when acked_at does), so a no-match means exactly "not fired at the instant of the ack". The handler no longer decides 409-vs-200 from the row it read before the UPDATE. - The invariant paragraph gains its missing sentence: "at that instant" means the commit instant, and the pin is what makes the predicate's instant and the commit instant the same one. Round-12 caveat carried: both runs were static reads (sandbox blocked Go's build cache), so "four" is a floor, not a measurement. Refs IDEA-2641 * fix(reminders): codex round 13 — a reminder's workspace must agree with its item's, at every read Every reader scoped by r.workspace_id and then joined the item without asserting the two agree. No door writes a disagreeing row today (CreateReminder derives the pair from the item; import maps within the workspace), and the table has nothing that forbids one — so a hand-edited bundle, a future move door, or a direct write would carry one workspace's item into another's dashboard, export, and webhooks. The identity goes into reminderFireable (scan + arbiter), the Postgres row pin, ListPendingReminders and the export query. One test writes the row raw — the only way one can exist — and asserts it is inert at each site; the predicate-removed mutant scans and fires it. Refs IDEA-2641 * fix(reminders): codex round 14 — the by-id and by-item reads assert the same identity as every other read GetReminder scoped by the row's own workspace_id and ListRemindersForItem by item_id alone, so a row whose two columns disagree — the class rounds 12 and 13 closed at the scan, the arbiter, the pin, the pending surface and the export — was still readable through the two reads that reach a single row. reminderOwned is that identity on its own, without the liveness half those two reads must not have (a fired reminder on an archived item is history worth showing). The write paths reach a row only through GetReminder, so scoping it scopes them; a row no door can write needs no door to delete it. ListRemindersForItem now takes the workspace its caller already resolved the item in. The raw-row test asserts both reads refuse the row from both sides; the reminderOwned-removed mutant surfaces it through GetReminder. Refs IDEA-2641 * fix(reminders): codex round 16 — an archived item's reminders are readable, and its verbs say "archived" The doors resolved the item live. Listing an archived item's reminders answered 409 from a GET, and ack/re-arm/delete answered a bare 404 for a reminder that exists on an item that exists — while the store, since round 14, deliberately keeps that history readable. The API already has a posture for archived items: GET reads them, mutations answer 409 "archived … restore it before editing" (writeItemResolveError). The list now follows handleGetItem; the lifecycle verbs load the item include-deleted, run the visibility check first, and then answer the same 409 every other item mutation does. One test walks archive → list 200 / ack 409 / arm 409 → restore → ack 200 on the same rows. Refs IDEA-2641 * fix(reminders): codex round 17 — one suggestion per item, the archived 409 by slug, and the door courtesy named Three findings on the server pass. (1) An item that was both a fired reminder and an ordinary candidate appeared in suggested_next twice; the ordinary entry is dropped, the reminder entry (which carries the ack id) stays, and two reminders on one item remain two entries. (2) Round 16's 409 for an archived item's reminder was written by re-resolving item.Ref, which is derived and empty for a legacy item with no item_number — so the class most likely to be legacy fell through to a bare 404. The slug is handed over instead. (3) The archived check in resolveReminderForWrite is check-then-write, and an archive landing in between lets the verb through: accepted and documented — it is the posture of every item mutation here (UpdateItem's UPDATE has no liveness clause), the outcome is benign, and putting liveness in AckReminder's WHERE would re-create the no-match ambiguity round 12 removed. Refs IDEA-2641 |
||
|
xarmian
|
dc3fc2d50e |
feat(server,cli): name undeclared field keys on the write response (BUG-2850)
Undeclared keys are ACCEPTED — the census found 168 live values under 14 such
keys, and refusing them would break read-modify-write on items nobody edited
wrongly. But once stored, a typo and a deliberate extra field are
indistinguishable, so the write now says which keys it did not recognize.
- models.Item gains `Warnings *ItemWriteWarnings` with `undeclared_fields`,
omitempty and additive. NEW API SURFACE: item write responses carried no
warnings element before. Wrapping the response as {item, warnings} was the
alternative and would have broken every existing parser; a clean write is
byte-identical to before.
- items.UndeclaredFieldKeys consults models.IsReservedItemField rather than
re-listing the reserved set — that set exists so callers ask, and its doc
comment records what re-listing cost last time. So a write carrying
implementation_notes or github_pr reports nothing.
- fields_patch reports only the PATCHED keys. A stray key already on the item
is not something this write introduced, and naming it on every touch would
train the reader to ignore the field.
- The CLI prints one line to STDERR. Never stdout: `--format json` output is
piped into scripts, and a warning there would corrupt the JSON they parse.
- CLAUDE.md documents the element as new surface.
Controls: never attaching the warnings fails the pin; reverting the HTTP
mapper's native overlay fails the remote-door type test; dropping the
reserved-key exclusion fails its own test.
Two coverage gaps the controls FOUND rather than confirmed, both now closed:
the remote door's native overlay was covered by no MCP test at all (a revert
left the package green), and the reserved-key exclusion had no test either.
Both were written after the control survived, which is the only reason they
exist.
Gates: gofmt clean, go vet clean, go test ./... 29 packages ok.
Claude-Session: https://claude.ai/code/session_011Q4b1iHtJtSyMs7BA2ySxo
|
||
|
xarmian
|
0d0f1c9125 |
fix(items): require and bound item titles at every write door (BUG-2833, BUG-2831)
`PATCH {"title": ""}` was accepted and applied while `POST` refused the same input
with 400 "Title is required": the guard was an inline literal inside
handleCreateItem, so the sibling handler on the same field never had it. Item
titles were also unbounded, and the slug derives from the title with no
truncation, so the same input was accepted on SQLite and refused by Postgres at
the UNIQUE(workspace_id, slug) btree with an unmapped SQLSTATE 54000 — a latent
`pad db migrate` failure as well as a create-path one.
One models.NormalizeItemTitle / models.ValidateItemTitle pair now backs every
door, enforced authoritatively in store.CreateItem and store.UpdateItem so a
future door inherits the rule rather than having to repeat it. The handlers keep
a pre-lock copy that REFUSES ONLY: it may answer 400 early and must not alter
the input, because its view of the row predates the write lock.
- trim: whitespace-only titles are refused, widening the create door. Artifact
import already trimmed while create tested == "" exactly, and its comment
claimed to mirror the gate it was stricter than.
- bound: 255 runes, matching MaxDocumentTitleRunes but justified for items —
slugify emits only [a-z0-9-] at one byte per rune and truncates nothing, so
255 runes bounds the slug well under the btree index-tuple cap. That cap is
2704 bytes in practice, not the 8191 the filing quoted; both figures and the
readings behind them are in the constant's comment.
- non-retroactive: a title identical to the stored one is not a rename, is not
validated, and is dropped rather than re-applied — so rows predating the bound
stay editable and a no-op echo cannot move an item's slug.
- import coerces rather than refuses (empty -> "Untitled", over-long ->
truncated, both logged, colliding truncations resolved), matching
coerceJSONForImport's recorded disposition three lines away. Refusing would
break restoring archives of data this product already accepted.
- cross-workspace copy propagates a legacy source title, by ruling. It takes no
title from the caller, so it cannot mint one.
The guarantee that holds across every path is narrower than "every stored title
satisfies the bound", and the comments say so: no CALLER-SUPPLIED title is
stored without being validated.
Seven codex rounds, 23 findings, ending CLEAN. Two of the findings were defects
introduced by earlier fixes in this same unit — an empty-title hole opened
through the legacy-protection clause, and a handler-side decision that dropped a
concurrent rename — both recorded on BUG-2833's trail. 38 mutants; every
behavioural fix has a mutant that is the defect at its site, killed by a named
test.
Prose sweep per CONVE-23: three comments asserting item titles are unbounded,
and a cost model resting on a ~2 MiB single-request title, corrected in place —
the guards they document still hold, because the bound is non-retroactive and
the cascade charges STORED titles.
Filed rather than bundled: BUG-2836, BUG-2839, BUG-2840, BUG-2842.
Closes BUG-2833, BUG-2831.
Claude-Session: https://claude.ai/code/session_01XLtX4dbjBpApbAv3SuBcTm
|
||
|
xarmian
|
b2c303c4bb |
docs(links,store,models): cite markdown.ts by symbol, and check it (BUG-2832)
Go comments describe the web renderer constantly and cite it by LINE NUMBER. Nothing verifies those citations — they cross a language boundary, so no compiler, test or linter has ever checked one — and they had drifted onto unrelated code. This converts all 32 to `markdown.ts::symbolName` form and adds the check that makes the conversion worth something. Scope note, because this is wider than the rider it was dispatched as. The BUG-2834 commit added the pattern constant near the top of markdown.ts, shifting the file by +45 lines and invalidating EVERY line citation into it — including the three BUG-2832 had confirmed were still accurate. Leaving 13 knowingly-wrong citations because they sit outside the files this unit otherwise touched is not the neutral option when this branch is what broke them. Happy to split this commit back out if the lead would rather hold the rider to its stated bound. While converting, five of the filing's six "suspect, not established" citations were settled by reading the shifted positions: :307, :478-481, :485, :513 and :516 point at a @param doc line, unescapeDocLinks, REF_PATTERN, the tail of parseCrossWorkspaceBody, and findItemByRef respectively. All substantively stale, not merely off-by-lines. That answers the filing's open question. Two guard tests, per the filing's own proposed fix shape: TestMarkdownCitationsNameLiveSymbols verifies every cited symbol is really declared in markdown.ts. This is the check a line number could never have. TestMarkdownCitationsAreNotLineNumbers bans the line-number form, so the fix cannot erode the next time someone reads a number off their editor gutter. The first version of the symbol check FAILED its negative control and that is the part worth reading. It asked strings.Contains(ts, "function "+sym) — a PREFIX match. Renaming resolveWikiBody to resolveWikiBodyRENAMED leaves "function resolveWikiBody" a substring of the renamed declaration, so the guard stayed green through precisely the rename it exists to catch. It passed its first real run and would have shipped as coverage. Fixed by requiring the following character to be one that cannot continue a JS identifier; the control now fires and names the symbol. Both guards are non-vacuity-asserted: the sweep fails if it finds fewer than 50 Go files, and the symbol check fails if it finds no citations at all. Currently verifying 7 distinct symbols across 29 citation sites. The line-number guard earned its keep before being committed — it caught three citations silently reverted when a file was restored from a snapshot taken before the conversion. |
||
|
xarmian
|
427540706c |
fix(documents): bound the rename cascade at the title and at the projected total (BUG-2798, BUG-2796) (#1218)
* fix(documents): bound the rename cascade at the title and at the projected total (BUG-2798, BUG-2796) A document rename rewrites [[oldTitle]] into every linking document. Neither factor of the output size was bounded: titles had no length validation, and the cascade holds every rewritten body in memory before writing any of them. One rename could project 10 GB from a 500 KB input -- 20,000x, measured -- and OOM while holding the workspace rename lock. Two walls, per Dave's day-63 ruling. 1. Title length, bounded at write time (models.MaxDocumentTitleRunes = 255). Runes, not bytes: "255 characters" is what a user and a UI counter mean. Existing over-limit titles stay valid until their next rename -- no retro-breakage of stored data. 2. The cascade's projected TOTAL, bounded at 16 MiB (store.MaxRenameCascadeProjectedBytes), accumulated across the linking set and refused before the first rewrite is built. The total is the right quantity and a per-document cap would not have been. Measured, with the title bound already in place: one linker holding the largest body a 2 MiB request can carry projects 108,632,370 bytes -- 51.8x -- and the aggregate is linear in the number of linkers (108.6 / 217.3 / 434.5 MB at k = 1/2/4, allocation tracking output at ~1.02x). A per-document cap of C still admits k * C, which is the same unbounded shape one level up. The 16 MiB figure has a receipt in the constant's doc comment: it sits above the absolute ceiling of any cascade this development instance could produce (its entire wiki-linking corpus is 10,077,476 bytes) and 6.5x below the single-document attack. The refusal is permanent-shaped and deliberately NOT in ErrLinkCascadeContention's family: 413 with the projection in the message and no Retry-After. Contention means "someone got there first, try again"; this means "this rename cannot be performed as asked". Answering it from the retryable family would tell a client to retry forever. BUG-2796 folds in at the same validation point, as ruled -- a title containing wiki-link syntax is emitted raw by links.ReplaceTitle, so renaming to `A]] [[A` produced two broken links and reported success. The rule is derived from the two mechanisms that consume a stored bracket (the grammar at markdown.ts:327 and the unescaper at markdown.ts:753) rather than from a character blacklist: the first version of this fix banned `]`, `\` and `|` because all three "look like wiki-link syntax", and the round-trip test refuted two thirds of that. `|` in particular is a title shape resolveWikiBody contains a dedicated branch to support, and `[` passes the grammar untouched. Doors enumerated rather than assumed (CONVE-24): store.CreateDocument and UpdateDocument have exactly two callers between them, both HTTP handlers. No CLI, import, or seed path writes a document title. Update previously validated doc_type and status and NOT title -- the one field that drives the cascade -- so the handler tests drive real requests through both doors (CONVE-19). BUG-2798, BUG-2796 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): count retained bytes, bound the retry path, escape the cascade's LIKE pattern (BUG-2798) Codex round 1 on #1218. Three findings, all real, all fixed here. 1. The guard bounded projected OUTPUT, which bounds nothing when the new title is SHORTER than the old one. Renaming a 255-character title to a one-character title makes each 2 MiB linker project ~40 KiB while the cascade still retains its 2 MiB read for the compare-and-set, so hundreds of linkers exhaust memory while the counter reports well under the cap. The counter now sums RETAINED bytes — read plus written, both alive at once — so the cap is a statement about resident memory rather than about output. MaxRenameCascadeProjectedBytes becomes MaxRenameCascadeRetainedBytes and moves 16 -> 32 MiB, because the legitimate ceiling it clears doubles under the new metric (that instance's whole wiki-linking corpus retains ~20,154,952 bytes); the single-document attack retains 110,729,522, so it is still refused by 3.3x. 2. The compare-and-set's retry path bypassed the guard entirely. On contention it re-reads the linker and calls ReplaceTitle on whatever the winner wrote — a NEW input, bounded by nothing the scan had checked — so a content edit landing inside the cascade's window could grow a linker from harmless to enormous and walk the rename back into the amplification it would have been refused for. Each document's compare-and-set now carries the cap less what the other linkers hold, and re-checks the grown body against it. 3. The cascade's `content LIKE ?` search term went in unescaped, so a document TITLE decided how the pattern was read. `\` is the default LIKE escape character on Postgres and NOT on SQLite, so `[[Alpha\Beta]]` was searched for as itself on one dialect and as `[[AlphaBeta]]` on the other: linkers not found, cascade rewrites nothing, rename reports success, every link left stale. Silent and dialect-dependent. Codex named the backslash; `%` and `_` are the rest of the class (CONVE-18) — wildcards on both dialects, so a title carrying them selects documents that do not link it. An explicit `ESCAPE '\'` clause plus escapeLikePattern makes both dialects agree, rather than leaving SQLite correct by accident. Finding 3 also constrains finding 3 of the ORIGINAL fix: models' validator allows a lone backslash in a title on the grounds that both renderers handle it, which was true of rendering and false of cascading. That comment now records the dependency — allowing it is only correct while the cascade's pattern stays escaped. Tests, four new, each mutation-verified against the code it guards: - CountsRetainedBytesNotJustOutput — the shrinking rename. Asserts as a PRECONDITION that the projected-output total stays under the cap, so the test cannot pass for the old reason. - RetryRecheckesTheBudgetAgainstTheGrownBody — drives the real race through the afterLinkCascadeRead seam. POSTGRES ONLY and skipped loudly elsewhere: SQLite's BEGIN IMMEDIATE closes the window structurally, so a green run there would be a property of the DSN. - FindsLinkersWhoseTitleContainsABackslash — Postgres only, same reasoning inverted: SQLite is the dialect that was accidentally right. - DoesNotSpendTheBudgetOnDocumentsThatDoNotLinkTheTitle — `%` and `_`. Its first version asserted the decoy's content was untouched and passed against the unescaped pattern, because over-matched rows rewrite to themselves. The observable harm is that they spend the caller's budget, so that is what it now asserts. Mutation matrix for this round: output-only counter -> only the shrinking test fails; retry check removed -> only the retry test fails (PG); LIKE unescaped -> the budget legs fail on SQLite and the backslash test fails on PG. Gates: `go test ./...` under Postgres 17 EXIT=0; SQLite packages EXIT=0; gofmt clean; `make lint` 0 issues. BUG-2798, BUG-2796 Claude-Session: https://claude.ai/code/session_011T365kP1N9V88y15HxL4YN * fix(documents): tighten the retry budget, stop charging no-op rewrites, order the typed check first (BUG-2798) Codex round 3 on #1218, an edge-case angle over the new arithmetic and control flow. Three findings fixed, one declined. 1. The retry budget credited back this document's own share, on the reasoning that the retry replaces it. It does not: the original read and rewritten bodies stay reachable through `updates` while the write loop runs, so the re-read and its rewrite are allocated ON TOP of them. The bound could be exceeded by up to one document's share while the arithmetic still reported it satisfied. The budget is now the genuine headroom, `cap - retained`. 2. A concurrent edit that REMOVES the link left a body with no occurrences, which cascadeRetainedBytes still charged twice — once for the read and once for a rewritten copy that does not exist, because strings.Replace returns its input unchanged when there is nothing to replace. That could refuse an otherwise valid rename for memory the cascade never allocates. 3. The handler classified this error by PROSE before testing it by identity. The UNIQUE-constraint arm matches a substring, and the refusal error embeds the caller's title verbatim, so renaming a document to a title containing the words "UNIQUE constraint" came back as a 409 name collision — advice to pick a different name, for a rename that was refused for size and would fail identically under any name. Typed sentinel now tested first. DECLINED: unchecked int64 arithmetic in the projection. The multiplicands are derived from the length of a string already resident in memory, so overflowing int64 needs a single document body of roughly nine exabytes; and the accumulator returns as soon as it passes the cap, so it cannot run away either. Saturating arithmetic here would be guarding a state the machine cannot reach. Tests, three new, each mutation-verified: - RetryBudgetExcludesThisDocumentsOwnStrings — deliberately separate from the existing retry test, because that one catches the check being ABSENT and this one catches it being too GENEROUS. The grown body is sized to fall BETWEEN the two budgets; a body far over the cap cannot tell them apart. - ConcurrentEditThatRemovesTheLinkDoesNotRefuseTheRename — its first version sized the link-free body against the CAP rather than against the retry's real headroom, so the refusal it caught was correct behaviour and the test was wrong, not the code. Re-sized against the headroom: fits when charged once, does not when charged twice. - IsNotMisreportedAsATitleCollision — at the handler, since the defect is entirely in its classification order. Mutation matrix for this round: credit the share back -> only the tight-budget test fails; charge the no-op body twice -> only the link-removed test fails; order the substring arm first -> only the misclassification test fails. Gates: `go test ./...` under Postgres 17 EXIT=0; touched packages re-run after the lint fix EXIT=0; gofmt clean; `make lint` 0 issues. CI green on |
||
|
xarmian
|
effea01666 |
fix(server): diff the activity change list against the store's pre-image (BUG-2776) (#1206)
handleUpdateItem built the activity's human-readable change list by diffing the item it read at the TOP of the request — before the permission checks, before the store's locks — against the row the store wrote. Anything a concurrent writer committed inside that window appeared in the difference and was stamped, with this request's actor and agent name, onto whoever sent the PATCH. Agent A sets status, agent B sets priority, and B's entry reads "status: open → done; priority: low → high" over B's name. Unlike BUG-2770, where a change went MISSING, here the timeline gains a confident false statement about who did what — and BUG-2770's debounce then merges that statement forward into the coalesced row, so it outlives the request that invented it. The fix is a carrier, not new machinery. The store has re-read the row under its own locks since TASK-2533, unconditionally, and diffs THAT snapshot for its status and assignment signals; the handler simply had no way to reach it. models.Item.PreUpdate hands the same snapshot back on the returned item — `json:"-"`, transient, populated only by the update path, following the LastMutation precedent. The title, role and assignment arms move onto it too, which makes the whole list committed-vs-committed: what this transaction wrote over, versus what it wrote. A missing pre-image DROPS the change list rather than falling back to the handler's stale read. The fallback is the defect wearing a warning: an entry that says nothing is recoverable, one that names the wrong author is not. The activity row is still written; a slog.Warn names the invariant. Also fixed, same lines: the title arm only recorded a rename when the field diff had produced nothing, so a PATCH that renamed AND edited a field silently dropped the rename. Two test seams are added (Server.afterItemPreRead, Store.afterItemPreLockRead), both nil in production and both documenting their reentrancy requirement. The second exists because of a mutation that SURVIVED the first matrix: with the rival's write landing before the store call, a pre-image taken from the store's pre-lock read is indistinguishable from one taken under the lock — the instrument could not see the difference the fix is about. That mutation now dies. Seven mutations aimed, six die; the seventh (aliasing the pre-image instead of copying it) survives by design and says so in the code. The Postgres leg earned its keep again: two store assertions compared `fields` blobs byte-wise, which passes on SQLite (TEXT, exact bytes) and fails on Postgres (JSONB, re-serialised) while proving nothing either way about which snapshot the blob came from. They compare by value now. |
||
|
xarmian
|
11f67b0a98 |
fix: timeline can answer has_more=true with zero entries, and the client cannot page past it (BUG-2765) (#1202)
* fix(server): timeline returns the cursor for its next page (BUG-2765) The timeline over-fetches 3x per source and drops rows that cannot render (read/searched actions, empty-metadata updates, activities a version or a comment already stands for, collapsed autosave bursts), so a page can carry fewer entries than the rows it consumed — or none, while has_more is true. The client derived its cursor from the last RENDERED entry, which fails in two ways. With no entries it cannot form a cursor at all, so the first page is a dead end. With a fully-dropped window LATER in the history it re-sends the same cursor forever: nothing is appended, the oldest entry does not move, and paging is wedged at that position permanently. The filing named the first; the second is the one that bites an ordinary item, since a run of read activities anywhere in its history is enough. Both are the same root cause — the response says WHETHER to continue and not WHERE — so the server now returns next_before / next_before_id whenever has_more is true: - page truncated: the last entry KEPT, because the ones cut off must be re-fetched. Unchanged from what the client derived. - window exhausted: the NEWEST tail among the sources that filled their window. A short source has nothing older to come back for and must not drag the cursor forward; resuming at the oldest tail instead would step over a newer source's unexamined rows, and repeats are absorbed by the client's dedup while gaps are not recoverable. Progress is guaranteed because every candidate is a row this page fetched and the store's cursor predicate is strict. Tests: an all-dropped window returns a cursor that reaches the history behind it; paging across a dropped MIDDLE stretch terminates and yields each renderable entry exactly once; and a control leg pins that an untruncated page still resumes at its last rendered entry — without it, "always resume from the oldest row touched" passes while silently skipping what truncation cut. The two-full-source selection rule is pinned as a unit test, because a full source whose rows RENDER puts 3x limit entries on the page and takes the truncation branch instead, so the handler cannot cheaply reach it. Mutation matrix, each independently detected: cursor from rendered entries; oldest tail instead of newest; the full-window flag ignored; no cursor at all. Client half follows in the next commit. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(web): page the timeline with the server's cursor (BUG-2765) Client half. The component derived its next-page cursor from the last RENDERED entry, so a page the server had emptied by dropping rows either gave it nothing to page from (the first page, where loadMore returned early on entries.length === 0) or gave it the SAME cursor it already held (a later page, where nothing was appended and the oldest entry did not move). The second case is a permanent wedge with a live button and a running spinner. It now pages with next_before / next_before_id when the server sends them, and falls back to the last entry otherwise — which is exactly the old behaviour, including its wedge, and is there only for a server that predates the field. One press walks at most MAX_EMPTY_HOPS pages while every row keeps dropping. That bound is UX, not correctness, and its comment says so: a single hop is already correct now that the cursor advances; the loop exists so a user crossing a long run of read activity sees entries appear rather than a spinner and nothing, and it is small so a pathological item cannot turn one click into an unbounded request fan. Tests assert the cursors the component ASKS FOR, not only what it displays — a component that shows the right thing by re-fetching page one forever is the bug. Mutation matrix: ignoring the server cursor fails three of four legs (the fallback leg survives, correctly, since that is the path it pins); a single hop fails the advance leg; an unbounded hop count fails the bound leg. vitest 109 files / 1862 tests pass; vite build clean; svelte-check 0 errors (6 warnings, all pre-existing and in other files). Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix: cursor must clear BOTH bounds, and a later page must merge not append (codex round 1) Two findings, both real, both consequences of the cursor itself. P1 — truncation and an exhausted window are INDEPENDENT bounds, and the second is not implied by the first. A source whose rows all drop contributes nothing to the page, so the truncation cursor can sit older than that source's tail, and every unexamined row between the two falls in a gap neither page fetches. The cursor is now the NEWEST candidate across both reasons. Its regression puts one renderable activity in exactly that gap: two comments forcing truncation at limit=1, three read rows filling the activity window above them, and the row at risk in between. Run against the previous commit it comes back 0 times — the row is not late, it is gone. P2 — a later page can legitimately carry entries NEWER than the oldest one already shown, because the cursor deliberately re-covers ground when one source's window ran out before another's. Concatenating printed those below older entries. The client merges by (created_at desc, id desc) now, comparing INSTANTS rather than strings: precision is not uniform — the store writes whole seconds but a structured note can carry a sub-second timestamp — and lexicographically "…:05.123Z" sorts before "…:05Z". Mutation checks: truncation ignoring the exhausted candidate fails the new gap test; concatenating instead of merging fails the new order test and nothing else. internal/server suite green; timeline vitest 9 files / 83 tests green. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(web): stop the hop loop when the cursor does not advance (codex round 2) Against a server that predates next_before, the fallback re-derives the same last entry on every hop, so a single Load More click fired five identical requests where the pre-fix component fired one. The client cannot give an old server a cursor it does not have — that wedge is the old behaviour and stays — but amplifying it was new, and mine. A cursor that did not move cannot make progress, so the loop stops on it. Its test asserts the request COUNT, which is the only thing that distinguishes this from the behaviour it replaces: the entries rendered are identical either way. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(web): order the SSE-refresh merge too, and share one comparator (codex round 3) The refresh prepended its genuinely-new entries on the assumption that a fresh first page is always newer than everything on screen. Normally it is — but a structured note or decision carries a hand-written created_at and can arrive backdated, and the assumption was never stated, only relied on. Both merge points now go through one byNewestFirst, which is the server's own ordering and compares instants rather than strings. Two of codex's three round-3 findings are not folded in: - The refresh not adopting the response's has_more / cursor is DECLINED, not missed. The refresh re-fetches the NEWEST window, which says nothing about where the reader's paging frontier is; the stored cursor stays valid because the refresh consumes no older rows, and adopting the fresh page's has_more after the reader has paged deeper would point the cursor back at history they already hold. One wasted request, absorbed by the no-advance stop, in exchange for a correctness claim I cannot make. - firstPageIds treating any entry missing from a refreshed first page as deleted is real, pre-existing, and a semantics call about what counts as a deletion rather than a patch: filed as BUG-2773. Refs: BUG-2765, BUG-2773 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * test(server): derive the expected entry set instead of naming two ids (codex round 4) The test claimed every renderable entry came back exactly once and checked the two ids it had seeded. The item's own `created` activity could have vanished or repeated underneath that claim — the same partial-verification shape as asserting one direction and writing the symmetric conclusion. The expectation now comes from the store: every activity on the item minus the kinds buildTimeline drops unconditionally, compared in both directions, so a fixture that grows a row cannot fall outside what the test says it covers. Still fails with the cursor withheld. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * docs: state the timeline cursor contract where consumers read it (codex round 5) The TypeScript type documented next_before/next_before_id, but the handler's own doc comment — what a REST consumer reads — still described before + limit only, and the API client method said nothing. A consumer following either could still derive a cursor from its last visible entry, which is precisely the invalid contract this change exists to replace. Both now state the pair, that it must be forwarded rather than re-derived, why (dropped rows make the last entry a different position, sometimes no position), and that the id is the tie-break among entries sharing a second — send both or neither. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(web): clear paging state when the timeline reloads (codex round 6) The previous item's entries stay on screen while the new one's page 1 is in flight — the list renders on `!loading || entries.length > 0` — so Load More is clickable during a switch, and its cursor was the OLD item's position aimed at the NEW item. Pre-existing in shape (the pre-fix code derived the same stale position from the same stale entries), but now it is one line to close, and the dedicated cursor variable is mine. loadTimeline replaces entries with page 1 when it resolves, so clearing the cursor and has_more before the await throws away no state that would have survived; it just stops offering paging for a position that is no longer known. Test holds the switch's fetch unresolved and asserts no request goes out in that window, with the button's absence as the observable. Fails with the two lines removed. Refs: BUG-2765 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
99ffad1bca |
feat(server): timeline comment rows carry the agent name (TASK-2760) (#1196)
* feat(server): carry the agent name onto comment rows in the timeline (TASK-2760) An agent's comment rendered under the human's name: the name is stamped only on the linked 'commented' activity, which the timeline suppresses because the comment card stands in for it. The comment list queries now LEFT JOIN that activity and surface the name as Comment.AgentName (top-level and nested replies, on the timeline and the comments endpoint alike, through one scan helper), mirrored onto comment-kind TimelineEntry.agent_name to match the actor_name idiom. The web comment card renders it verbatim in an isolated <bdi>, separate from the human author. Store join rather than a handler-side match: the two lists are paginated independently, so a handler join misses at page edges and reads as intermittently-correct attribution. Metadata is parsed in Go, not SQL, to keep the query free of a SQLite/Postgres dialect fork. * test(store): make the activity-window premise strict, not a same-second coin flip (TASK-2760) * fix(server): replies log + link their commented activity so the agent name reaches them (TASK-2760, codex r1) The dedicated reply route wrote no 'commented' activity, and the activity is the only row that carries the writing agent's name — so a reply through the web UI rendered under a generic chip no matter what the client sent. Also rewrites the README + SKILL.md claim that comments never show the name, moves the reply test onto the real route, and asserts order/limit under the join. * fix(store): exclude comment-linked activities in the timeline's activity query (TASK-2760, codex r2) buildTimeline suppressed a comment's linked activity only when that comment was on the same page; the two sources are paginated separately, so an activity could slip through as a standalone 'commented' card. The query now excludes linked rows via NOT EXISTS on idx_comments_activity (both dialects), exact regardless of either window, and the page-local guard is removed rather than kept as a dead one that reads as load-bearing. * fix(store): item-scope the comment/activity link and freeze comment-linked activities against debounce merges (TASK-2760, codex r3) The join keyed on activity id alone while nothing in the schema ties a comment's activity to its item — scope both the LEFT JOIN and the NOT EXISTS to the item. And CreateActivityDebounced could merge a later update into the 'updated' row a comment links to, overlaying its agent stamp and bumping created_at, so two agents under one set of credentials would silently re-attribute an earlier comment; comment-linked rows are no longer merge targets. Prose corrected: the linked row is a 'commented' row OR the 'updated' row of an update that carried the comment. * fix(server,web): keep the read-skew guard beside the SQL exclusion; nowrap on every 24ch agent label (TASK-2760, codex r4) The page-local guard covers a distinct failure from the query exclusion — a comment fetched then hard-deleted before the activity query runs — so it returns with that reason written down. Sweep: of the seven 24ch agent-label rules, three lacked white-space: nowrap (both timeline cards and EpisodeFeed), so a name with spaces wrapped instead of ellipsizing; the other four already had it. Prose nits corrected; the pre-link debounce race on update-with-comment is recorded on BUG-2716 with a pointer in the handler. * docs(server,cli): state the reverse read-skew at the guard and the CLI non-rendering decision (TASK-2760, codex r5) * fix(store): debounce merge refuses a comment-linked row inside the UPDATE itself (TASK-2760, codex r6) The read-then-write left a window in which a comment could link the chosen row before the merge overwrote its agent stamp. The merge is now one statement whose predicate re-checks the link under the row write, and a zero-row merge falls through to a fresh insert. Prose corrected: a later update looks past a frozen row, to an older unlinked one or a fresh one. * fix(store,test): one freeze mechanism, and the window-edge leak proven end to end (TASK-2760, matrix survivors) The debounce SELECT-side exclusion became redundant once the UPDATE's own predicate refused linked rows, and its 'look past to an older unlinked row' semantics folded a later change into an earlier entry — a linked row now simply ends the coalescing run. And the server suite could no longer tell the SQL exclusion from the restored in-memory guard, because it only exercised the same-page case; a test now drives the page-edge case codex found (comment outside its window, activity inside), where only the query can help. * fix(web): drop a duplicate nowrap in EpisodeFeed — the rule already had it (TASK-2760, codex r7) Corrects the round-4 sweep count: of seven 24ch agent-label rules, two lacked white-space: nowrap (both timeline cards), not three. |
||
|
xarmian
|
402f79e016 |
feat(store,server,web): collection kernel traits — de-hardcode conventions/playbooks slugs (TASK-2657, BUG-2702) (#1171)
Implements SPEC-5 §Collection traits (approved v1.1) — the first unit of
PLAN-2656 phase 0. Three kernel behaviors were keyed on the literal collection
slugs "conventions" and "playbooks": what the agent bootstrap loads, which
items route by invocation slug, and which items export as portable artifacts.
Collections now DECLARE those behaviors and the kernel resolves them from the
declarations.
Fixes the KERNEL half of BUG-2702, which stays open for the rest (see below).
A slug is not a stable identifier —
UpdateCollection re-slugs on any name change, and renaming a collection is a
documented onboarding step (TASK-1510) — so renaming either collection silently
detached all three behaviors from it, with the items still present and no error
anywhere. Measured on origin/main before the fix: conventions and
convention_index dropped 1 -> 0, playbooks 1 -> 0, and GET /playbooks/{slug}
went 200 -> 404, so `/pad ship` stopped resolving with no sign the playbook
still existed. Both halves are locked by regression tests observed failing on
unfixed code.
BUG-2702 is NOT fully closed here, deliberately. Every kernel behavior follows
the trait, and library activation on the MCP dispatcher and CLI was converted
too — but the pack's own dedicated web routes (/conventions, /playbooks list and
detail, /library) still address their collection by literal slug and render
empty after a rename. Filed as BUG-2705 with the route paths and the likely fix
shape; 2702 closes when that lands. Degradation there is bounded: no data loss,
and the collection stays usable at its own /[collection] route and in the
sidebar.
SPEC-5 was amended to v1.1 BEFORE any code, per the spec tree's own discipline:
bootstrap_include becomes a LIST of {mode, filter, key} because v1.0 could not
express convention_index at all; the conventions filter is now normative and
includes status=active, which v1.0's shorthand omitted and which the
implementation does enforce (implementing v1.0 literally would have leaked
draft conventions into every agent's boot payload); v1 filters are field-
equality maps with query/1 named as the widening path, since SPEC-2 is phase 1
and PLAN-2656 forbids growing toward it; and invocation_field is constrained to
the literal `invocation_slug`, because any other field name falls outside the
partial unique indexes in migrations/054 and pgmigrations/033 that are the real
uniqueness guard.
Traits get their own column rather than a key inside the schema JSON. The
schema column is overwritten wholesale on update and every client rebuilds it
fields-only, so a traits key stored there is destroyed by one ordinary
collection edit — measured during this task, not assumed. Trait authority
cannot rest on a value an unrelated UI save deletes. UpdateCollection writes
traits only when explicitly supplied, so pre-existing clients leave them alone;
an explicit "{}" still clears.
Bootstrap keeps its three payload keys as first-party views fed from the
declarations, and gains a generic bootstrap_includes array for any other
declared key — so the boot surface is genuinely generic rather than three
hardcoded payloads, and no consumer breaks.
Existing workspaces are backfilled slug-keyed in both dialects, guarded on
traits='{}' so a re-run cannot clobber a workspace's own declarations. The
backfill inherits today's blind spot (a workspace that renamed the collection
before upgrading is not reached) but cannot do worse than the status quo, which
is itself slug-keyed; from the backfill forward the hazard is structurally gone.
Malformed declarations are refused at create and update rather than stored:
an unparseable blob degrades to "declares nothing", which is silently the wrong
behavior instead of a loud error (SPEC-0 L6).
Web groups agent-facing collections by bootstrap_include presence, replacing a
hardcoded two-slug array repeated at five call sites.
Not done, deliberately: no MCP catalog change (traits are first-party kernel
declarations, no agent needs to set them, and the separate column means
pad_collection.update passes through harmlessly — no ToolSurfaceVersion bump);
bootstrap's collections[] projection does not carry traits (PLAN-1410 trimmed
that payload and nothing consumes them there); prefix.go's NormalizeSlug is
untouched (a pure function with no workspace context, and de-hardcoding two of
its six slugs would make it less coherent, not more).
Eight Codex review rounds found nineteen real defects, all fixed here. The
serious one:
bootstrap_include filter keys FAIL OPEN. The item store's field-filter path
drops any key its sanitizer rejects, removing the predicate rather than matching
nothing, so a declaration filtering on `"stat us"` would narrow nothing and ship
every convention — drafts included — to every agent at boot, defeating the
status=active guarantee this change makes normative. Filter keys are now
validated against the store's own sanitizer shape and pinned by a cross-package
agreement test, since models cannot import store and a future divergence would
silently reopen it. SPEC-5 amended to v1.2 with the rule and its fail-open /
fail-closed asymmetry. Also fixed: an unknown declared artifact_kind reached
artifact.Encode and surfaced as a 500 (now a 400 at the export boundary, since
SPEC-5 permits unknown kinds as legal non-round-tripping declarations); and
workspace import validated traits as JSON only, so an archive could persist a
declaration that degrades to "declares nothing" (now validated, degrading to
"{}" with a warning rather than refusing an import that may be the only copy).
Later rounds found more, and several were defects this change itself created.
A hidden collection could SHADOW a visible one: resolution used to name exactly
one collection, so with several declaring, resolving across all of them and
rejecting afterwards on visibility made a visible playbook unreachable behind a
hidden one — candidates are now filtered by visibility before selection, in both
playbook resolution and artifact import. Importing a pre-traits archive produced
an INERT workspace: the migration backfill cannot reach rows inserted long after
it ran, so conventions/playbooks arrived declaring nothing, and canonical
declarations are now inferred from the slug when a collection declares none
(never overriding declarations that survived the round trip). The generic
include path had no L4 boot budget and is now capped with an overflow count.
Trait parsing claimed to be strict but json.Decoder ignores trailing bytes, so
`{...} garbage` parsed cleanly. First-party payload keys are now mode-pinned,
since their projections have fixed shapes and declaring the other mode would be
silently ignored. Duplicate artifact_kind / invocation_field declarations are
refused at the collection API, and a conflicting archive warns on import.
Agent-facing text was updated with the rest, not after it: SKILL.md,
instructions.md and the MCP catalog said the literal slugs, which is exactly the
artifact an agent acts on. ToolSurfaceVersion 0.24 -> 0.25 for the
pad_library.activate behaviour change.
Trait uniqueness is a documented BEST-EFFORT gate, not an invariant, by lead
ruling. The gate reads then writes without a lock, import bypasses it, and a
rename can mint a duplicate without touching that path. The database-level
enforcement (partial unique indexes on the extracted traits) cannot ship first:
existing deployments can already hold duplicates via rename-then-reseed, so the
index would fail the migration on precisely the databases that most need
repairing. TASK-2710 carries the de-duplication pass and the indexes; SPEC-5
v1.3 records the deferral and the reason. L6's requirement that conflicts fail
loud is met by the refusal plus the warning — the mechanism is deferred, the
principle is not.
Gates: build · make lint 0 issues · go test ./internal/... · make test-pg ·
svelte-check 0 errors · vitest 99 files / 1734 tests. Mutation-verified across
four matrices, 20 mutations, 19 caught; the survivor is a seeding path whose
trait-vs-slug difference is unreachable today (SeedCollectionsFromTemplate
creates any missing template collection before it seeds items), recorded on the
task trail rather than papered over with a test that proves nothing.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
|
||
|
xarmian
|
449ac109e9 |
fix(server,mcp,cli): refuse system-metadata keys in fields_patch + retry-hostile error code (BUG-2627 part 2, BUG-2675) (#1166)
* fix(server,mcp,cli): refuse system-metadata keys in fields_patch + retry-hostile error code (BUG-2627 part 2, BUG-2675) Part 2 of BUG-2627 closes the door that mints the defect parts 1 and 3 dealt with: `--field implementation_notes=<json>` stored the entries as a JSON-ENCODED STRING, which is invisible to every reader and — since part 3's guard — disables `pad item note` on that item until the row is repaired. Refused SERVER-SIDE in `fields_patch`, not at the CLI as the item's scope line proposed. The deviation is deliberate and recorded on the trail: the CLI is one of three clients, and all three lower a user field-setter into the same key (`pad item update --field` at cmd_item.go, the MCP `field` param via dispatch_http_advanced.go on remote, and stdio by shelling out to that CLI). One gate closes all three; a CLI-only refusal would have left remote MCP writing the key. Both call sites were read, and the CLI's lowering is now pinned by a test rather than left as an assumption. Scope, stated because it is deliberate: this closes UPDATE only. The full `fields` blob stays open because that door is SHARED — `pad item note` / `decide` / `github link` send one, and so does convention activation via BuildConventionItemFields -> ItemCreate. Closing it would break the system writers the gate exists to protect. Item create therefore remains a mint site, tracked with the rest of that surface in BUG-2685. The refusal message is per-key: implementation_notes -> `pad item note`, decision_log -> `pad item decide`, github_pr -> the GitHub link flow, and `convention` refuses WITHOUT naming a command, because none writes it. PATTE-135 wants a remedy that works in the failing state; a single "use pad item note" line would have been wrong for three of the four keys. BUG-2675 rides along on one ToolSurfaceVersion bump, as ruled. The append refusal from part 3 reached MCP agents as `server_error` — not our fault, and not transient, so agents could reasonably retry a failure that is deterministic forever. New closed-set code `stored_state_unreadable`, emitted on BOTH transports: HTTP classifies the sentinel error directly, stdio via a `pad-structured-error/v1:` marker the CLI now writes for its own local refusal (the first marker generated without an upstream APIError). v0.16-then-v0.17 is what a one-transport fix costs. Also here: - items.ReservedOverrideKeys -> ReservedFieldKeysIn. The second caller passes a patch, not an override map, and the old doc comment said fields_patch was an open exposure — true until this commit. - `Extract* returns nil for THREE reasons` -> FOUR. The comment listed four; the count was corrected everywhere except the code. - Consumer-read artifacts updated where the claim is ACTED on, not only where it is documented: instructions.md (incl. a "do not retry this code" section), the catalog `field` param description, `pad item update --help`, README. Gates: build · make lint · go test ./... · make test-pg · Codex. Eleven-mutation matrix run against the new tests; every one killed by an assertion (two were rewritten after killing by compile error / surviving, which proves nothing). Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(server,mcp): honest remedy when the stored value is already unreadable; name the MCP-facing code (Codex round 1) Three findings from the pre-push review, all real: P2 — the refusal named `pad item note` unconditionally, but on an item whose stored value is ALREADY undecodable that command refuses too (part 3's guard). The caller was routed in a circle: field write refused -> run the note -> refused -> back again. That is exactly the failure PATTE-135 exists to prevent, and my own trail had reasoned the remedy was safe on the strength of the HEALTHY case only. The message now inspects the item's stored value and, when the key is unparseable, says so and points at the one action that works in that state (inspection), noting that the repair needs a full `fields` write no CLI flag exposes. P2 — two doc claims were false where an actor reads them. The catalog said reserved keys are refused "on every action that accepts field", which includes CREATE, and create is deliberately NOT gated; and both the catalog and instructions.md named `validation_error` (the HTTP code) where an MCP client actually receives `validation_failed`. Both corrected, and the create exception is now stated rather than implied by omission — an agent that reads only "refused on update" will otherwise assume create is fine, which is how a hole gets used. nit — the destructive-downstream sentence claimed every reserved key becomes unreadable and trips an append guard. True only for the two append-backed keys; github_pr and convention are simply overwritten. The clause is now per-key, because a confident wrong explanation is worse than a vague right one. Two more mutations run against the new branch: always-readable (the circular remedy returns) and never-readable (the working remedy disappears) — both killed by assertions. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(models,mcp,cli): one appendability predicate, per-key docs, stdio hint parity (Codex round 2) Five findings, all real. P2 — the message's readability check and the guard it describes were two different decodes. Mine unmarshalled into []json.RawMessage; the guard uses []ItemImplementationNote. A stored `[1]` passed mine and fails the guard, so the message would again have prescribed a command that refuses — the same circularity round 1 caught, through a narrower door. Replaced with models.StructuredFieldIsAppendable, which ASKS the guard rather than re-deriving it, plus an agreement test over 12 shapes x 2 keys that compares the predicate against the real Append* helpers. Verified by restoring the RawMessage version: the table catches it on `[1]`. P2 — stdio lost the new code's hint. Remote MCP told the agent retrying is pointless and how to inspect; stdio got the code with an empty hint, because the CLI's marker envelope carried none and the classifier parsed none. Both fixed, with the hint hoisted into paired constants (the same duplication StructuredErrorMarker already uses) and the test comparing the two TRANSPORTS' envelopes rather than either against a literal. P2 — doc text was still false for `convention`: the catalog, the instructions and `--help` all said reserved keys are maintained by note/decide/the GitHub flow, which is true of three of the four. Each key now names its own writer, and `convention` names library activation. Also dropped the `malformed_override` advertisement — that is the SERVER's code; an MCP client sees validation_failed for both refusals. nit — the classification test called structuredAppendErrorResult directly, so deleting either dispatcher call site left it green. Added dispatcher-level tests driving the real server + store, asserting the code, the hint, and that the item's stored fields are byte-identical afterwards. Mutation-verified by reverting the note call site. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(items,models,mcp): github_pr stays writable through fields_patch; no nil-map panic (Codex round 3) P1 — the gate refused `github_pr`, and that was wrong. My model was "system writers use the full fields blob, user setters use fields_patch", which holds for three of the four reserved keys and fails for this one: `pad github link` needs a local git checkout and the `gh` CLI, so it is excluded from remote MCP BY NAME, and internal/mcp/dispatch_http.go's noRemoteEquivalent map tells remote agents in so many words to use `item update --field github_pr=...` instead. For that audience the patch door is not a bypass of the writer — it IS the writer. So the refusal deleted a documented capability from remote agents, and answered with a message naming a command they cannot run: the same circular remedy round 1 caught, aimed this time at the people the gate was meant to help. items.PatchRefusedFieldKeysIn now exempts the key and records the rule being applied — refuse a raw write where a real writer exists — rather than the list it produces. Whether remote agents should get a proper PR-link action, so the key can be closed too, is a product question and is left as one. P2 — the hint told agents to read the bad value with `pad_item action=get`. They cannot: stripDuplicatedFieldsKeys removes implementation_notes and decision_log from every MCP response's fields blob, and the top-level arrays come from the extractor, which returns nil for exactly this shape. The value is invisible on the whole surface. The hint now says so and routes to a human, who can read it with `pad item show --format json`. P2 — `fields` holding a literal `null` unmarshals into a NIL map with no error, and both Append* helpers assign into what they get back, so `pad item note` PANICKED ("assignment to entry in nil map") instead of appending. Reproduced, fixed in parseMutableItemFields, and pinned by a test that fails on a panic rather than taking the process down. An absent blob and a null blob mean the same thing to every caller. Pre-existing, but it sits in the function family this bug is about and the message was about to recommend the command that panics. nit — README claimed a "closed eight-code taxonomy" (17 codes, and I had just added one) and read as if create lowers into fields_patch. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(models,mcp): predicate matches the append on malformed blobs; stop promising a broken workaround (Codex round 4) P1 — round 3 exempted `github_pr` from the update gate on the strength of noRemoteEquivalent's documented workaround. That workaround does not work: ingestFieldKVP (remote) and parseFieldFlag (CLI, and so stdio) both store a `field` value as a STRING, so the PR data lands double-encoded and no link appears — the BUG-2627 shape one key over. Filed as BUG-2696 with the three candidate fixes; NOT folded in, because the narrowest of them changes how every field value is typed. The exemption stands regardless: refusing would leave remote agents with strictly less than a broken door. What changes is what we may PROMISE. The catalog, instructions.md, version.go and README said "this is how you link a PR"; they now say the door is open and broken, and to hand PR linking to a human. Advertising a capability that isn't there is the failure mode this whole unit keeps circling. P2 — StructuredFieldIsAppendable returned TRUE when the whole fields blob was unparseable, on the reasoning that a broken outer blob is a different problem. True of the cause, irrelevant to the caller: the Append* helpers bail on that same parse, so the message again named a command that fails. It now returns false, which is simply the honest answer to the question asked, and the agreement table grew a malformed-outer-blob leg — the gap that let the disagreement through. P2 — the message claimed a raw field write always stores something Pad cannot read back. That holds for the CLI and MCP (a `--field` value is typed by schema lookup and these keys are in no schema) but not for a direct REST caller sending a valid array, who is refused for ownership reasons alone. Reworded to say both parts. nit — a misplaced parenthetical in the README read as if item CREATE lowers into fields_patch. It does not; it sends the full blob. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(mcp,models): stop the remote hint advertising the broken PR workaround; classify an unparseable blob as retry-hostile (Codex round 5) P1 — I corrected four artifacts that pointed agents at the github_pr field write and missed the fifth: noRemoteEquivalent's own text, which IS the message a remote agent receives when it calls `github link`, and which Codex had quoted at me in round 3 to establish the workaround existed. The nearest artifact to the actor was the one I did not open. Both entries now say there is no working remote path and name BUG-2696, with a test pinning the negative so a future edit cannot quietly reinstate the advice while the write is still broken. P2 — a fields blob that will not parse at all produced a bare parse error, so `note` / `decide` reached agents as `server_error`: transient- looking, and therefore retried, for a failure that is as deterministic as the per-key one BUG-2675 exists for. Both Append* helpers now wrap that parse failure in ErrStructuredFieldUnreadable, which both transports already classify, and the malformed-blob test asserts the sentinel rather than just an error. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * docs(mcp,cli): qualify what an agent can actually see when the state is unreadable (Codex round 6 nit) Round 5 widened stored_state_unreadable to cover a fields blob that fails to parse outright, which made half of its own hint false: MCP's normalization strips a broken structured KEY (so `get` hides it), but leaves an unparseable BLOB as a raw string (so `get` shows it). The hint and instructions.md asserted the first case for both. Now stated per layer, in the two paired constants and the instructions. The reason it is worth the words rather than being cut: an agent told 'you cannot see this' does not look, and would have missed a value that was in fact right there in the response it already had. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(mcp): classify the move/copy reserved-key refusal as validation on stdio too (Codex round 7) P2 — carried over from v0.22, surfaced because THIS bump documents the two reserved-key refusals as agreeing across transports. The move/copy message ("Field(s) reserved for system metadata and not settable here") matched none of the stdio validation patterns, so the same deterministic 400 arrived as validation_failed on remote and server_error on stdio — and server_error reads as transient, so an agent retries a refusal that can never pass. One pattern added, plus a test that drives both real classifiers with the real server message text for both refusals, so a reworded message that stops matching fails here rather than in the field. nit — the github_pr exemption is UPDATE-only; move and copy still refuse it, because there the argument is BUG-2674's (an override reintroduces the key the migration just dropped), not this one's. The catalog and instructions said "not refused" without that qualifier. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(mcp): cover the copy path's own refusal wording in the stdio classifier (Codex round 8) P2 — round 7 fixed the MOVE wording; the copy path words the same class of refusal differently ("Destination collection has no field(s): ..."), so it kept arriving as server_error on stdio and validation_failed on remote. Third message in one family, and the round-7 test used the move text for every case, which is why it missed this. The parity table now carries all three real messages plus a control leg using one the pattern list already covered — without it the table could pass by matching everything. Recorded in the pattern list's comment rather than left implicit: matching prose is a stopgap, the structural fix is the pad-structured-error/v1 marker that carries the code instead of inferring it, and until a refusal emits one, this test is where a new wording has to be added. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * test(mcp): use the real upstream codes in the parity fixtures (Codex round 9 nit) The copy legs carried `validation_error` where the handlers actually emit `malformed_override` and `invalid_override`. The 400 branch ignores the body code today, so the test passed either way — which is exactly why the fixture mattered: it was quietly recording a wrong contract, and a future code-aware classifier would regress against a table that agrees with it. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * docs(mcp): the upstream code is not forwarded to MCP clients (Codex round 10 nit) The catalog said the server's own code (validation_error / malformed_override) appears in the MCP message. It does not: the 400 branch emits code=validation_failed with a fixed "Validation failed." message and the server's text in the HINT, discarding the finer-grained code. Reworded to say what an agent actually receives, and to say that telling the two refusals apart means reading the message. Also carried the update-only qualifier on the github_pr exemption into the README, matching the catalog and instructions. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * docs(items): state the exemption predicate, not the exemption list (lead ruling) The lead's ruling on the github_pr reversal: make the REASON what the code says, so the next key added to reserved metadata is evaluated against 'does this audience have a real writer?' rather than pattern-matched onto a list that happened to be wrong for one key. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V |
||
|
xarmian
|
de96cce900 |
fix(items,server,web): reserved metadata survives a move; referential metadata travels only within its context (BUG-2674) (#1165)
* fix(items,server): reserved metadata survives a move, and dropped fields are reported (BUG-2674)
Moving an item destroyed its implementation notes, decision log and linked-PR
metadata. Well-formed data, on a routine documented operation, silently, with a
success message.
Reproduced before the fix: a note written through `pad item note` — correct
shape, visible on every surface — was gone after `pad item move`, leaving
fields as `{"status":"new"}`.
## Why it happened
items.MigrateFields drops every key absent from the TARGET schema. The reserved
keys — implementation_notes, decision_log, github_pr, convention — are system
metadata that NO collection schema declares; each renders from its own dedicated
surface rather than as a generic field. So they are absent from every targetDefs
and were dropped on every move.
That blindness is structural, not incidental: any code path reasoning about
fields BY CONSULTING A SCHEMA cannot see these keys. It is the shared root of
this bug and of BUG-2627, where the CLI types a --field value by schema lookup
and these keys fall through to a raw string.
## The enumeration comes first, deliberately
Before this there were four constants and exactly ONE non-test consumer treating
them as a set — an inline || chain in a CLI display path. Naming the set inline
again here would have created the SECOND hand-maintained list, which is the
generator pattern behind both bugs reproduced inside its own fix: the next
reserved field lands in the constants, gets wired into whichever surface
prompted it, and silently misses the other.
So models.IsReservedItemField is now the single place that knows, MigrateFields
consults it, and the CLI's || chain is converted to it — the only way it is
provably THE list rather than A list. (formatChangeValue keeps its per-key
switch: it needs to know WHICH reserved key it has, to say "notes" vs "entries",
not whether the key is reserved.)
`convention` is IN the set, settled with evidence rather than by the principle
alone: 35 of 36 conventions in a live workspace do not store the key at all, and
the one that does holds a blob that is a redundant mirror of the alias keys
beside it. No user types a `convention` object — ApplyItemConventionMetadata
writes it, via library activation and the web form. System-stamped.
## Contract
System-minted non-referential data carries; anything dropped is reported.
PLAN-2357 DR-17 settled the analogous case — tags carry because "there is no
workspace-scoped foreign key to break, so dropping them would lose information
for no safety reason". These are the same shape: inert JSON with nothing that
could dangle in a destination. The plan's carry list simply never considered
them, so there was no deliberate semantics to defer to. DR-17's own heading is
"None of this may be silent."
## The reporting half
MigrateResult.Dropped has always existed and the single-move handler has always
thrown it away, so the only record of a field disappearing was the field being
gone. It now rides the move's audit metadata — not the response body, which is
the bare item and would break every consumer, and the activity timeline is where
someone asking "what happened to my item" looks. Joined into one string because
that map is map[string]string and a raw array renders as a Go map literal in the
timeline (BUG-2628).
## Verified
Unit: reserved keys carry with their payload INTACT (asserted on the value, not
merely the key — a carry that re-encoded or zeroed it would pass a presence
check), and bypass schema matching entirely, so a target declaring
`implementation_notes` as `text` cannot coerce them. Mutants run: guard removed
-> both new tests fail; carried-but-also-reported-dropped -> the not-dropped
assertion fails; carry-everything -> the control leg fails alongside three
pre-existing tests.
Live, against a server built from this branch: the note survives the move
byte-identical, and the move's activity metadata carries
`dropped_fields: "priority, status"` for the values the target schema genuinely
has no home for.
## Known scope limit
The BULK move path still discards its Dropped list — a reporting gap only, since
the carry-through lives in MigrateFields and bulk inherits it. Threading the list
out crosses two function boundaries whose signatures serve every bulk operation,
so it is a refactor of the bulk dispatch's return contract rather than a line.
Filed as BUG-2683 rather than smuggled in here.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
* fix(server,web): close the four gaps Codex round 1 found on the carry-through (BUG-2674)
Round 1 raised no P1 and four P2s. Three were real defects introduced or exposed
by the carry-through; one was a genuine overclaim in the previous commit. All
four closed here, each mutation-verified rather than asserted.
## A schema may no longer declare a reserved key
MigrateFields carries these keys by identity, but every caller then validates
against the target schema — and ValidateFieldsDetailed iterates schema.Fields,
so it DOES see a declared key. A target declaring implementation_notes as `text`
would receive the carried array and reject it, turning a move that previously
destroyed the notes into one that fails outright. That is a worse failure than
the one being fixed: loud, but it blocks an operation that used to work.
The gate already existed — validateNoReservedFieldKeys, with its
grandfathering — and listed only parent/plan. The four metadata keys join it,
sourced from models.ReservedItemFieldKeys() so the two lists cannot drift.
Forbidding the declaration is the honest fix; coercing the value, or skipping
validation for a key the schema genuinely declares, would be guessing at which
meaning the author wanted.
The web's RESERVED_FIELD_KEYS gains the same four, preserving the existing
deliberate asymmetry (the client lowercases and is therefore stricter than the
server's exact match) so the UI steers authors away before the 400.
## The copy preflight no longer under-reports
`carried` is built by walking the DESTINATION SCHEMA, and these keys are declared
by no schema anywhere — so after the carry-through they appeared in NEITHER
bucket. A copy of an item whose content is its notes would report "nothing
carries over" while in fact retaining them. Before the carry-through they at
least showed under `dropped`, accurately. Reporting in neither is a regression
in the preflight's honesty, which is the same defect class as the move that
reported nothing.
They are now appended to `carried` after the schema-ordered entries, marked
`type: "system"` with a rendered label since they have no author-supplied one.
The bucket's doc comment says so: a client must no longer assume every `carried`
entry resolves to a destination FieldDef.
## The audit report now reaches a human
The previous commit claimed the activity timeline is where someone asks "what
happened to my item" — true, and the timeline renderer ignored the key, so the
report existed only for API and CLI consumers. Stored-but-invisible is not
reported. TimelineActivityCard renders the dropped keys on a move.
## Test aliasing
The "untouched" assertions compared the result against the SAME objects passed
in, so an in-place mutation would change both sides and DeepEqual would stay
true. The expectations are now independent deep copies — the only thing that
makes "untouched" mean untouched.
## Mutants, each run
Preflight pass removed -> the carried assertion fails. Timeline block disabled
-> the render assertion fails. Timeline action guard dropped -> the non-move
negative leg fails (a presence-only test would have passed it). Reserved-set
helper returning everything -> the IsReservedItemField control leg fails.
## Not fixed here
Codex's remaining observation — that a cross-workspace copy now carries
github_pr into a workspace whose repository it does not describe, and leaves a
convention blob detectable on an item outside the conventions collection — is a
product question about what a copy MEANS, not a defect in this mechanism. Raised
for a ruling rather than decided inside a bug fix.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
* fix(items,server): referential system metadata travels only within its context (BUG-2674)
Lead ruling on the copy-semantics fork Codex round 1 raised. It does not add an
exception to the carry rule — it applies the qualifier the rule already had.
The contract was "system-minted NON-REFERENTIAL data carries". github_pr is
referential: it names a repository that is a property of the SOURCE workspace's
project, and it hydrates into code_context and renders as a live PR link. Carried
into another workspace that link is a false statement about the destination's
project, not preserved information. implementation_notes and decision_log
describe the item's own history and are true wherever the item is.
So the rule stays one sentence: non-referential system data carries everywhere;
referential system data carries only where its referent's context still holds.
## Scope is a required argument
MigrateFields takes items.MigrateScope. Required rather than defaulted because
BOTH wrong answers lose something: SameWorkspace on a cross-workspace copy
carries a PR link into a workspace it does not describe, and CrossWorkspace on
an ordinary move DROPS metadata from an item whose repo context never changed. A
caller that must name its scope cannot pick one by omission.
The two move handlers pass SameWorkspace as a property of the endpoint, not a
guess — a move changes an item's COLLECTION and cannot change its workspace.
The copy and its preflight COMPUTE it by comparing workspace ids rather than
assuming cross-workspace, because that endpoint accepts a target_workspace equal
to the source; hardcoding would drop a github_pr from a same-workspace duplicate.
Both sides use the same helper, or the preview promises a carry the copy drops —
the DR-6 divergence the shared endpoint exists to prevent.
## The drop is reported, with a reason that explains itself
PLAN-2357 DR-17: "None of this may be silent." It would be perverse to
reintroduce a silent drop inside this fix's own new branch.
The preflight reports it as `referent_not_portable` rather than the generic
`no_target_field`. That generic reason would be actively misleading here: no
schema declares these keys ANYWHERE, so "the destination has no such field" is
equally true of the source and explains nothing about why the value is being
left behind.
## Verified
Mutants run: scope ignored (always carry) -> the cross-workspace leg fails;
generic reason on the preflight drop -> the reason assertion fails. The
same-workspace leg and the non-referential-sibling leg are what stop an
implementation that ignores scope in EITHER direction from passing — each half
alone is satisfiable by a constant.
Gates re-run for THIS commit: lint 0 · go test ./... 0 · make test-pg 0 (3282).
Web gates NOT re-run and not claimed: this commit touches no web file (the web
half of BUG-2674 shipped in
|
||
|
xarmian
|
6a2910d45a |
fix(models): refuse an append that would destroy an unreadable structured field (BUG-2627, part 3) (#1164)
* fix(models): refuse an append that would destroy an unreadable structured field (BUG-2627, part 3)
AppendImplementationNote and AppendDecisionLogEntry rebuilt their entry slice
from Extract*, then assigned it over the field key UNCONDITIONALLY. When the
stored value was something Extract* could not decode, Extract* returned nil and
the assign overwrote that value with a one-element slice -- reporting success.
Observed live, not hypothesised: an item whose implementation_notes held a
JSON-ENCODED STRING lost its stored notes to a single `pad item note` call, with
no warning on any surface. Reproduced on a scratch item with two notes; both
were gone and the command printed "Added implementation note".
The guard cannot key on Extract* returning nil, which is the obvious shape and
the wrong one. Extract* returns nil for three different reasons:
1. the key is absent -- the first append on an item. Must proceed.
2. the key holds an empty array -- well-formed, just empty (Extract* has an
explicit len == 0 -> nil). Must proceed.
3. the key holds a value that does not decode -- the defect. Must refuse.
`if Extract(...) == nil { refuse }` passes every refusal test and breaks every
first append. So assertStructuredFieldAppendable tests decodability against the
raw value in the fields map, which is the only check that separates (3) from (1)
and (2). Applied to both helpers; ErrStructuredFieldUnreadable is exported so
callers can match on it.
The refusal message deliberately does NOT name an append path. Per PATTE-135 a
suggested remedy has to work in the state where the message appears, and every
append path is precisely what is being refused; `pad item show --format json` is
read-only and does surface the raw value, so it is the one action safe to
suggest. A test asserts the message never names `pad item note`.
Tests are mutation-verified per assertion. Each mutant was run and the killing
assertion recorded: guard removed -> the refusal legs; the plausible-wrong
`Extract(...) == nil` guard -> the empty-list and explicit-null CONTROL legs
(it passes all three refusal tests, so without those controls the wrong
implementation ships green); returning mutated fields alongside the error -> the
`fields != ""` assertion, which an errors.Is check alone would not catch;
message naming an append path -> the message assertion.
Verified end to end against a binary built from this branch: the trace that
destroyed two notes now exits non-zero and both notes read back byte-identical,
while a healthy item still takes a first note and appends onto an existing one.
This is part 3 of BUG-2627 and ships FIRST by design. Parts 1 (repair the
affected row) and 2 (refuse --field for structured keys at the CLI) follow,
because part 2's error message names a remedy that destroys affected rows until
this guard exists.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
* test(models): close the coverage asymmetry Codex round 1 found (BUG-2627, part 3)
Round 1 accepted the guard condition and the error path, and was right that the
tests did not carry the bite the commit message claimed. All four gaps closed,
each mutation-verified rather than assumed:
- decision_log had no control legs of its own. The two helpers carry INDEPENDENT
guard calls, so coverage on the notes side says nothing about the log side --
an Extract-nil guard on AppendDecisionLogEntry passed every existing test.
Mutant run: it now fails the empty-list and explicit-null legs.
- No ordering assertion, so a helper that PREPENDED satisfied every length
check. Mutant run: prepending now fails "existing entry is preserved".
- The non-string refusal shapes (wrong element type, list of strings, bare
number, incompatible nested value) were untested, and the object case asserted
only the error, not the empty fields return.
- No test covered a sibling reserved field surviving a successful append.
Mutant run: rebuilding fieldsMap fresh instead of mutating the parsed one now
fails, where it previously left every notes assertion green. github_pr is the
witness because it shares the fields blob.
Codex also reported two findings that are NOT fixed here, deliberately:
P1 -- moving an item drops implementation_notes / decision_log / github_pr
entirely, because items.MigrateFields drops any key absent from the target
schema and these are reserved metadata that no schema declares. Verified by
reading migrate.go and then reproduced live: a well-formed note written through
`pad item note` was destroyed by `pad item move`, silently, with a success
message. That is worse than the defect this part guards -- it destroys VALID
data on a routine operation -- but it is a different mechanism at a different
door, so it is filed as BUG-2674 rather than folded in.
P2 -- generic field writers (--field implementation_notes=...) still reach the
fields patch and overwrite. That is BUG-2627 part 2, which ships after part 1 by
the ordering already recorded on the item.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
* test(models): pin the guard's type parameter against a wrong-but-compiling swap (BUG-2627, part 3)
Codex round 2, and it is a real hole rather than a coverage complaint. The guard
is generic, so instantiating it with the WRONG entry type still compiles:
assertStructuredFieldAppendable[ItemImplementationNote](m, ItemFieldDecisionLog)
Every test written so far passes under that swap. The two structs disagree only
on shapes nothing exercised: `{"decision":{"nested":"object"}}` is ACCEPTED by
ItemImplementationNote (unknown key, ignored by encoding/json) and REJECTED by
ItemDecisionLogEntry (Decision is a string). So the guard would permit an append
that ExtractItemDecisionLog then reads as empty -- silently destroying the stored
entry. That is precisely the guard/extractor divergence this change exists to
prevent, reintroduced one type parameter away.
Both directions are now pinned, each mutation-verified:
- decision-log cases only ItemDecisionLogEntry rejects (a `decision` holding an
object, a `rationale` holding a list). Mutant run: the notes type parameter on
the log guard fails both.
- the mirror for notes (`summary` holding an object, `details` holding a list).
Mutant run: the log type parameter on the notes guard fails both, plus the
pre-existing incompatible-nested-value case.
Correcting the previous commit message: it said round 1's four gaps were "all
closed", which was overstated -- the malformed-entry matrix still ran only
against AppendImplementationNote, which is how this hole survived it.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
* docs(models): correct the guard's own comment — four nil cases, and pin the type-parameter warning where it is read (BUG-2627, part 3)
Codex round 3 returned CLEAN with three nit-level accuracy notes against my
commit messages. Two are already self-corrected in a later message; the third
matters because the SAME undercount sits in the code comment, which is the
artifact a maintainer actually reads.
- Extract* returns nil for FOUR reasons, not three: absent key, empty array,
explicit JSON null, and an undecodable value. The code always handled null
(its own branch), the comment just did not count it.
- The type-parameter hazard round 2 found now lives in the function's doc
comment rather than only in a test name. A wrong-but-compiling instantiation
is silently DESTRUCTIVE, not merely wrong, and the next person to add a third
structured field will reach for this function without reading the tests first.
Comment-only; no behaviour change.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
|
||
|
杨成锴
|
22c5a858a1 |
fix(server): stop counting disabled conventions as completed work (#1152)
Merged after two codex review rounds (converged) on top of the community-loop supply-chain/static review. Review found two narrow follow-ups — the guest item-grant leg of the grouped terminal query keeps pre-PR over-matching semantics, and the standup/changelog display layer hardcodes `status` — both pre-existing edges, filed internally as follow-up work. Thanks @asjdf for a well-tested fix, and for honoring the per-collection terminal_options contract on both the CLI and server paths. |
||
|
xarmian
|
6f16003199 |
fix: surface implementation notes + decision log in the item timeline (BUG-2301) (#1144)
* fix(server): merge implementation notes + decision log into the item timeline (BUG-2301) `pad item note` and `pad item decide` have written structured entries since |
||
|
xarmian
|
ec7fd027fc |
feat(server,cli): watches, user-scoped event stream, plugin monitor command — PLAN-2469 Phase 1 (TASK-2533) (#1082)
* feat(store): race-free status/assignment mutation signal (TASK-2533)
Adds models.Item.LastMutation (ItemMutationSignal), populated inside the
SAME transaction that already writes status_transitions / assigned_user_id
in UpdateItemWithParentLink and MoveItemWithPreCheck. This is the
foundation for TASK-2533's watch-notification pipeline: a before/after
snapshot taken in the HTTP handler layer would race concurrent writers of
the same item, so the signal is computed where the authoritative diff
already happens, in-transaction.
* feat(store): watches table migration, both drivers (TASK-2533)
watches(id, workspace_id, user_id, item_id, predicate, created_at) per
DOC-2479's subscription-table design: durable, server-side subscriptions
that survive both the plugin-monitor process and a padd restart.
uq_watches_user_item makes `pad watch <ref>` idempotent (re-watching
upserts the predicate). Wires watches into the workspace-purge child-delete
list, mirroring item_stars.
* feat(watchevents): add in-process notification bus (TASK-2533)
New package: a global (not per-workspace) in-process pub/sub bus carrying
watch-worthy Notifications (status-change / assignment / comment; ask
reserved in the enum with no producer yet — see the follow-up server
commit). Bus is an interface specifically so a Redis-backed implementation
can slot in later without touching any caller; only MemoryBus exists today.
Package doc comment states the single-process/multi-instance limitation
explicitly, mirroring internal/events' shape.
* feat(store): watches CRUD (TASK-2533)
models.Watch + Store.CreateWatch (upsert on user+item)/GetWatchByUserItem/
ListWatchesForUser (unscoped by workspace — a watch is personal, and the
event-stream handler needs every watch a caller holds across all their
workspaces)/DeleteWatch.
* feat(server): watch/nudge event stream + CRUD endpoints (TASK-2533)
GET /api/v1/events/stream (DOC-2479): a user-scoped, cross-workspace SSE
stream, filtered server-side to the caller's watches (with optional
--until field=value predicate) plus "addressed to you" — narrowed to
assignment-to-you only for Phase 1, confirmed with the dispatcher: this
codebase has neither a Collection.Kind field nor any user->active-role
binding to ground DOC-2479's "human-gate-shaped collection targets your
active role" half mechanically. watchevents.KindAsk stays in the wire
enum with no producer. `pad session register` is the natural future hook
for a session-carried role identity.
POST/DELETE .../items/{slug}/watch, GET /api/v1/watches (unscoped,
mirrors /auth/tokens' shape for a personal, not workspace, resource).
Producer wiring (TASK-2533 audit) publishes from every live mutation path
that can produce a LastMutation signal or a new comment: handleUpdateItem
(incl. its collab sub-paths and the comment-attached-to-update path, which
bypasses handleCreateComment entirely), handleMoveItem, handleCreateComment,
item creation with an initial assignee, and the bulk-items loop (covers
archive/restore/move/set-priority/tag/untag/assign uniformly via one call
site). Named, not silent, bypasses: import bundle, status_transitions
backfill, workspace restore/purge — none are live human-facing mutations.
Known Phase-1 tradeoff, flagged not fixed: bulk mutations are NOT batched
into one notification the way the existing SSE/webhook bulk path is — a
bulk-assign of N items surfaces N individual notifications. Each is still
correctly scoped by the recipient's own watches/addressed-to-you filter
(a narrower audience than the workspace-wide SSE firehose the existing
batching protects), so this is a noise-discipline tradeoff, not a leak.
* feat(cli): pad watch + pad session register (TASK-2533)
pad watch <ref> [--until field=value] creates/upserts a durable watch;
pad watch list / pad watch remove <ref> are the hygiene companions the
dispatcher asked to be included explicitly rather than silently added.
pad watch --stream --for-session is the plugin-monitor command: one
stdout line per matching event ("PAD TASK-214 -> kind (actor): summary"),
silent on startup with no .pad.toml (hourly retry) or an unreachable padd
(backoff retry) per DOC-2479's noise-discipline contract. The retry/
backoff math and line formatting are pure, unit-tested functions; the
actual sleep loop is not (per the dispatcher's ask).
pad session register writes ~/.pad/sessions/<pid>.json (pid, cwd,
CLAUDE_CODE_MESSAGING_SOCKET when set) -- forward-looking infra for
Phase 3's live-sessions/presence surface; nothing consumes it yet in
Phase 1/2.
* fix(server): comment replies never published a watch notification (TASK-2533)
Codex round 1 finding 2 (verified real, not a false positive):
handleCreateReply is a SEPARATE code path from handleCreateComment — it
calls store.CreateComment directly via POST .../comments/{id}/replies,
not POST .../comments — and was missing the watch-notification hook
entirely. A reply to a comment on a watched item produced zero
notification. Same kind=comment publish as the top-level path, plus a
regression test covering the reply route specifically.
* fix(server): re-check current access before serving/delivering watches (TASK-2533)
Codex round 1 finding 1: ListWatchesForUser filtered only by user_id — a
watch row survives a revoked workspace membership or grant (nothing
deletes it), so GET /api/v1/watches and the event-stream's notification
filter could keep leaking item title/ref, workspace slug, actor, and
summary for access the caller no longer has.
Adds Store.ListWatchesForUser's ItemCollectionID column (needed for the
visibility check) and server.filterWatchesByCurrentAccess, which mirrors
computeSSEVisibility's RBAC resolution (handlers_events.go) — admin
bypass, VisibleCollectionIDs for member/guest full-collection access,
GuestVisibleResources for item-level grants — grouped by workspace since
a caller's watches can span many, unlike a single SSE connection scoped
to one. Fails closed on any lookup error.
Wired into handleListWatches here; the event-stream's loadWatchPredicates
call site picks up the same filter in the next commit, which also
restructures that function's Subscribe/replay sequence and therefore
touches the same lines.
* fix(watchevents): atomic ID assignment + subscribe-and-replay (TASK-2533)
Codex round 1, findings 3 and 4 (same subsystem, fixed together):
Finding 4 — sequence assignment and replay-buffer insertion happened
under SEPARATE locks in MemoryBus.Publish. Two concurrent Publish calls
could append to the ring buffer out of ID order, corrupting since()'s
ordering assumptions (it walks the ring oldest→newest assuming monotonic
IDs). Fixed by unifying seq assignment, buffer append, and the
subscriber-list snapshot under one lock; the (already non-blocking)
fan-out send still happens after releasing it.
Finding 3 — GET /api/v1/events/stream called Subscribe() and, later
(when resuming via Last-Event-ID), EventsSince() as two separate calls.
A Notification published in the window between them landed in BOTH the
replay result and the live channel, double-delivering it. Bus gains
SubscribeAndReplaySince(sinceID), which atomically subscribes and reads
the replay buffer under the SAME lock; the stream handler now uses it
whenever a Last-Event-ID is present (this commit carries that call-site
change, plus the finding-1 loadWatchPredicates filter wiring from the
previous commit — both land in the same lines of this function).
Adds a concurrent-publish ID-ordering test and a subscribe-then-
concurrent-publish no-duplicate test, both run with -race.
* fix(cli): monitor silent-start ordering + sync_required handling (TASK-2533)
Codex round 1, findings 5 and 6:
Finding 5 (P1) — runWatchMonitor called getClient() once, before the
loop and before the .pad.toml check. getClient() -> getConfiguredConfig()
os.Exit(1)s when unconfigured with no TTY, or launches an INTERACTIVE
configuration wizard when one is attached — either way a direct violation
of DOC-2479's silent-start contract, which requires "not ready yet" to be
a silent retry, never a crash or a prompt. Adds monitorClient(), which
builds the client the same way but returns a plain error instead of
exiting or prompting; client construction now happens INSIDE the loop,
after the .pad.toml gate, on every iteration, and its failure folds into
the existing padd-unreachable backoff path.
Finding 6 (P2) — streamWatchEvents ignored "sync_required" (the server's
signal that the requested Last-Event-ID was evicted from its replay
buffer), so a stale cursor got resent on every reconnect forever. Now
clears the cursor on sync_required so the next reconnect is a fresh,
non-resuming subscription instead.
Both covered by tests that assert the goroutine returns promptly on
context cancellation (proving no os.Exit / no blocking prompt was hit,
since the test process itself is still running to observe the return)
and that streamWatchEvents clears/re-tracks the cursor correctly around
sync_required.
* fix(server): uniform current-access gate for watch AND addressed-to-you delivery (TASK-2533)
Codex round 2, findings 1 and 2 — same subsystem (watch/nudge delivery
access control), fixed together; finding 2 explicitly falsifies finding
1's fix's own admin-bypass argument, so this replaces that reasoning
rather than patching around it.
Finding 1 (confirmed real): VisibleCollectionIDs / GuestVisibleCollectionIDs
deliberately over-widen for navigation — a collection ID is included if the
caller has an item grant on ANY item inside it, explicitly leaving
item-level narrowing to the caller (their own doc comments say so).
computeWatchAccessVisibility used that over-wide set directly as the
"fully visible" gate, so a guest granted item A was treated as having full
access to A's WHOLE collection, including an ungranted sibling item B.
Fixed by building the "genuinely full access" set from
GuestVisibleResources' fullCollectionIDs (populated only from direct
collection_grants, never widened by an item grant) + GetMemberCollectionAccess
/ ListSystemCollectionIDs for an actual member — exactly computeSSEVisibility's
own fullCollSet construction, not an approximation of it.
Finding 2 (confirmed real): the addressed-to-you (KindAssignment) branch in
watchNotificationVisible returned true unconditionally, with NO access
check. validateAssignmentScope (internal/store/items.go) only checks
WORKSPACE membership, never collection access, so an item can be assigned
to a "specific"-access member whose granted collections don't include it
at all — an ordinary assignment, no revocation timing required. Fixed by
gating EVERY notification kind — watch-matched and addressed-to-you alike —
through the SAME watchAccessVisibility check before either branch runs.
watchevents.Notification gains CollectionID so the check has what it needs
without a second lookup; the stream handler resolves it lazily per
workspace via a small connection-scoped cache (workspaces aren't known in
advance for addressed-to-you the way watch workspaces are), cleared on the
same reval tick that reloads the watches map.
This also required replacing computeWatchAccessVisibility's admin-bypass
argument, not just its code: "every call site filters the caller's OWN
watches" stopped being a sufficient justification once addressed-to-you
(which is fundamentally about *this* caller's own assignment activity
across every workspace) shares the same gate — a bearer-borne admin token
unconditionally trusted for that is exactly BUG-1616's blast radius. Now
mirrors computeSSEVisibility's cookie-vs-bearer distinction exactly.
Tests: guest-with-item-grant no longer sees a sibling item's watch or
stream notification (filter-level and HTTP/SSE-level); an assignment
outside a restricted member's granted collections is denied at both
levels; addressed-to-you is proven still gated (denied with no access,
visible once granted) as a pure unit test.
* fix(store): always re-read existing under lock, not just for precheck/patch updates (TASK-2533)
Codex round 2 finding 4, verified real: updateItemWithParentLinkOnce's
`existing` snapshot was only refreshed under the write lock when precheck
!= nil, ExpectedUpdatedAt != "", or FieldsPatch != nil — any update
touching none of those (e.g. a plain title-only PATCH) kept the STALE
pre-tx `existing` for the rest of the function, including the
LastMutation assignment-delta comparison added in TASK-2533's first
round. A concurrent OTHER transaction's assignment change landing between
this transaction's pre-tx read and its lock acquisition would get
misattributed to THIS transaction: a title-only update could report a
spurious, wrongly-attributed AssignmentChanged for a transition it never
made, duplicating the one the other transaction already reported
correctly (or missing a real one, depending on interleaving).
The status-transition capture already defended against exactly this with
its own separate conditional re-read; the assignment-delta capture added
later did not replicate that guard. Fixed by making the re-read
unconditional — once, right after the locks are held, before any SET-
clause building or the UPDATE itself — so every existing.* comparison in
this function is race-free by construction, not by each caller
remembering to guard itself. Also removes the now-redundant duplicate
re-read the status code had of its own.
Reproduces the exact race deterministically using UpdateItemWithPreCheck's
precheck hook as a synchronization point (TX2's assignment change blocks
mid-transaction while TX1's title-only update races its own pre-tx read
against it) — the new test fails reliably against the pre-fix code and
passes reliably (including under -race, and in Postgres mode) against
the fix.
* fix(watchevents): send under the same lock Unsubscribe/Close use (TASK-2533)
Codex round 2 finding 3, confirmed real and high-severity: Publish
snapshotted subscriber channels under the lock, released it, and only
then sent to them. A concurrent Unsubscribe or Close could close one of
those channels in the window between the snapshot and the send — a send
on a closed channel PANICS in Go, which crashes the whole padd process,
not just one subscriber's connection. The reasoning for releasing the
lock before sending ("a slow subscriber would stall everyone else") didn't
hold up: the send is already non-blocking (select/default — a full
channel is dropped-and-logged, never awaited), so holding the lock
through it costs nothing and closes the window structurally.
Adds a hammer test (many iterations of concurrent Publish / Subscribe /
Unsubscribe / Close, short-lived churned channels, recover()-wrapped so a
regression fails cleanly instead of crashing the whole `go test` run) that
reproduces "send on closed channel" dozens of times per run against the
pre-fix code (plus an independent -race detection) and passes cleanly,
repeatedly, against the fix.
* fix(server): re-fetch the user, not just the vis map, on each reval tick (TASK-2533)
Codex round 3, confirmed real: watchVisCache captured *models.User ONCE
at connect time (newWatchVisCache) and never re-fetched it; reset()
cleared only the per-workspace visibility map. computeSSEVisibility's own
doc comment explains why it re-fetches the user fresh on every call —
"so mid-stream role changes (admin demotion, user.disabled flips) take
effect on the next tick" — and the round-2 commit claimed to mirror that
"exactly," but only carried over the collection/bearer logic, not the
re-fetch itself. Net effect: a demoted or disabled admin kept fullAccess
on an open stream (both watch-matched and addressed-to-you delivery,
since both go through this same cache) until reconnect.
Adds watchVisCache.refreshUser, called by both the constructor and
reset() so the cadence matches computeSSEVisibility's actual cadence in
handlers_events.go (that function is invoked once at connect and again
only on each membershipCheck tick — never per event — so "per cache
reset" here is the same cadence, not a narrower one). Deliberately fails
CLOSED (not open-to-stale like computeSSEVisibility's own transient-error
fallback) on a fetch error, a deleted user, or a disabled user — a nudge
stream's wrong failure mode is delivering a fact to someone who
shouldn't see it, not a dropped UI update, so this trades
computeSSEVisibility's availability-leaning fallback for a stricter one
and says so in the comment rather than repeating the "mirrors exactly"
claim the fix falsified.
Tests: a unit-level pair (mirroring handlers_events_revalidation_test.go's
existing admin-demotion/disable coverage of the analogous SSE gap
exactly) proves an admin loses fullAccess after a demotion + reset(),
and a disabled user is denied outright; an HTTP/SSE-level test proves a
live stream stops delivering entirely once its connected user is
disabled and a reval tick passes. All three reproduce the bug reliably
against the pre-fix code and pass cleanly against the fix.
The HTTP-level test deliberately runs serially (not t.Parallel()): it
mutates the package-level watchListRevalInterval var, which every other
parallel watch-stream test in this package also reads via its own
ticker — writing to it from a t.Parallel() test raced against those
reads under -race (misattributed by the race detector to a whole
cluster of unrelated concurrently-running tests before this was
diagnosed). Full server package -race pass is clean after the fix.
* fix(server): decouple vis-cache reset from watch-list reload success (TASK-2533)
Codex round 4, confirmed real: on a reval tick, if ListWatchesForUser
errored, the handler's `continue` skipped visCache.reset() entirely —
the two were coupled, with reset() only reachable on the reload's
success path. A demoted or disabled user's stale identity/visibility
(round 3's fix) stayed live for exactly as long as that UNRELATED query
kept failing, so the round-3 leak reopens for the duration of any
watch-list reload error.
Fixed by running visCache.reset() first, unconditionally, before
attempting the watch-list reload. On a reload failure, the stale watch
list is kept (its own staleness is already bounded by
watchListRevalInterval's "eventually consistent" contract) but is now
gated by the FRESH visCache regardless — a demoted/disabled user is
denied via visCache even while the watch list itself lags a tick.
Chose this over dropping all delivery for the tick (the other option the
finding offered) because tying stream availability to an unrelated
query's transient health seemed like the wrong tradeoff; the comment at
the call site states this choice explicitly.
Adds a watchPredicatesLoadFault test seam on *Server (mirrors the
existing restoreAckFault pattern) so the reload failure can be forced
deterministically without breaking the DB connection for the whole test.
Reproduces the exact bug: forces the reload to fail on every tick while
concurrently disabling the connected user, and asserts addressed-to-you
delivery (which depends only on visCache, never the watch list) is
denied anyway. Fails reliably against a reverted (pre-fix, coupled)
version of the reval branch and passes cleanly against the fix.
Full server package -race pass, full suite (SQLite + Postgres) pass,
lint clean — this is the pre-PR verification matrix; round 5 will be a
narrow re-verify of this fix only.
* fix(server): bound stale watch set under persistent reload failure; atomic test seam (TASK-2533)
Codex round 5, two P2s, both confirmed real:
Finding 1 — `watches = fresh` only ran on the reload's success path, so
under a PERSISTENT (not single-tick) reload failure the watch set stayed
live indefinitely: a dead watch (removed, item deleted) kept matching
forever, and a watch created during the outage was silently missed
forever — visCache (round 4) gates current ACCESS, not whether a watch
still legitimately exists, so it couldn't catch this on its own. Fixed
by tracking consecutive reload failures and clearing the watch set once
maxConsecutiveWatchReloadFailures (3 ticks) is crossed, failing closed
on watch-matched delivery specifically while addressed-to-you delivery
(visCache-only, unaffected either way) continues throughout. Updated the
tradeoff comment at the call site so the "eventually-consistent" claim
now matches the bounded, not unbounded, behavior it actually describes.
Finding 2 — the watchPredicatesLoadFault test seam was a plain `func()
error` field, written by a test AFTER the SSE stream's background
goroutine was already running and reading it on every reval tick:
genuinely racy, unlike restoreAckFault's own use of the identical field
shape, which is set once, synchronously, before the single HTTP request
that reads it — goroutine creation's happens-before edge makes THAT
usage safe without any extra synchronization. Verified restoreAckFault
does not share the flaw and left it untouched. Fixed the watch seam with
atomic.Pointer[func() error] instead.
Test for finding 1: forces maxConsecutiveWatchReloadFailures+1
consecutive reload failures via the (now-atomic) fault seam and asserts
watch-matched delivery is suppressed once the bound is crossed while
addressed-to-you keeps delivering, then clears the fault and confirms
watch-matched delivery resumes on the next successful reload — a bounded
outage response, not a one-way ratchet. Fails reliably against the
bound disabled, passes cleanly restored.
This is the (re-run) pre-PR verification matrix per the dispatcher:
SQLite + Postgres + full-suite -race + lint + gofmt, all clean. Round 6
is a narrow re-verify of these two fixes only.
* test(store): bound the concurrent mutation-signal test's wait (TASK-2533)
CI-triage follow-up: PR #1082's plain Postgres step hit go test's default
10-minute per-binary timeout. Investigated whether any store test added by
this branch scales with runner slowness (lock-wait defaults, sleep-based
polling, transaction-hold durations):
- Watches CRUD tests (8): 0.63-0.80s each under Postgres, isolated and in
the full 741-test package run.
- Mutation-signal tests (6), including the precheck-hook two-transaction
race test: 0.48-0.80s each; the race test held at 0.48-0.51s across 10
consecutive runs (no variance) and across the full-package run.
- Full store package under Postgres: 279.17s and 277.12s across two runs
on this branch, matching the ~275s/297s baseline team-lead measured
locally and on PR #1081 — no reproducible slowdown from anything this
branch adds.
No pathological test found locally. The one test with genuine
cross-goroutine DB lock contention (TestLastMutation_AssignmentDelta_
NotMisattributedUnderConcurrentWrite) had an unbounded wg.Wait() as its
only unbounded wait — TX2's release was already unconditional (fixed 50ms
sleep, not gated on TX1's progress), so there's no deadlock risk, but
there was no ceiling on how long legitimate lock contention could
stretch it under a slow/shared runner. Replaced with a bounded 10s wait
that fails fast with a diagnostic instead of silently consuming
test-binary budget if it's ever exceeded. Verified the regression test
still fails reliably (5/5) against a revert of the round-2 fix it guards.
Could not reproduce the CI timeout locally; likely the pre-existing
~297s CI baseline (already noted as close to the 10-minute ceiling)
plus environmental variance on the shared runner, not a specific test
this branch adds.
|
||
|
xarmian
|
1eb1c9eda6 |
feat(server): expose ACL-gated moved-to pointer on item GET (TASK-2359)
An item MOVED to another workspace — copied, then archived — can now say where it went. GET on a single item gains an optional `moved_to` block naming each destination in displayable terms (workspace slug + item ref + title + collection slug), so a consumer can render a link without a second call. No HTTP redirect, no resolver change. The ACL gate is the point. A destination is revealed only after the caller independently passes AuthorizeCrossWorkspaceRead (TASK-2358) with an ITEM scope on the destination item itself. Workspace-level access is not sufficient: a restricted member of the destination workspace, or a guest holding one unrelated item grant there, has a role in that workspace while having no right to the copied item's collection. A caller who fails that check sees NO hint a destination exists. The key is omitted entirely — not a null, not an empty array, not a boolean — so the response is byte-identical to an archived item with no move record at all. A structurally distinguishable response is itself the leak. Restore decision: the block is OMITTED for a non-archived source. Restoring a moved-out source leaves two live items with the same content in two workspaces, which is legitimate, but at that instant the source has not moved anywhere and the response must stop asserting that it did. Past-tense provenance is the back-pointer question and applies equally to plain copies, which this field must never claim as moves. Also honored: DR-2a (only archived_source rows feed the pointer; plain copies are back-pointer material only), per-destination filtering over the forward lookup's SET with no short-circuit on the first hit or first denial, newest-first ordering, a scan bound on the per-GET authorization cost, and deliberate isolation of the hand-rolled public share-link DTO — pinned by an explicit negative test that freezes its key set. Claude-Session: https://claude.ai/code/session_01E2fRi12n8rARczvdEa2LYT |
||
|
xarmian
|
bf14c1168a |
feat(store): add item_workspace_moves provenance table (TASK-2356)
Phase 1 of PLAN-2357. Durable record of "this item was copied/moved from workspace A to workspace B", backing the forward redirect (TASK-2359) and the destination's back-pointer. Implements DR-2 / DR-2a. Paired, dual-dialect, forward-only migrations (migrations/077 + pgmigrations/055). archived_source distinguishes a move from a plain copy (INTEGER on SQLite, BOOLEAN on Postgres, written through dialect.BoolToInt). source_seq is a NULLABLE per-source move ordinal that exists solely so two moves inside the same second are orderable — created_at is second-precision RFC3339, so archive -> restore -> move again would otherwise resolve to an arbitrary destination. Partial index (source_item_id, source_seq DESC) WHERE archived_source, deliberately NOT unique: restore-then-move-again legitimately repeats. The back direction IS uniquely indexed — a destination item is created by exactly one copy, in the same transaction that writes its provenance row, so a duplicate there would silently change which source the back-pointer names. Cascade is asymmetric on purpose, inverting item_collection_moves: the archived source is precisely the row whose pointer must survive, so source_item_id carries no FK at all; target_item_id cascades, because a pointer at a vanished destination is worse than no pointer. Store accessors: a tx-taking insert helper (no self-committing variant — the row must land in the copy transaction), a forward lookup returning a SET newest-first, and a back lookup. The insert rejects an archived row with no seq and a copy row with one, so DR-2a's ordering invariant is enforced at the write boundary rather than assumed. NULL ordering is normalized with COALESCE because SQLite and Postgres disagree on DESC NULL placement. Also wires workspace purge, which the two-workspace shape requires: both workspace columns are RESTRICT references, so a purge clearing only one direction would fail outright when the purged workspace sits on the other end. Tests cover insert-in-tx, forward lookup with multiple destinations ordered newest-first and scoped to one source, back lookup, rollback leaving no row, and both DR-2a criteria. The ordering and scoping tests use fixed row IDs whose lexical order contradicts the expected answer, so deleting the ordering term or the WHERE clause under test fails them on every run rather than half the time; verified by mutating the production query. Claude-Session: https://claude.ai/code/session_01E2fRi12n8rARczvdEa2LYT |
||
|
xarmian
|
40f88052cd |
fix(collab): version restore via prune+reseed (BUG-2264) (#990)
Version restore didn't reconcile the live Y.Doc: peers kept editing a Y.Doc built on pre-restore ops, and their next collab-snapshot flush clobbered the restored items.content. Reworked restore to prune+reseed — the restored content becomes canonical and every peer converges on it (unflushed edits are discarded, which is exactly restore semantics), replacing the earlier applier/epoch/watermark routing. handleRestoreItemVersion drives RoomManager.ForceRefreshRoom under the per-item lock. Hardened across Codex xhigh review rounds: - Atomicity: pre-prune MAX(op-log), the items.content write, the "Restored from…" version, the op-log wipe, AND both durable restore boundaries all run in ONE store transaction. A failed commit rolls back all of it — no divergent state, no fail-open boundary. - Unambiguous commit signal: UpdateItem reads the updated row WITHIN the tx (getItemTx) before commit, so a read failure can't make a committed update look failed and the returned seq is this restore's. - Restore freeze: conns are paused via a dedicated rc.frozen flag (NOT canWrite) so the auth-revalidation loop can't thaw the freeze mid-restore or promote a viewer; pickApplier + the applier-ack handler reject frozen conns so a concurrent external PATCH can't falsely succeed. - Stale-flush boundary: pre-prune MAX+1 fences in-flight snapshot cursors under the same item lock. - force_refresh fan-out deadlock: per-conn timer-close so a wedged writeLoop can't hang the fan-out + item lock. - Stale-SEED clobber: the client announces the item.seq it seeded from (?content_seq=) on every (re)connect; Join force_refreshes any seed that predates the last restore. Residual #1 (restart-durability) CLOSED durably, for BOTH stale vectors — the in-memory fences didn't survive a restart, so a surviving cursor-0 pre-restore browser tab wasn't fenced on reconnect. Two nullable per-item columns (migration 075 SQLite / pg 053), both stamped in the restore's own tx (atomic with the content write + op-log prune): * items.last_restore_seq — the content generation. Join's stale-seed fence reads it (via store.ItemLastRestoreSeq) when the in-memory fast-path misses (after a restart); if that read errors, Join fails CLOSED via a RETRYABLE plain close (not a force_refresh, which would discard the Y.Doc and spin an unbounded refresh loop) so the client reconnects with backoff, Y.Doc intact. * items.restore_boundary_op_id — the op-log-id boundary. The collab-snapshot flush gate reads it (via store.ItemRestoreBoundaryOpID) when the in-memory RestoreBoundary misses (after a restart), failing closed (409) on a read error, so a surviving tab's stale HTTP flush is fenced too. No SCHEMA_VERSION bump — durable columns are not a Y.Doc node-spec change. Deferred to BUG-2276: (a) a Postgres commit whose ack is lost is treated as rolled-back (needs commit-outcome reconciliation; SQLite unaffected); (b) a restore rollback racing an in-flight external-applier ack can drop the ack and retry/fall back (needs the applier flow serialised under itemLock at a 30s-stall cost). NOTE(BUG-2270): ForceVersion can mint same-second version rows; the item_versions ordering tie-breaker is tracked separately. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra |
||
|
xarmian
|
1dbe04399a |
fix(store): optimistic concurrency + sibling broadcast for collection settings (BUG-2265) (#989)
* fix(store): optimistic concurrency for collection settings writes (BUG-2265) Collection-level settings (e.g. quick_actions) were written by reconstructing the whole settings JSON from a caller's local Collection snapshot, and UpdateCollection replaced the column with no concurrency check. Two ItemDetails in the same collection (full-page pane host master + pane) hold independent snapshots and clobbered each other. Mirror the item optimistic-concurrency pattern (IDEA-1480): add CollectionUpdate.ExpectedUpdatedAt; when set, UpdateCollection re-reads updated_at atomically under the workspace write lock (SQLite BEGIN IMMEDIATE / Postgres advisory xact lock) and returns CollectionUpdateConflictError on a mismatch. Empty token keeps the legacy last-write-wins path unchanged for CLI/MCP/API callers. No DB migration — reuses collections.updated_at. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * feat(server): collection.updated broadcast + 409 conflict mapping (BUG-2265) - handleUpdateCollection boundary-validates expected_updated_at (400 on a malformed token) and maps store.CollectionUpdateConflictError to the shared update_conflict envelope (HTTP 409) — byte-identical wire shape to the item path, via the extracted writeUpdateConflictEnvelope helper. - Add the collection_updated EventBus type and publish it after a successful update so sibling ItemDetails / collection pages refresh their independent Collection snapshot proactively, shrinking the 409 window. Routed by Collection (slug) through the existing SSE visibility filter. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(web): 409-aware collection settings writes + sibling refresh (BUG-2265) - CollectionUpdate carries expected_updated_at; add isUpdateConflictError. - QuickActionsMenu sends the token and, on a 409, refetches the collection, re-appends the new action onto the FRESH settings, and retries once — no silent loss, no user-visible error. - EditCollectionModal captures the token at open-time (edge-gated seed so a concurrent broadcast can't wipe in-progress edits) and shows a non-destructive "changed elsewhere, reload" message on 409 rather than auto-merging a full-form edit. - Subscribe to collection_updated over SSE: ItemDetail and the collection page refresh their own Collection snapshot (gen/slug-fenced against the persistent pane host's no-remount switch), so siblings converge before the next save. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(store): harden collection optimistic concurrency + web fetch ordering (BUG-2265, Codex round 1) Address Codex review findings: - [P1] same-second clobber: now() is one-second precision, so two guarded writes in the same second kept an identical token. The accepted write now advances updated_at strictly past the token (only when now() hasn't already moved on), making a stale-token replay deterministically conflict. Add a same-second regression test. - [P1] tokenless-writer race on Postgres: the advisory lock only serialized writers that also took it. Replace it with a `FOR UPDATE` row lock on the in-tx re-read (Postgres) — SQLite's BEGIN IMMEDIATE already serializes every writer — so a concurrent tokenless UpdateCollection can't slip between the re-read and the UPDATE. - [P2] rename broadcast: only publish collection_updated when the slug is unchanged. A rename's old-slug event would make siblings refetch a dead slug (404) and a new-slug event can't reach old-slug visibility snapshots; renames are handled by the existing navigation path. - [P2] out-of-order refreshes: ItemDetail and the collection page now use a dedicated monotonic refresh counter so two rapid collection_updated fetches can't resolve out of order and clobber newer state (loadSeq/loadGeneration only bump on route/item loads). Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(store): make collection updated_at strictly monotonic for ALL writes (BUG-2265, Codex round 2) The previous same-second advance ran only on guarded updates, so a tokenless UpdateCollection could write the current second over a forced expected+1s, regressing the concurrency token and letting a stale guarded client clobber newer data (Codex P1). Route every collection update through one small transaction that re-reads updated_at (FOR UPDATE on Postgres; SQLite BEGIN IMMEDIATE covers it) and derives the new timestamp atomically: strictly advance past the row's current value when now() hasn't already moved on. This makes updated_at a reliable concurrency token for guarded AND tokenless writers. Add a tokenless-monotonic regression test. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: close remaining collection-concurrency gaps (BUG-2265, Codex round 3) - [P1] Board column reordering (handleGroupReorder) rebuilt the full schema from a stale local snapshot and wrote it with no token — a lost-update path identical to the bug being fixed. Now sends expected_updated_at and, on 409, refetches, re-applies the reorder onto the fresh schema, and retries once. - [P2] The workspace settings page seeded EditCollectionModal from a page-load-time collections list, so a change that predated editing produced a false 409. It now refreshes the list on collection_updated (seq-guarded). - [P2] collection.updated is now delivered to item-grant-only SSE subscribers for collections they can see — it's itemless but leak-free (only the slug), so guests' ItemDetail schema/settings snapshots converge too. Filter test extended. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(web): switch-safety + conflict-merge fixes for collection writes (BUG-2265, Codex round 4) - Board column reorder now ABORTS on a 409 (with a "reorder again" toast) instead of replaying a stale option order onto the fresh field, which would silently drop a concurrent option add/remove/rename. Reordering is cosmetic; never worth clobbering a real schema edit. Also captures ws/slug/base before the await and fences the write against a route switch. - QuickActionsMenu captures workspace + collection identity BEFORE the first await, so a mid-save navigation can't make the 409 refetch/retry target the wrong collection (no guaranteed remount). - Settings-page SSE refresh captures the workspace and drops the result if the workspace changed while fetching, so a slow refresh for workspace A can't overwrite workspace B's freshly loaded collection list. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(store): sub-second collection updated_at token, no future drift (BUG-2265, confirming pass #6) The same-second monotonic advance manufactured whole-second FUTURE updated_at values; sustained >1 write/sec on one collection drifted arbitrarily ahead of wall-clock. collections.updated_at is TEXT on both dialects and never compared lexically (only via time.Equal + display), so switch the update write to sub-second nowNano(): same-second collisions become near-impossible, so the token advances naturally. Keep a strict-monotonicity guard but step by a single NANOSECOND on the (now near-impossible) coarse-clock/step-back collision, so any drift is bounded to nanoseconds. Dual-dialect; covered by make test-pg. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(server): sanitize + always-broadcast collection event (BUG-2265, confirming pass #2,#3) - #2 (P1): collection_updated is delivered to item-grant guests, but the event carried ActorName/Source, leaking the owner's identity + edit source. Strip them — publishCollectionEvent now emits workspace + slug (+ new_slug) only. - #3 (P2): always broadcast (including on rename), routed by the OLD slug and carrying the NEW slug via a new Event.NewSlug field, so remote tabs on the old slug can re-target instead of silently 404ing on their next action. Tests: assert no actor/source leak on a settings update; assert a rename routes by old slug + carries new_slug. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(web): decisive switch-safety + rename handling for collection writes (BUG-2265, confirming pass #1,#3,#4,#5) - #1 (P1): EditCollectionModal captures the target collection id/slug/name/ws + updated_at when the form is SEEDED, and handleSave/handleArchive now operate on that captured identity (not the live props). The seed effect re-seeds when the collection IDENTITY changes (not on a same-id broadcast refresh), so a reused route can't leave A's form saving/deleting to B. - #3 (P2): on a rename event the collection route navigates to the new slug (preserving the pane query) and ItemDetail refetches by new_slug; the SSE event type carries new_slug. - #4 (P2): the reorder-conflict path refetches the collection (reseeds the token) before prompting, so a missed SSE event doesn't make every retry 409 forever. - #5 (P2): QuickActionsMenu only invokes oncollectionupdated when the live workspace/slug still match the captured identity, so a reused route can't assign an old response to the newly-navigated page. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(server): nano token round-trip, rename visibility, publish-before-migration (BUG-2265, confirming pass 2) Address the round-2 confirming-pass findings (server-only): 1. (P2) The shared update_conflict envelope formatted actual_updated_at with second precision (time.RFC3339), truncating the now sub-second collection token so the client's retry token never matched — a permanent 409 loop. Format with time.RFC3339Nano. Item tokens are zero-nanosecond, so RFC3339Nano emits no fractional part — the item 409 wire shape is byte-identical and the item path still compares via time.Equal. Added a test that the returned token round-trips as a usable retry token. 2. (P2) Rename events are routed by the OLD slug, but a subscriber that revalidated after the rename only has the NEW slug in visibleSlugSet, so the visibility check dropped the event before the new_slug branch. Accept a rename when EITHER the old slug or the (authorized) NewSlug is visible; downstream item-grant gating uses whichever slug is visible. Filter test extended. 3. (P2) The collection_updated event was published only after field migrations succeeded, but UpdateCollection already committed (updated_at advanced). On a migration failure clients got a 500 and no refresh, leaving siblings with stale tokens that 409 blindly. Publish on the commit (before the migration), so siblings always resync regardless of migration outcome. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: atomic collection update+migration; modal same-id rename retarget (BUG-2265, confirming pass 3) Address the round-3 confirming-pass findings; defer cross-tab rename RE-NAVIGATION to BUG-2272 (placeholder) per coordinator. 1. (P1) Migration atomicity. UpdateCollection committed the schema + concurrency token BEFORE MigrateItemFieldValues ran, so a migration failure returned 500 with the row already changed → the retry was guaranteed to 409 and item values were left inconsistent with the committed schema. Made the two ATOMIC: extracted applyFieldMigrationsTx and run it inside UpdateCollection's own transaction (after taking the workspace seq lock), so a migration failure rolls back the schema AND the token — nothing changes, the retry works. The handler now passes migrations through instead of running them separately, and publishes the event only after the fully-atomic commit. store/tx work → make test-pg run green. 2. (P2) EditCollectionModal same-id rename. The round-1 identity capture ignores same-id prop refreshes (to preserve edits), but a concurrent RENAME changes the slug (not the id), so handleSave/handleArchive PATCHed a dead slug → 404 before the token could 409. On a same-id prop change whose slug changed, the seed effect now retargets the endpoint slug + re-captures the token WITHOUT reseeding the form (in-progress edits preserved). Deferred (BUG-2272, TODO comments added, already broken on main — no regression): - ItemDetail full-page item URL/collSlug not retargeted after a remote rename. - Collection route chained-rename events during SSE replay landing on a dead intermediate slug. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(web): keep seeded token on same-id rename retarget (BUG-2265) On the EditCollectionModal same-id rename branch, retarget the endpoint (slug/name/ws) only — drop the token re-capture. Re-capturing let a later handleSave succeed against the renamed collection and apply the modal's stale pre-rename full form, silently REVERTING the concurrent rename (the exact stale-snapshot clobber BUG-2265 prevents). Keeping the seeded token means a concurrent rename correctly yields a 409 → the non-destructive "collection changed, reload" message. Slug-retarget without token-recapture gives both: no 404 (right URL) and no clobber (409 fires). Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: lock-order deadlock + unified collection-snapshot fences (BUG-2265, confirming pass 4) 1. (P1) DEADLOCK regression. UpdateCollection's atomic migration path took the collection-row FOR UPDATE lock and THEN the workspace seq lock, but item creation takes them in the reverse order (workspace advisory lock first, then the collection-row FK lock on INSERT) — a concurrent item-create + schema-migration ABBA-deadlocks on Postgres. Fix: acquire the workspace seq lock BEFORE the collection-row FOR UPDATE (matching item-create's order). Every store path that locks both now takes them workspace-seq → collection-row (tryCreateItem, UpdateItem, MigrateItemFieldValues, UpdateCollection). Added a concurrency regression test (item-create racing schema-migration); make test-pg green. 2+3. (P2) Cross-generation fence gap. The SSE collection refresh and route/item loads used SEPARATE counters, so a stale in-flight load could complete after a fresh SSE refresh and revert the collection + its concurrency token. Unified to a SINGLE monotonic collection-snapshot generation in BOTH the collection route and ItemDetail — every collection-snapshot write (loadCollection/loadData, the SSE refresh, reorder, and the quick-action/edit-modal callbacks) bumps it on start and gates its assignment on "still latest". ItemDetail's load keeps a switch-escape so a stale refresh for the OLD collection can't block loading a NEW one. Settings page unified the same way over its collections-list writes. 4. (P2) Settings page fed a stale editingCollection to the edit modal after a remote rename (its prop never changed → the same-id-rename retarget never fired → 404). The unified refresh now re-points editingCollection at the refreshed object for the same id, so the modal's retarget fires. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: emit item-changes signal on field migration; QuickActions retry-by-id (BUG-2265, confirming pass 5) 1. (P1) A collection update that runs a field migration mutates item `fields` JSON and advances item `seq`, but only collection_updated was published — open item views refreshed collection METADATA and returned without reconciling the migrated items, so clients kept stale field JSON under the new schema and a later full-fields item update could UNDO the migration (a clobber). UpdateCollection now returns the migrated-item count; when > 0 the handler ALSO emits the existing bulk item-mutation signal (items_bulk_updated, Op=migrate) so open views reconcile via /items-changes. Fires only when the migration touched >= 1 item — a pure settings/quick-actions update emits nothing extra. No store SQL/locking change (Go signature + count plumbing only); make test / make test-pg both green. 2. (P2) QuickActionsMenu's 409 retry GET-by-slug 404s if the competing update renamed the collection. Resolve the fresh collection by STABLE id (list + find by id) before re-appending + retrying, mirroring EditCollectionModal's identity approach; the result-propagation guard is now id-based too so a rename doesn't spuriously drop it. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: uniform sweep of item-grant delivery, rename routing, and 409/404 retries (BUG-2265, confirming pass 6) One pattern-sweep instead of per-site patches. Audited every event this PR publishes and every client retry path, applying three patterns uniformly: PATTERN A (item-grant SSE reconcile) + B (old-slug rename routing): instead of a SEPARATE items_bulk_updated migration event (which carries op/count for items an item-grant subscriber can't see and isn't rename-routed), FOLD a SANITIZED `items_changed` bool onto collection_updated — already item-grant-delivered (round 3) and already old-slug-routed with new_slug (round 2). On it the client triggers a /items-changes deltaSync (server-filtered to the caller's grants) and ItemDetail refetches its open item, so item-grant EDITORS reconcile migrated field JSON — closing the clobber where a stale full-fields update would UNDO the migration. Leak surface: "a collection you can see items in changed [+ renamed + had item changes]" — a bool, no per-item data. Removed the round-5 items_bulk_updated publish. The pre-existing items_bulk_updated (archive/move) is untouched and correctly stays suppressed for item-grant users. PATTERN C (409 AND 404 in retries): a competing RENAME can 404 a slug-targeted write before it can 409, bypassing recovery. Added isNotFoundError / isConflictOrNotFound helpers; every write/retry path now treats BOTH: QuickActions save resolves-by-id and retries on either; board reorder reseeds-by-id and aborts on either; EditCollectionModal save shows the reload prompt and archive resolves-by-id and retries on either. Tests: server asserts collection_updated sets items_changed on migration (not on settings-only) and stays sanitized; the SSE-filter test asserts the migration variant reaches item-grant subscribers for a visible collection; web unit tests assert 404/409 classification and a real component-driven not_found -> resolve- by-id -> retry in QuickActionsMenu. make test / make test-pg / npm run test all green. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: stable collection-ID identity for collection events + request-based items_changed (BUG-2265, confirming pass 7) 1. (P1) Collection events were identified only by MUTABLE, reusable slugs, and events replay — so a stale rename event's old slug, once re-owned by a DIFFERENT collection, could pass a slug-based match and misroute a client (navigate away / load the wrong schema) or leak the new slug. Fix at the ROOT: carry the STABLE CollectionID on collection_updated (Event.CollectionID) and match by ID everywhere: - Server visibility: sseEventVisibleFor matches collection_updated on a new visibleCollIDSet (built from the same VisibleCollectionIDs), not the slug — so an event for a collection the subscriber can't see by ID is dropped even if its (reused) slug is in visibleSlugSet. Filter test proves the slug-reuse drop. - Clients: ItemDetail and the collection route match `event.collection_id === <their collection>.id`, not slug. Slug(s)/new_slug stay only for the rename-navigation URL. Settings refreshes its whole list (already id-safe). 2. (P1) items_changed was keyed off the affected-ROW count, delivered to item-grant subscribers → a subscriber whose own items were unaffected could infer that HIDDEN items matched the migrated value. Now keyed off whether a field MIGRATION WAS REQUESTED (len(input.Migrations) > 0), independent of row count — leaks nothing about hidden item values. Reverted round-5's UpdateCollection count-return (no longer needed). Test: a migration matching ZERO items still sets items_changed. Deferred with markers: - NOTE(BUG-2273) at ItemDetail's reconcile-skip AND updateField: the web editor's full-fields field write lacks item-level OCC (never adopted IDEA-1480/v0.14), so the migration reconcile is best-effort. - TODO(BUG-2272) at the reorder 404 reseed: it refreshes `collection` but not the route `collSlug` (renavigation, deferred). Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: archive OCC (no destructive wrong-target) + settings load fence (BUG-2265, confirming pass 8) 1. (P1) EditCollectionModal handleArchive resolved the target by stable id but the server DELETE re-resolves by the MUTABLE slug — a rename that re-owned that slug before the delete landed would archive the WRONG collection. Close the TOCTOU with an expected_updated_at OCC on the delete, mirroring the update OCC: DeleteCollection re-reads updated_at under a lock (FOR UPDATE on Postgres) and 409s on mismatch; the handler validates the token + maps the 409; the client sends it as a query param; handleArchive passes the seeded token (and the fresh token on the resolve-by-id retry). A reused slug or a concurrently-changed target now yields a clean 409 → the reload message, never a wrong-collection archive. Server test: stale token 409s (and the collection survives); current token 204s; malformed 400s; no token 204s. 2. (P2) settings load(): the generation was bumped AFTER awaiting setCurrent, so a slow load for workspace A could resume after B's load and clobber B's name/context/collections/members. Capture a dedicated loadGen at load() ENTRY (before any await) and fence EVERY continuation on it; the collections write additionally respects collectionsGen so it can't revert a fresher SSE refresh. Using a dedicated loadGen (not the SSE-shared collectionsGen) means an SSE collections-refresh mid-load doesn't drop the name/members writes. Deferred: TODO(BUG-2272) at the collection route's rename-navigation site — the global collectionStore (sidebar/pickers) isn't refreshed and the workspace layout ignores collection_updated, so the sidebar keeps the dead slug. Layout- level renavigation, deferred. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(web): dedicated item-snapshot fence + id-based rename comparisons in ItemDetail (BUG-2265, confirming pass 9) One comprehensive ItemDetail async-snapshot fence sweep so this file's item/ collection fencing is uniform and ID-based. 1. (P1) The migration item-refetch and loadData shared loadGeneration, so the refetch could apply migrated fields and then a stale loadData response overwrite them (a later full-fields edit then undoes the migration). Added a DEDICATED itemGen (separate from loadGeneration and collectionGen), bumped at the start of BOTH loadData's item load AND the migration refetch, and gated BOTH `item = ` writes on "still latest itemGen" — neither can stale-overwrite the other. Swept the other PASSIVE item snapshot-refreshes onto itemGen too (SSE item_updated/archived/restored, onSync deleted/incremental/full, the collab refresh) so they're ordered against each other and the migration/load. 2. (P2) A settings update that follows a rename before the rename fetch completes requested the OLD slug and bumped collectionGen, cancelling the valid rename fetch. Fetch slug is now `event.new_slug || event.collection || slug`. 3. (P2) The loadData collection fence-escapes compared the stale load's SLUG vs the freshly-renamed snapshot's slug (they differ on a rename → escape let the stale result overwrite). They now compare stable collection IDs; the SSE refresh's post-fetch identity check is id-based too. Audit (site -> generation -> id?): every PASSIVE snapshot-refresh (loadData item+collection, migration refetch, SSE x3, onSync x3, collab) bumps the correct dedicated gen (item->itemGen, collection->collectionGen) and compares identity by id. The DELIBERATE user/action writes (title/field/tag/assignee/role/content/ link/version/restore saves) keep loadGeneration + item-id switch-safety; their item-snapshot concurrency vs the migration refetch is the deferred item-OCC gap (BUG-2273, best-effort) — reordering them last-started-wins is orthogonal to that. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra |
||
|
xarmian
|
c72fe5a663 |
feat(items): add unparented filtering contract (#926)
* feat(items): add unparented filtering contract * fix(items): preserve unparented projection state * fix(views): preserve reserved filter on reset * fix(items): resync projection scope changes * fix(items): address PR 926 review findings - localIndex: fetch snapshot before clearing store/cache in resyncProjectionScope (no data-loss window on fetch failure) - items: degrade to committed item when post-parent-link readback fails instead of 500 - items: treat unparented=<non-true> as a field filter so a schema field named unparented still filters - persistence: delete dead persistCursor - mark validateUnparentedListRequest canonical; cross-reference the 3 early-feedback copies * fix(items): resync race + purge safety per Codex review (round 1) - resyncProjectionScope: merge-reconcile instead of blunt clear so a higher-seq upsert/delta racing the snapshot fetch is preserved (not erased) and the cursor never regresses below it - recheck generation after persistWipe so a sign-out/403 purge during the wipe can't resurrect purged rows via persistDelta - snapshot rows authoritatively replace local copies (drop is_unparented on downgrade); mergeRow's projection-preservation is bypassed for resync * fix(items): sanitize projection bit on preserved racing rows per Codex review (round 2) When a projection resync lands a restricted snapshot, strip is_unparented from any racing higher-seq row kept by the seq guards — the old scope no longer grants it. Keep the row itself (dropping it would reintroduce the racing-mutation data loss; server 403 enforces real visibility). * fix(items): transactional cache replace in resync per Codex review (round 3) Replace wipe()+persistDelta() in resyncProjectionScope with a single persistReplace() transaction (clear + write in one tx). Avoids the deleteDatabase() onblocked cross-tab hang where a pending delete stalls the following reopen+write indefinitely, wedging the resync promise. wipe() stays for the sign-out / schema-mismatch full-teardown paths. * fix(items): drop-and-replay resync reconciliation per Codex review (round 4) Rework resyncProjectionScope: drop every row absent from the authoritative snapshot (not just older-than-cursor ones) and pin the cursor to the snapshot cursor. A post-snapshot mutation the client can still see is re-fetched by the next /items-changes?since=cursor under the NEW scope, so visible rows return and old-scope-hidden rows stay gone — no old-scope row survives the resync, and nothing is permanently lost. Present-in-snapshot racing edits are still kept (is_unparented stripped under a restricted scope). * fix(items): continue delta poll after resync so replay actually fires (round 5) The drop-and-replay resync (round 4) pins the cursor to the snapshot cursor so post-snapshot mutations re-fetch under the new scope — but both poll loops broke out / returned immediately after the resync, so the replay never ran until an unrelated sync/reload. Both callers now continue the loop from the pinned cursor; resync already aligned the scope so the branch can't re-fire, and the existing 50-iteration cap bounds it. * fix(items): keep pendingResync set until replay catches up (round 6) resyncProjectionScope cleared pendingResync after installing the snapshot but before the pinned-cursor replay drained. If that replay later failed or hit the 50-page cap, pendingResync stayed false and the next bootstrap() no-opped with racing mutations still missing. Let the reconcile loop's caughtUp logic own the flag instead. * fix(items): set pendingResync when any resync begins (round 7) Round 6 removed the premature clear but only the bootstrap path pre-sets pendingResync; a page deltaSync resync ran with it false, so a failed/capped replay there wouldn't trigger a bootstrap resume. Set pendingResync=true at the start of resyncProjectionScope so any caller marks catch-up pending; the reconcile loop clears it on caughtUp. * fix(items): fence stale optimistic writes + epoch-guard resync catch-up (round 8) Adds a resync-epoch + fenced-id mechanism to close the last two race classes: - fencedIds: a resync records the ids it dropped (hidden under the new scope). upsert() refuses a fenced id, so a stale old-scope create/update response resolving after the resync can't resurrect a now-hidden row that no new-scope delta would evict (P1). An authoritative applyDelta re-add un-fences; the next resync recomputes the set (re-upgrade clears it). Self-contained in the store — no epoch threading through the optimistic callers. - scopeEpoch: bumped when a resync installs a new snapshot. Both reconcile loops capture it before each /items-changes and skip treating a response that raced a concurrent resync as caught-up, so a stale in-flight delta can't clear pendingResync without validating the pinned cursor (P2). Regression test covers fence → reject stale upsert → authoritative re-add un-fences → later edits accepted. * fix(items): bump scope epoch before resync fetch (round 9 P2) scopeEpoch advanced only after listIndex() returned, so a reconcile response racing the fetch saw the old epoch and could clear the pendingResync the resync set at start. Bump the epoch before the network await instead. |
||
|
xarmian
|
bfa32dde5a |
fix(security): encrypt webhook HMAC secrets at rest, mask in responses (BUG-2057) (#915)
Webhook signing secrets were stored plaintext in the webhooks.secret column and echoed back in every API response. Encrypt them at rest (reusing the existing AES-256-GCM store helpers, same pattern as TOTP secrets) and return the raw secret ONLY in the creation response; list responses now mask it and expose a has_secret flag instead. - store: encrypt on CreateWebhook, decrypt on Get/ListWebhooks so the dispatcher still signs with the plaintext secret. Reuses the secret column with the "enc:" prefix — no new column/migration. Keyless self-host stays a no-op fallback (encrypt returns plaintext; decrypt passes legacy rows through unchanged). - BackfillEncryptWebhookSecrets encrypts pre-existing plaintext rows on startup once a key is configured (idempotent), mirroring the TOTP backfill. - model: add HasSecret so masked responses still signal presence. - handlers: mask secret on list; document raw-only-on-create. - tests: encrypt-at-rest round-trip + HMAC validity, list decrypt, plaintext backfill/back-compat, and the API mask-except-on-create contract. Claude-Session: https://claude.ai/code/session_015yuBJQYfDj95cgX3DaD8SF |
||
|
xarmian
|
3f69b76b06 |
feat(security): enforce session UA binding under strict mode (TASK-2056) (#912)
Session IP/User-Agent binding was log-only by default, so a stolen session token granted durable any-origin access. IP-change enforcement already existed behind PAD_IP_CHANGE_ENFORCE=strict; this extends the same single toggle to also enforce the User-Agent-hash binding. When strict enforce is ON, a request whose client IP OR User-Agent hash no longer matches the session's stored binding now revokes the session (DeleteSessionIfExists) and rejects the request (401 for API, revoked-passthrough for public/browser paths), killing the stolen token. When enforce is OFF (default), behavior is unchanged: UA mismatch is logged (slog only, no new audit row) and the request proceeds, so existing self-host users see no behavior change and routine client churn (browser/WebView updates, DevTools emulation, mobile-app rebuilds) is tolerated. The UA hash is stable within a real session, so UA-mismatch enforce carries fewer false positives than IP enforce (mobile roaming, VPN toggles, carrier NAT) — documented in the handler comment. Adds the ActionSessionUAChanged audit action, emitted only in strict mode. No DB migration: reuses the existing IPChangeEnforce config flag and the existing session store primitives. Claude-Session: https://claude.ai/code/session_015yuBJQYfDj95cgX3DaD8SF |
||
|
xarmian
|
bed933d7fd |
feat(items): field-level PATCH + conflict envelope + read-only version history (TASK-2022) (#876)
* feat(items): field-level PATCH + conflict envelope + version history Adds three related item-update primitives (TASK-2022 / IDEA-1480): - Field-level merge: PATCH `fields_patch` shallow-merges onto the item's current fields INSIDE the write transaction (null deletes a key), so concurrent single-field updates no longer clobber each other via the full-blob read-modify-write. `pad item update` and the MCP `pad_item.update` action now send only the changed keys. - Optimistic concurrency: optional `expected_updated_at` on update; on mismatch the store returns *UpdateConflictError and the handler emits the pad-structured-error/v1 conflict envelope (HTTP 409, code=update_conflict). Surfaced on CLI (`--expected-updated-at`) and MCP (`expected_updated_at`). - Read-only version history: `pad item history <ref>` (alias `versions`) and MCP `pad_item.history`, reusing the existing item_versions store + versions endpoint (no new store, no schema change). MCP ToolSurfaceVersion bumped 0.9 -> 1.0 (new action + param; update behavior change). No migration required. Claude-Session: https://claude.ai/code/session_019knGmnHcx5rrgWXQ8V8DZS * fix(items): address Codex review — dispatcher fields_patch, OCC ordering, date/required guards Round 1+2 review fixes for TASK-2022: - HTTP MCP dispatcher (dispatch_http_advanced.go) now sends fields_patch (only changed keys) instead of a client-side merged full fields blob, and forwards expected_updated_at — remote MCP callers get the same race-free merge + optimistic concurrency the CLI/HTTP paths do. - ValidatePartialFields rejects null-deleting a schema-declared REQUIRED field (would otherwise persist a blob the full-update validator rejects). - Open-children guard on the fields_patch path merges the patch onto the IN-TX locked row inside the precheck (not a stale pre-lock preview), so a priority-only patch can't false-fire the guard. - Optimistic-concurrency check now runs BEFORE the open-children precheck in the store, so a stale expected_updated_at yields update_conflict (not open_children) — single in-tx re-read shared by both. - Date auto-population on the patch path only fills an EMPTY current date; an existing end_date the caller isn't touching is preserved. Tests added for each fix. Claude-Session: https://claude.ai/code/session_019knGmnHcx5rrgWXQ8V8DZS |
||
|
xarmian
|
c846cff4fd |
feat(project): agent-accessible activity feed (pad project activity + MCP action) (#877)
* feat(project): agent-accessible activity feed (pad project activity + MCP action)
Add a non-streaming, bounded activity query so agents can catch up on
what other agents/users changed since they last worked — the query
counterpart to the live `pad project watch` SSE stream.
- CLI: `pad project activity [--limit N] [--actor user|agent] [--since DATE]`
backed by the existing GET /workspaces/{ws}/activity feed.
- MCP: `pad_project.activity` action (passThrough) + cloud HTTP route.
- Extend the activity endpoint with a server-side `since` date filter
(handler parse + store SQL clause) so limit/actor/since behave
identically across CLI, stdio MCP, and cloud HTTP transports.
- Bump ToolSurfaceVersion 0.11 -> 0.12 (drift guard, README, CLAUDE.md,
instructions.md) and add a SKILL.md querying-guidance line.
Tests: store since-filter test, HTTP dispatch test, catalog action test.
Claude-Session: https://claude.ai/code/session_019knGmnHcx5rrgWXQ8V8DZS
* fix(mcp): mark pad_project.activity read-only in tool surface
Add activity to readOnlyActions so the serialized MCP tool surface emits
read_only:true (missing entries default to write). Spot-check it in
tool_surface_test.go to guard against regression.
Claude-Session: https://claude.ai/code/session_019knGmnHcx5rrgWXQ8V8DZS
|
||
|
xarmian
|
48104a5eff |
perf(server): collapse dashboard/bootstrap N+1 into set-based queries (BUG-2002) (#847)
The dashboard builder (also reused by the bootstrap endpoint and every pad_set_workspace) ran ~1000 queries on a large workspace: one GetItemLinks + per-link GetItem for every non-done item (blocked attention + suggested_next filter), a GetChildItems per active plan (progress + suggested_next), a GetItemIncludeDeleted per recent-activity row, a GetCollection per visible collection, and a per-collection COUNT via ListCollections whose result the dashboard never uses. Replace every per-item/per-row loop with a set-based query: - GetBlocksEdges: one workspace-wide JOIN of blocks-links -> blocker essentials, ordered created_at DESC to preserve the old first-active- blocker selection. Drives both blocked attention and the suggested_next blocked-filter (retires itemBlockedByActive). - GetChildItemsForParents: one IN query grouping all active-plan children by parent (progress + suggested_next; no-content projection). - GetItemsByIDsIncludeDeleted: one IN query batch-hydrating recent-activity items (include-deleted). - ListItemsParams.NoContent: skip loading full markdown bodies on the count/summary scans (allItems, plans, stalled, orphaned). - ListCollectionsMinimal now also selects slug; the dashboard uses it in place of ListCollections, dropping the unused per-collection COUNT N+1 and the GetCollection-per-visible-id loop. Per-item N+1s are gone; query count is now constant in workspace size. Verified byte-identical dashboard + bootstrap JSON against three live workspaces (docapp/claude/apm); dashboard latency ~376ms -> ~198ms on the 1907-item docapp workspace. New store methods are unit-tested. |
||
|
xarmian
|
0aa431f132 |
fix(server,cli,mcp): default item list to per-collection non-terminal filter (BUG-2001) (#845)
The CLI's default `pad item list` (no --status/--all) sent a hardcoded ~20-status allowlist as the status filter. Collections with custom status vocabularies (blog: drafting/scheduled; human-tasks: todo) fell outside the list and had their open items hidden. MCP inherited the same bug via the CLI default and the HTTP route table's mirrored allowlist. Replace it with a server-side `non_terminal` filter: ItemListParams.NonTerminal resolves each collection's terminal set from its schema's terminal_options (falling back to DefaultTerminalStatuses) and keeps only items NOT in that set — reusing the existing doneFiltersForWorkspace + buildChildrenDoneExpr machinery, applied in both the normal and FTS query paths. The CLI default and both MCP dispatch paths (ExecDispatcher via the CLI, HTTPHandlerDispatcher via mapItemList) now send non_terminal=true. --status X and --all semantics are unchanged. |
||
|
xarmian
|
b0eeef16ce |
feat(store): email_verification_tokens + SendEmailVerification + token reaper (TASK-1936) (#806)
Wave 2 of PLAN-1933 — verification-token infrastructure (pure infra; no endpoint consumes it until Wave 3). - Migration 071 (SQLite) / 049 (Postgres): email_verification_tokens table, cloning the password_resets shape (id/user_id FK/token_hash/expires_at/ used_at/created_at + token_hash + user_id indexes), per-dialect created_at default. - Store email_verification.go: 256-bit crypto/rand token, padver_ prefix, SHA-256-at-rest, non-destructive Lookup, atomic UPDATE...RETURNING Consume. Deltas from password_resets (DR-2): 24h TTL, keep invalidate-prior-on-mint (resend burns the old link), consume side-effect sets users.email_verified_at (RFC3339-with-Z, same format Wave 1's migration used) in one transaction — no password reset, no session mint. - Email SendEmailVerification: clones SendPasswordReset, "1 hour" -> "24 hours". - Token reaper (DR-5): lifecycle-safe background sweep (mirrors orphanGC/opLogGC — self-registers on Server.bg, context-cancellable via stop channel, started only from cmd/pad/main.go so unit tests don't leak goroutines) calling the four previously-unwired CleanExpired* methods (email verifications, password resets, sessions, CLI auth sessions) hourly. Adds CleanExpiredEmailVerifications. - Audit consts ActionEmailVerified + ActionEmailVerifiedByAdmin. Gates: make check + make test-pg green (store + migration on both dialects). Claude-Session: https://claude.ai/code/session_01HxBkAMiFBtCRJ2tKSCt3ST |
||
|
xarmian
|
6a63fba188 |
feat(store): add users.email_verified_at column + model plumbing (TASK-1935) (#805)
Wave 1 of PLAN-1933 (email verification). Pure infra — nothing reads the column until Wave 3, so this is behaviourally a no-op and mergeable early. - Migration 070 (SQLite) / 048 (Postgres): add nullable email_verified_at TEXT, mirroring disabled_at. UNCONDITIONALLY backfill every existing row to verified (RFC3339 'Z'-suffixed) so no existing / OAuth / self-host account is write-locked on deploy (inverted vs password_set's conditional backfill). SQLite ALTER without IF NOT EXISTS; Postgres with it. - SAFE default = verified (DR-3): CreateUser / CreateOAuthUser write a verified timestamp unless UserCreate.Unverified is explicitly requested (only the future cloud self-serve branch will set that). A missed call site fails SAFE (verified), not write-locked. - models.User.EmailVerifiedAt + IsEmailVerified() (mirror IsDisabled). - Update userColumns + BOTH scan sites (scanUser AND the inline SearchUsers scan) so the admin user list keeps working. - Expose derived email_verified bool in sessionUserPayload for a later wave. Gates: make check + make test-pg both green (dual-dialect verified). Claude-Session: https://claude.ai/code/session_01HxBkAMiFBtCRJ2tKSCt3ST |
||
|
xarmian
|
010af13abd |
fix(ci): gofmt internal/models/workspace.go to unbreak Go job (BUG-1911) (#784)
The mid-struct doc comment added in #781 split the Workspace struct into two gofmt alignment groups; the file landed without re-running gofmt, leaving golangci-lint red on main and every PR since. Claude-Session: https://claude.ai/code/session_01CL1pBjNpPUX6SWkuAuYXHS |
||
|
xarmian
|
584ac9a806 |
fix(web): treat CLI/MCP-created workspaces as agent-connected (BUG-1557) (#781)
`pad init` connects an agent (installs the skill, stores credentials) and creates a workspace, but the web UI still showed the "connect an agent" banner and onboarding launchpad. The only signal for "agent connected" was has_agent_activity — an item existing with source cli/mcp — and a fresh pad-init workspace has zero items, so the UI nagged to connect an agent the user already had. Give the server a truthful signal: a workspace created through an agent surface already has an agent wired up before it creates its first item. Add a `source` column to workspaces (web/cli/mcp), attributed authoritatively server-side from the request auth shape (actorFromRequest) — never from the request body, so a web client can't spoof "cli" to self-suppress the prompts. The dashboard ORs source in (cli,mcp) into has_agent_activity when the cheap item check comes up empty. - migrations 069 (sqlite) / 047 (postgres): workspaces.source NOT NULL DEFAULT '' (legacy rows stay "unknown", never treated as agent-created) - models.Workspace.Source + WorkspaceCreate.Source (json:"-", server-set) - thread source through the CreateWorkspace INSERT + all 7 workspace scan sites (workspaces.go, workspace_members.go) - handleCreateWorkspace derives source from actorFromRequest - OnboardingLaunchpad step 1 collapses to "Agent connected" when the agent is already wired up, shifting emphasis to "tell it to set up" Web modal and cloud-signup auto-create flows are unchanged and still correctly prompt to connect (source web / empty). Tests: store source round-trip across reads; dashboard reports agent-connected for a cli-created workspace with zero items; web-created stays not-connected until an agent item exists; a web body-spoofed source is ignored. Claude-Session: https://claude.ai/code/session_01HxBkAMiFBtCRJ2tKSCt3ST |
||
|
xarmian
|
915f7e66c5 |
fix(web): show issue ID and status pills in activity log (BUG-1748) (#776)
Activity rows (the dedicated Activity page and the dashboard's Recent
Activity list) showed only the item title, never the issue ID. Add the
ref (e.g. BUG-1748) as a leading monospace badge on both surfaces.
The ref rides on the per-row item lookup that already runs to populate
the title, so there are no new DB queries — enrichActivities and the
dashboard recent-activity builder now also copy item.Ref after
ComputeRef(). New item_ref field on models.Activity, DashboardActivity,
and the TS Activity / recent_activity types.
The Activity page now renders field changes as structured pills
("status: open → fixing") instead of a raw string, via a new shared
parseFieldChanges util that also replaces the private copy in
TimelineActivityCard.
Claude-Session: https://claude.ai/code/session_01HxBkAMiFBtCRJ2tKSCt3ST
|
||
|
xarmian
|
3704cc2c9f |
fix(store): cast jsonb metadata to text for Postgres LIKE + gofmt (BUG-1702) (#693)
The status-transition backfill query used `a.metadata LIKE '%→%'`, but activities.metadata is jsonb on Postgres where LIKE (~~) is undefined, failing TestBackfillStatusTransitions(_SeedSeqBelowHop) and erroring in any Postgres deployment. Cast to ::text on Postgres (dialect-guarded), matching AttachmentReferenced. Also gofmt comment.go + the share-links test that were tripping golangci-lint. |
||
|
xarmian
|
076fb9b2e7 |
feat(comments): comment editing backend — user_id, UpdateComment, PATCH, SSE (TASK-1663) (#665)
* feat(comments): comment editing backend — user_id, UpdateComment, PATCH, SSE (TASK-1663)
Foundation for comment editing (PLAN-1662). No migration — comments.user_id
already exists (012_users.sql) but was never written or exposed.
- Populate user_id on create/reply: CreateComment takes an explicit userID
param (passed from currentUserID by the handlers, not via the request body
so it can't be spoofed). Expose user_id on models.Comment + all comment
SELECTs/scans. The workspace export path is left as-is — imported comments
keep NULL user_id (admin-only edit), matching the pre-identity fallback.
- Store.UpdateComment(id, body): replaces body + bumps updated_at; the
comments_fts_update trigger re-indexes.
- PATCH /workspaces/{ws}/comments/{commentID}: author-or-admin only
(canEditComment), rejects empty body. Editing is an authorship op, distinct
from delete (item editors). NULL user_id → admin-only.
- comment_updated SSE event: broadcast from the handler; added to the web
sse allowlist + ItemTimeline refresh set.
- web: api.comments.update(), Comment.user_id type.
Tests: author edits own (200), non-author non-admin (403), admin edits
anyone (200), empty body (400), NULL-user_id comment is admin-only.
Parent: PLAN-1662.
* fix(account): detach authored comments on account deletion per Codex review (round 1)
Now that TASK-1663 populates comments.user_id (FK to users.id),
DeleteAccountAtomic would fail on the FK for any user who authored a
comment. Null comments.user_id for the user before deleting the row —
comments live on in soft-deleted/other workspaces; the display-name
author is preserved and the comment just becomes admin-only to edit.
Regression test added.
|
||
|
xarmian
|
1b1068537c |
feat(tags): workspace tag enumeration endpoint + cross-collection filter (TASK-1653) (#658)
* feat(tags): workspace tag enumeration endpoint + cross-collection filter (TASK-1653)
Foundation for the tags feature (PLAN-1652 / IDEA-1649). The write path and
per-collection ?tag= filter already existed; this adds tag enumeration and a
verified cross-collection read so a single tag can group items of any type.
- store: dialect.JSONArrayElements unnests a JSON text-array column
(json_each on SQLite, jsonb_array_elements_text on Postgres);
Store.ListWorkspaceTags returns distinct tags + item counts, ordered by
count desc then tag asc, with the same collection/item ACL filters as
ListItems so counts never leak hidden items.
- server: GET /workspaces/{ws}/tags (handleListTags), respecting collection
visibility + guest item grants.
- models: TagCount{tag,count}.
- cli: client.ListTags + `pad tag list`.
- web: api.tags.list + TagCount type (items.list already forwards `tag`).
- tests: store-level (cross-collection aggregation, collection scoping,
non-nil-empty = empty, archived excluded) and handler-level (a Task + an
Idea sharing one tag; GET /tags counts + ordering).
Parent: PLAN-1652.
* fix(tags): count distinct items per tag, not tag occurrences per Codex review (round 1)
COUNT(DISTINCT i.id) so an item with duplicate tags (e.g. ["ux","ux"]) is
counted once — the write path doesn't enforce per-item tag uniqueness.
Adds a regression test.
|
||
|
xarmian
|
eeff78118b |
feat(insights): per-user layout customization + persistence (TASK-1634) (#645)
* feat(insights): per-user layout customization + persistence (TASK-1634)
Let users personalize the Insights surface, persisted per-user per-workspace:
toggle which metric cards show, and remember the window + collection filter.
Backend:
- migrations 064/043: user_report_layouts (user_id, workspace_id, config JSON,
PK(user_id,workspace_id), ON DELETE CASCADE) — dual-dialect.
- models.ReportLayout (hidden_cards/default_window/default_collections) +
ReportCardIDs/ValidReportWindow validation.
- store.GetReportLayout / SaveReportLayout (ON CONFLICT upsert, both dialects).
- GET/PUT /workspaces/{ws}/report/layout — per-user; PUT sanitizes window +
filters hidden_cards to the known card set. web client + TS type.
Frontend (Insights page):
- loads the saved layout, hydrates window/collections/hidden cards
- a "Customize" panel toggles each card (SvelteSet-backed); each section gated
on !hiddenCards.has(id); Totals always shown
- debounced auto-save, gated on a per-workspace `hydrated` flag so it never
saves during load or stomps another workspace's layout on switch
Single config per user (no named/multiple layouts — deliberate v1 scope).
Parent: PLAN-1628.
* fix(insights): save layout only on explicit user changes, not on load per Codex review (round 1)
The auto-save $effect ran once after hydration (loadLayout assigns reactive
state, then flips hydrated=true), firing a PUT /report/layout on mere page
view — which 401s on no-user/legacy-token sessions and bounces the user to
/login. Replace the effect with a scheduleSave() called only from explicit
handlers (toggleCard, selectWindow, toggleCollection, clearCollectionFilter);
hydration never saves. Also capture wsSlug at schedule time and drop the
pending save if the workspace changes mid-debounce, so A's edit can't land
on B.
|
||
|
xarmian
|
5dfc2921b2 |
feat(store): structured status-transition log + backfill (TASK-1637) (#637)
* feat(store): structured status-transition log + backfill (TASK-1637)
Add a status_transitions table capturing every item status change as a
structured, queryable row — written in the same tx as the item update and
never debounced — so the Reports surface (PLAN-1628) can reliably compute
the completed-throughput and cycle-time series.
- migrations/063 + pgmigrations/042: status_transitions table (dual-dialect),
indexed on (workspace_id, created_at) and (item_id, created_at)
- write-path hook in UpdateItemWithPreCheck records from→to on status change
- BackfillStatusTransitions: one-time startup replay parsing the historical
activities.metadata.changes blob (mirrors BackfillWikiLinks), gated on an
empty table; wired into cmd/pad/main.go
- models.StatusTransition + tests (capture, multi-hop, no-op, parser, backfill)
Spike (TASK-1629) found the activity log records status changes only as a
human-readable, debounce-coalesced metadata string — unusable for aggregation.
This is the foundation TASK-1630 (report aggregation) builds on.
* fix(store): record status transitions on item move too per Codex review (round 1)
MoveItemWithPreCheck rewrites fields outside UpdateItemWithPreCheck, so a
status-changing move override (pad item move ... --field status=done) was
not recorded in status_transitions, making the table non-canonical. Insert
the from→to row in the move tx as well, stamped with the target collection.
Adds move-path capture tests (status override + status-preserving move).
* fix(store): make status-transition backfill idempotent per Codex review (round 2)
The empty-table gate isn't atomic, so concurrent replays (a future
multi-replica Postgres deploy; single-instance today) could double-insert
historical rows and overcount reports. Give backfilled rows a deterministic,
activity-derived primary key ("bf_" + activity id) and a dialect-aware
conflict clause (ON CONFLICT DO NOTHING / INSERT OR IGNORE) so a re-run
no-ops instead of duplicating. Count only rows that actually land.
Write-path rows keep using a random newID(), so live data never collides.
* fix(store): accurate from_status under lock + document backfill caveats per Codex review (round 3)
1. from_status was read from the pre-lock `existing` snapshot. When no
precheck ran, a concurrent update (serialized behind the locks we hold)
could make it stale. Capture the status from a fresh in-tx read BEFORE
the UPDATE (reading after would see the new value and drop the hop).
Applied to both UpdateItemWithPreCheck and MoveItemWithPreCheck.
2. Backfill stamps historical rows with the item's current collection_id;
reconstructing the collection at each past status change would require
replaying move history. Documented as a best-effort, historical-only
caveat (exact for the common never-moved case; live write/move paths
stamp the collection at transition time).
* feat(store): track collection done-field + seed create-time transitions per Codex review (round 4)
1. Generalize capture from hard-coded "status" to each collection's done
field (DoneFieldKey: status, or BoardGroupBy field like stage/result for
hiring/interviewing). Add a field_key column recording which field the
row tracks (robust to later BoardGroupBy changes). Applied to update,
move, and backfill paths.
2. Seed a create-time "entered initial status" transition on CreateItem and
in the backfill (Pass 2), so an item created directly in a terminal value
still counts as a completion. Initial value reconstructed from the item's
earliest recorded change, else its current value.
Also: item_id FK is ON DELETE CASCADE so hard-deletes clean up transitions.
Tests cover non-status done-field, create-in-terminal, create-seed, and
cascade-on-delete; full store suite green.
|
||
|
xarmian
|
905876af04 |
feat(backlinks): cross-workspace wiki-links + request-independent ACL (Phase 2b) (#622)
* feat(backlinks): cross-workspace wiki-links + request-independent ACL (Phase 2b) Phase 2b of PLAN-1593 (TASK-1597). Completes the wiki-link reverse index by indexing and surfacing `[[workspace::REF]]` cross-workspace references. Builds on Phase 2a's title work (PR #621). Phase 3 (TASK-1596) owns the UI/MCP/CLI rendering changes. What changed - internal/store/backlinks_visibility.go (new): request-independent ACL helper `Store.ResolveBacklinksVisibility(userID, workspaceID, includeDeletedItems)`. Mirrors the role-determination + collection- merge logic from server.guestResourceFilterCore but doesn't depend on a request context, so cross-ws traversal can compute per-source- workspace ACLs without a `workspaceRole(r)` lookup. The Codex planning-round review caught the prior plan reusing the request- scoped helper as a hidden architectural cost; this is the resolution. - internal/server/server.go: guestResourceFilterCore refactored to delegate to the new store helper. Keeps the request-scoped wrapper signature stable for all existing handler call sites; only the internals move. - internal/links/extract.go: lift the Phase-2a workspace_ref emit gate. WikiLinkKindWorkspaceRef now flows through ExtractWikiLinks alongside ref and title kinds. parseBody recognition was already in place from earlier rounds. - internal/store/wiki_links.go: WikiLinkKindWorkspaceRef branch in replaceWikiLinks stores (target_workspace_id, target_ref) verbatim, resolving the slug→ID via new resolveWorkspaceSlugTx (with per-call cache so repeated `[[ws::X]]` in one body don't re-query). Unknown slugs persist with target_workspace_id=NULL — broken-link semantics, identical to existing ref/title patterns. - internal/store/wiki_links.go: new `Store.GetCrossWorkspaceBacklinks` enumerates accessible workspaces via Store.GetUserWorkspaces (which includes guest-only access — broader than membership query), then per-workspace computes visibility via ResolveBacklinksVisibility and runs the SQL backlinks query with the per-ws (FullCollectionIDs, GrantedItemIDs) predicate inline. Results sorted by updated_at DESC in Go, paginated globally. Per-workspace safety cap (offset+limit) prevents one workspace from dominating the global slice. - internal/store/wiki_links.go: new `Store.CountBacklinks` for same-ws pagination boundary detection. Needed so the handler knows where the cross-ws tier begins for pages 2+. - internal/models/backlink.go: new `SourceWorkspaceSlug string` (omitempty) field. Populated only by cross-ws rows; same-ws rows leave it empty so the existing wire shape is preserved. - internal/server/handlers_backlinks.go: union pagination across same-ws and cross-ws tiers. Same-ws first (matches the renderer's UI mental model — your own workspace's links at the top of the panel). Count-based slice math handles pages 2+ correctly when same-ws is exhausted. Tests - internal/links/extract_test.go: workspace_ref forms emit correctly (bare, display alias, mixed case, invalid-slug fallback to title). - internal/store/wiki_links_xws_test.go (new): six cross-ws scenarios plus a role-matrix test: - end-to-end cross-ws index + query - non-member sees nothing - guest with collection grant sees only that collection - guest with item grant sees only the granted item - unknown workspace slug → broken row, no query results - same-ws rows leave SourceWorkspaceSlug empty - ResolveBacklinksVisibility role matrix (admin/full member/guest with grants/non-member non-grant) Out of scope (Phase 3 / TASK-1596) UI rendering of cross-ws backlinks (workspace badge + workspace- prefixed ref), MCP `pad_item.action: backlinks` cross-ws fields, CLI display tweaks. PLAN-1593 / TASK-1597. * fix(backlinks): admin enumeration + cross-prefix ref fallback + unbounded perWsCap (Codex round 1) Three P2 findings from Codex round 1 against PR #622: Finding 1 — admin users miss cross-ws backlinks. `GetUserWorkspaces` returns only memberships + grant-only guest workspaces, but RequireWorkspaceAccess (middleware_auth.go:481) gives admins implicit access to every workspace. An admin querying for backlinks would silently miss links from workspaces they're not explicitly a member of. Fix: in GetCrossWorkspaceBacklinks, branch on user.Role: - admin → s.ListWorkspaces() (every non-deleted workspace) - non-admin → s.GetUserWorkspaces (memberships + grants) Stale user IDs return empty result rather than erroring. Finding 2 — cross-ws ref matching doesn't handle cross-prefix moves. Same-ws is immune because target_item_id is resolved at parse time and survives renames/moves; cross-ws resolves at query time, so a `[[other-ws::OLD-42]]` row written before the target moved from OLD→NEW collection wouldn't match a query under the NEW ref. Fix: in queryCrossWorkspaceBacklinksForWorkspace, dual ref-match clause: exact `LOWER(wl.target_ref) = LOWER(?)` OR `LOWER(wl.target_ref) LIKE LOWER('%-N')` where N is the item_number from the target ref. Pad prefixes are alphanumeric with no internal `-`, so trailing `-N` uniquely identifies the number suffix — no false positives like "TASK-142" matching "%-42" (LIKE anchors to the trailing literal). Finding 3 — per-workspace cap of 1000 silently broke pagination beyond offset>=1000. The 1000 ceiling was defensive paranoia; the correct math is offset+limit per workspace (worst case all rows come from one workspace and the global slice still needs that many). Fix: drop the 1000 ceiling. perWsCap = offset+limit unconditionally. For runaway offsets the per-workspace transfer cost is proportional; documented as a known characteristic (callers shouldn't be paging past offset=10000 anyway). Regression tests: - TestWikiLinks_CrossWorkspaceAdminSeesAllWorkspaces: admin sees cross-ws backlink without being a workspace member. - TestWikiLinks_CrossWorkspaceRefNumberFallback: move target to new collection, query under new ref, old-ref-stored row still surfaces. PLAN-1593 / TASK-1597. * fix(backlinks): honor OAuth/MCP token workspace allow-list (Codex round 2) Codex round 2 P1: cross-workspace backlinks bypassed the OAuth/MCP token's workspace allow-list (TASK-952). A token consented for workspace A but with the underlying user having access to B would still surface source rows from B via the cross-ws query — leaking data outside the token's consent scope. Fix: thread `allowedWorkspaceSlugs []string` through GetCrossWorkspaceBacklinks. Handler populates it from TokenAllowedWorkspacesFromContext(r.Context()): - nil → no token gate (PAT or pre-TASK-952 token, allow all) - "*" wildcard → allow all - explicit list → strict slug membership Workspace enumeration skips any source workspace whose slug isn't in the allowlist. The same-ws path is unchanged because RequireWorkspaceAccess already gated the target workspace against the allow-list (so we only reach this handler when the target IS in the list). Regression test in wiki_links_xws_test.go covers four shapes: nil, wildcard, target-only (blocks cross-ws), explicit source-workspace (allows cross-ws). PLAN-1593 / TASK-1597. * fix(backlinks): normalize limit at handler boundary (Codex round 3) Codex round 3 P2: the backlinks handler parsed ?limit=N but didn't normalize it before computing the same-ws/cross-ws pagination split. GetBacklinks and GetCrossWorkspaceBacklinks each clamp >300 internally, but the handler's 'remaining := limit - len(sameWs)' used the original (potentially huge) value. With ?limit=301 and more than 50 same-ws backlinks, the first page would mix cross-ws in before same-ws was exhausted, violating the documented tier order. Fix: clamp 'limit' to <=300 at the handler boundary, before any pagination math runs. PLAN-1593 / TASK-1597. * fix(backlinks): normalize same-workspace [[ws::REF]] to ref-kind (Codex round 4) Codex round 4 P2: `[[<current-ws>::TASK-1]]` was being indexed as a workspace_ref row with target_workspace_id = current workspace. But the same-ws GetBacklinks query requires target_item_id (workspace_ref rows leave it NULL), AND GetCrossWorkspaceBacklinks explicitly skips the target workspace — so the link rendered and navigated correctly in the UI but no backlink ever surfaced. The renderer's L307 short-circuits same-workspace fully-qualified form to behave identically to `[[REF]]`; the index must follow. Fix: in replaceWikiLinks, normalize a workspace_ref link to ref-kind when its slug resolves to the current workspace. The promotion canonicalizes the ref (via new links.CanonicalizeRef exported alias) so `[[ws::task-5]]` stores the same canonical shape as `[[TASK-5]]`. Tests: - TestWikiLinks_CrossWorkspaceSameWorkspaceQualifiedNormalized: same-ws fully-qualified `[[ws::REF]]` surfaces in same-ws backlinks and is absent from cross-ws backlinks. PLAN-1593 / TASK-1597. * fix(backlinks): same-ws qualified ref miss doesn't title-fallback (Codex round 5) Codex round 5 P2: my round-4 normalization was too aggressive. It promoted `[[<current-ws>::REF]]` to ref-kind and let the regular ref branch handle it — including the title-fallback path that runs on ref miss. But the renderer's same-ws qualified branch (markdown.ts:472-481) does NOT title-fallback: a ref miss in that path returns the wiki-link verbatim (broken). Only the bare `[[REF]]` path (markdown.ts:513) falls through to title lookup. So my normalization could create ghost backlinks for source bodies like `[[ws::ISO-9001]]` when an item titled "ISO-9001" exists but no ISO collection — the renderer renders broken text, but the index would point at the title-matching item. Fix: handle same-ws qualified refs inline at the top of the loop, BEFORE the switch dispatches. Insert as ref-kind row (resolved or NULL) and `continue` past the switch. Bypasses the title-fallback path entirely, mirroring the renderer's behavior. Regression test in wiki_links_xws_test.go pairs same-ws qualified miss (must NOT title-fallback) with bare ref miss (SHOULD title-fallback) to lock the asymmetry in. PLAN-1593 / TASK-1597. |
||
|
xarmian
|
8e7d4040fd |
feat(backlinks): server-side reverse index for [[...]] (Phase 1) (#620)
* feat(backlinks): server-side reverse index for [[...]] wiki-links (Phase 1)
First phase of PLAN-1593. Today [[REF]] is parsed only at render time
on the client and there's no way to ask "who links to TASK-5?" without
a full-text scan. This change adds a materialized reverse index
(item_wiki_links) that's written every time an item's content changes
and exposes it via REST + CLI.
Phase 1 covers ref-form links only (`[[TASK-5]]` / `[[TASK-5|Display]]`).
Phase 2 (TASK-1595) will extend to titles + cross-workspace; Phase 3
(TASK-1596) adds the web UI panel + MCP action.
What lands here:
* Migrations 061 (SQLite) and 040 (Postgres) create item_wiki_links
with partial indexes on target_item_id, (target_workspace_id, target_ref),
and target_title — the schema accommodates all 5 wiki-link forms
up-front so Phase 2 doesn't ALTER.
* internal/links/extract.go is the canonical parser. It strips fenced
and inline code regions before extracting [[...]] occurrences, so
example refs in docs / code blocks don't pollute the index. Phase 1
emits only WikiLinkKindRef rows; title and workspace_ref kinds parse
successfully but are gated out until Phase 2.
* internal/store/wiki_links.go (replaceWikiLinks + GetBacklinks +
helpers) handles write-time bookkeeping and the read query. Resolution
to target_item_id happens at parse time inside the same transaction
as the items INSERT/UPDATE, so partial state never lands. Broken refs
(target_item_id IS NULL) intentionally persist — they feed a future
broken-links report.
* internal/store/wiki_links_backfill.go + cmd/pad/main.go hook the
idempotent backfill into server startup. Existing items get indexed
on first boot after the migration; subsequent boots are near-no-ops
via an EXISTS short-circuit.
* internal/store/items.go is amended in two places: tryCreateItem
always calls replaceWikiLinks (empty content → no-op DELETE), and
UpdateItemWithPreCheck re-parses whenever input.Content was supplied.
* internal/server/handlers_backlinks.go serves
`GET /api/v1/workspaces/{ws}/items/{itemSlug}/backlinks` with
visibility + guest-grant filtering on the source items.
* internal/cli/client.go adds GetBacklinks; cmd/pad/main.go adds the
`pad item backlinks <ref>` command (registered in groups.go).
Behavior decisions (per PLAN-1593):
- code blocks excluded (fenced + inline)
- self-links filtered at query time (kept in storage)
- repeated mentions stored as separate rows by position
- ordering: source updated_at DESC, position ASC
Tests:
- internal/links/extract_test.go: 26 sub-cases covering ref/title/
workspace-ref discrimination, code-block exclusion (fenced + inline +
unclosed fence), position-is-byte-offset (UTF-8 safety), and edge
inputs.
- internal/store/wiki_links_test.go: 8 integration tests covering the
create/update/delete/self-link/broken-ref/repeated/code-block
scenarios plus backfill idempotence.
All pass. `make check` clean (lint + go test + web build).
Refs: TASK-1594, PLAN-1593, IDEA-1577
* fix(backlinks): visibility-aware pagination + case-insensitive refs per Codex review (round 1)
Two fixes from Codex code review:
P1 — GetBacklinks now takes a visibleCollectionIDs []string argument
that's applied INSIDE the SQL WHERE clause. Previously the handler
fetched LIMIT raw rows and filtered visible ones in Go, so a
restricted user asking for limit=50 could receive an empty page even
when later visible backlinks existed. Pushing visibility into SQL
makes LIMIT/OFFSET count visible rows.
nil → no restriction (owners, editors, root tokens)
[] → see nothing (returns early, no SQL)
[..] → AND s.collection_id IN (?, ?, ...)
Item-level guest grants still apply post-fetch — they're rare enough
that the residual page shrink is acceptable and pushing them into SQL
would balloon the query.
P2 — refPattern now accepts mixed/lowercase refs and parseBody
canonicalizes the prefix to uppercase at the single chokepoint.
Previously the renderer accepted `[[task-5]]` as a real link (its
REF_PATTERN is case-insensitive) but the indexer's ^[A-Z]... pattern
silently dropped it — divergent parsing on the same input. Storage
shape is canonical uppercase so the (workspace, prefix, number)
lookup against collections.prefix (also uppercase) has one shape.
New helper: canonicalizeRef("task-5") → "TASK-5".
Regressions:
internal/links/extract_test.go
+ TestCanonicalizeRef — helper unit tests
+ TestExtractWikiLinks_RefVsTitleFallback updated to assert
mixed/lowercase parses-as-ref-and-uppercases
+ edge-case test renamed from "lowercase ref" to "number-led
not a ref" (lowercase IS a ref now per Codex P2)
internal/store/wiki_links_test.go
+ TestWikiLinks_MixedCaseRefIndexed — `[[task-5]]` produces a
backlink row whose target_ref is "TASK-5"
+ TestWikiLinks_VisibilityAwarePagination — three sub-cases:
nil → all 3, visible-only limit=2 → 2 visible rows (not 1 with
hidden one consuming a slot), empty → 0
All call sites updated (8 in tests + 1 in handler).
`make check` clean (lint + tests + web build).
Refs: TASK-1594, PLAN-1593
* fix(backlinks): SQL-level item-grant filter per Codex review (round 2)
Round 1 fixed pagination for collection-level visibility but Codex
round 2 correctly flagged the same class of bug at the item-grant
layer: `visibleCollectionIDs` returns the UNION (full grants ∪
collections containing granted items), and the handler then
filtered each row's item-level visibility in Go AFTER fetching —
letting hidden rows in a granted-item's collection consume LIMIT
slots.
The refactor moves the precise predicate into SQL. New shape:
type BacklinksVisibility struct {
Unrestricted bool // admin / full-access member
FullCollectionIDs []string // direct collection grants
GrantedItemIDs []string // item-level grants
}
// SQL predicate when Unrestricted=false:
// AND (s.collection_id IN (?...) OR s.id IN (?...))
This matches `guestResourceFilter` (which returns the precise
primitives), so the handler now passes them straight through and
drops the post-fetch filter loop entirely. Pagination is correct
for guests, restricted members, and unrestricted users alike.
New test:
TestWikiLinks_ItemGrantPagination — guest with item-grant on ONE
item in an otherwise-hidden collection sees exactly that one item;
hidden siblings in the same collection do NOT leak in, and limit=2
returns 1 row (not silently shrunken).
Other call sites updated:
- TestWikiLinks_VisibilityAwarePagination → uses
BacklinksVisibility{FullCollectionIDs: ...} and
BacklinksVisibility{} for the no-access case.
- 8 existing tests → BacklinksVisibility{Unrestricted: true}.
- handlers_backlinks.go → no longer calls visibleCollectionIDs;
uses guestResourceFilter exclusively and skips the Go-side filter.
Verification:
- make check clean
- All TestWikiLinks_* pass
Refs: TASK-1594, PLAN-1593
* fix(backlinks): scan EXISTS into bool not int for Postgres parity (Codex round 3)
`SELECT EXISTS(...)` returns boolean on Postgres but integer 0/1 on
SQLite. Scanning into `int` happened to work on SQLite (the modernc.org
driver coerces) but would fail on Postgres — silently disabling the
backfill short-circuit there and meaning upgraded Postgres installs
wouldn't populate backlinks for pre-existing content until each item
got edited.
Fix: scan into bool. Both database/sql drivers in use (modernc.org/
sqlite and lib/pq) coerce their native representation into Go's bool,
so this single shape works on both engines.
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): allow CommonMark 0-3 space indented fences per Codex (round 5)
Round 5 flagged two edge cases in the code-stripping pass:
1. Multi-backtick inline code (``see [[X]]``) — traced through the
parser; my permissive close-on-next-backtick logic already covers
it correctly (range = [opener-start, after-closer-run]). Added
a regression test to lock this in:
TestExtractWikiLinks_CodeBlocksExcluded /
"multi-backtick inline code excludes ref"
2. Indented fenced blocks — CommonMark allows 0-3 leading spaces of
indentation before a fence opener (4+ spaces makes it an indented
code block, a different construct). My fencedCodeRanges only
matched fences at column 0, so ` ```\n[[X]]\n```` ` would
render as code in the UI but leak a false backlink. Fixed both
fencedCodeRanges (opener) and findFenceCloser (closer) to skip
up to 3 leading spaces, with a hard cap at 4 (which would be
indented-code, not a fence). Regression test:
TestExtractWikiLinks_CodeBlocksExcluded /
"indented fenced block (CommonMark 0-3 spaces)"
Not addressed:
- Round-4 escape-body parity finding. extract.go mirrors
renderMarkdown's regex (web/src/lib/utils/markdown.ts:300), which
is the actual render-time link parser; wikiLinksToMarkdown's more
permissive escape grammar is editor-serializer-side and the
renderer can't even consume its escaped output. Indexing what the
user actually sees as a link is the correct invariant.
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): tilde fences + strict closer lines per Codex (round 6)
Two CommonMark conformance gaps in the code-block stripping pass:
1. Tilde-fenced code blocks (~~~) were ignored. marked() treats them
the same as backtick fences, so a [[REF]] inside a tilde block
would render as code in the UI but leak as a false backlink.
Fixed by parameterizing fenceChar across fencedCodeRanges and
findFenceCloser, with separate handling for the backtick-specific
"no backtick in info string" rule (CommonMark §4.5).
2. Closer-line strictness — CommonMark requires the closing fence
line to contain only the fence + optional trailing spaces. The
previous accept-any-fence-prefixed-line check would terminate
a still-open fence prematurely on a line like ```not-closed,
leaking later refs in the still-rendered code block.
Refs reside in 4 new sub-tests under TestExtractWikiLinks_CodeBlocksExcluded:
- tilde fence excludes refs inside
- tilde fence with language tag
- mixed fence types don't pair
- closer-line strictness — backticks plus other text is not a closer
- closer-line strictness — trailing spaces OK
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): inline code closer must match opener length per Codex (round 7)
CommonMark §6.1 requires an inline-code span opened with N backticks
to close on a run of EXACTLY N backticks. The previous "close on next
backtick run of any length" logic would prematurely end the excluded
range on a stray single backtick inside a ``...`` span, leaking any
[[REF]] in the latter half of the code text as a false backlink.
Concrete failure case:
``has ` inside [[X-1]] and more``
→ old: range [0, 7], [[X-1]] indexed (bug)
→ new: range [0, end-of-closer], [[X-1]] excluded (correct)
Fix: track the opener-run length and scan only for matching-length
closer runs. Wrong-length runs in between are code text.
Two new regression sub-cases under TestExtractWikiLinks_CodeBlocksExcluded:
- inline code closer matches opener length — the main case
- single-backtick span unaffected by adjacent multi-backtick run —
asserts the opposite direction (opener=1 doesn't close on ``)
Not addressed:
- Re-flagged round-4/round-7 escape-body parity finding. extract.go
intentionally mirrors renderMarkdown's regex (markdown.ts:300), not
wikiLinksToMarkdown's more permissive escape grammar (markdown.ts:461).
renderMarkdown is the actual link parser at display time; its regex
rejects escaped-`]` bodies, so any link with an escaped `]` in its
body is NOT shown as a clickable link in the UI. Indexing it would
produce phantom backlinks the user can't see. The wikiLinksToMarkdown
permissive grammar is paranoid serialization that the renderer can't
consume — that's a pre-existing inconsistency in the editor pipeline,
not a backlinks bug.
make check clean (lint + tests + web build).
Refs: TASK-1594, PLAN-1593
* fix(backlinks): rune-align snippet end-edge to keep UTF-8 valid (Codex round 8)
The previous snippetAround() trimmed `start` to a rune boundary (so
the leading edge of the snippet was always at a valid codepoint) but
left `end` as a raw +40-byte clamp. When that landed in the middle of
a multi-byte rune — common around emoji or accented text — the
resulting slice was invalid UTF-8 and the JSON encoder would emit
replacement characters in backlink snippets.
Fix: same forward-advance pattern at the end as at the start.
Continuation bytes (10xxxxxx) get skipped until we land on a leading
byte. Going forward keeps the snippet anchored slightly past the
match rather than slightly before it, which is a small UX win
(emoji or accented text right after the link survives intact).
Regression test:
TestWikiLinks_SnippetIsValidUTF8 — pads body with enough 4-byte
emoji on each side that the ±40-byte window cuts through one;
asserts utf8.ValidString on the resulting snippet.
make check clean (lint + tests + web build).
Refs: TASK-1594, PLAN-1593
* fix(backlinks): inline code spans cross newlines, break on blank lines (Codex round 9)
CommonMark §6.1: an inline-code span can cross single newlines but
terminates at a blank line (a line containing no chars or only
whitespace, which ends the enclosing paragraph). My previous scanner
broke at every newline, so multi-line spans like
`pre
[[INSIDE-1]]
post`
would treat the opener as unclosed and leak [[INSIDE-1]] as a false
backlink. Fixed by:
1. The newline branch in the closer scan now peeks ahead via the
new isBlankLineAt() helper. Same-paragraph newlines are
traversed; blank-line breaks terminate the span unmatched.
2. isBlankLineAt() treats any line with only space/tab as blank
(mirroring CommonMark's blank-line definition).
Three new regression sub-cases under TestExtractWikiLinks_CodeBlocksExcluded:
- inline code spans single newline (CommonMark §6.1)
- inline code breaks at blank line (paragraph boundary)
- inline code breaks at whitespace-only blank line
Trade-off: a truly-unclosed inline backtick now consumes from the
opener up to the next blank line instead of just the rest of the
line. False-positive on wiki-links in that span, but the surface
area is small (unclosed backticks are rare in published prose) and
matches the renderer's behavior.
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): accept escaped wiki-link bodies per editor grammar (Codex round 10)
After 3 rounds of disagreement, capitulating on the escape-body parity
finding. My position was technically correct for the CURRENT
renderMarkdown behavior (which uses [^\]]+ and can't parse escaped-
bracket bodies), but the editor's wikiLinksToMarkdown grammar at
markdown.ts:461 explicitly produces such bodies — making the
renderer's regex the inconsistent half of the pipeline, not mine.
Mirroring the editor's grammar in the extractor makes the index
forward-compatible: when the renderer eventually gets fixed, no
change here is needed. The cost is a few "phantom" rows in the
interim (indexed links the renderer doesn't currently display as
clickable), but those are harmless and aligned with author intent.
Changes:
- wikiLinkPattern now uses `\[\[((?:\\.|[^\]\\])+)\]\]` — mirrors
markdown.ts:461 verbatim.
- New splitOnUnescapedPipe() helper — scans for the first `|`
that isn't preceded by `\`. Mirrors splitWikiBody at
markdown.ts:664.
- New unescapeWikiBody() helper — undoes `\]`, `\|`, `\\` escapes
in display text and key. Mirrors unescapeWikiBody at markdown.ts:657.
- parseBody() now uses both helpers — split on unescaped `|`,
unescape both sides.
Regression coverage:
- TestExtractWikiLinks_EscapedBodyChars (5 sub-cases): escaped `]`,
escaped `|`, escaped `\`, non-escape backslash passes through,
Position still points at opening `[[` despite escapes.
- TestSplitOnUnescapedPipe + TestUnescapeWikiBody: direct unit
tests for the helpers (round-trip safety vs the editor's
escape/unescape pair).
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): preserve display text verbatim per Codex round 11 P3
The previous parseBody trimmed the display side of [[X|Display]] but
the WikiLinkRef.Display contract promises verbatim storage and the
renderer at markdown.ts doesn't trim either. Trimming would silently
diverge on padded display text like [[TASK-1| spaces ]] (renderer
keeps the spaces, extractor stripped them).
Fix: drop TrimSpace from the suffix half of the split. Keep trimming
the key/ref side because refPattern is anchored — a leading or
trailing space in the key would force the body to fall through to
the title kind even though the renderer resolves it as a ref.
Regression test:
TestExtractWikiLinks_EscapedBodyChars / "display text preserved
verbatim (no TrimSpace)"
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): distinguish empty display override from no-pipe per Codex round 12
[[REF|]] (explicit empty display) and [[REF]] (no display) are distinct
shapes in the editor: splitWikiBody returns displayOverride="" for the
former, null for the latter, and the renderer uses `displayOverride ??
title` (nullish coalescing, NOT empty-string fallback) so "" is
preserved. The previous extractor collapsed both into display_text=NULL,
violating verbatim-display preservation for the empty-string edge case.
Fix:
- WikiLinkRef gains a HasDisplay bool. parseBody sets HasDisplay=true
iff splitOnUnescapedPipe found a pipe; downstream uses HasDisplay
(not Display!="") to decide whether to persist the override.
- replaceWikiLinks in store: NullString.Valid is keyed off HasDisplay.
display_text='' for explicit empty, NULL for no override.
Regression coverage:
- internal/links/extract_test.go:
"explicit empty display override is distinguished from no pipe"
- internal/store/wiki_links_test.go:
TestWikiLinks_EmptyDisplayDistinct (two-source assert: NOT NULL
for [[REF|]], NULL for [[REF]])
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): pointer-typed DisplayText to preserve empty distinction over JSON (Codex round 13)
Round 12 added HasDisplay on the parser side and made the store
preserve display_text='' vs NULL on the DB row, but the wire model
collapsed the distinction at JSON-serialization time:
DisplayText string `json:"display_text,omitempty"`
`omitempty` drops empty strings, so [[REF|]] (empty override) and
[[REF]] (no override) serialized identically on the API and CLI JSON
output. The end-to-end goal of round 12 wasn't reached.
Fix: change DisplayText to *string. nil → no override (field omitted
from JSON via omitempty), pointer to "" → explicit empty override
(field present with empty value). The store's NullString.Valid drives
the assignment, so the SQL round-trip matches the JSON shape.
Knock-on: the CLI's `pad item backlinks` now dereferences the pointer
and prints both populated and empty overrides ("displayed as: ").
Regression coverage:
- TestWikiLinks_EmptyDisplayDistinct extended to assert
withBL.DisplayText is non-nil-pointing-at-"" and noBL.DisplayText
is nil after a GetBacklinks round-trip.
make check clean.
Refs: TASK-1594, PLAN-1593
|
||
|
xarmian
|
0c5ec04fac |
feat(admin): extend user list with aggregations + sort/filter (TASK-1544) (#599)
* feat(admin): extend user list with aggregations + sort/filter (TASK-1544)
GET /admin/users now returns per-user workspace_count, storage_bytes,
last_write_at, and a computed status pill (disabled / no-workspace /
inactive / active, with documented precedence). Adds sort and filter
knobs so the table can scale beyond the existing fixed offset/limit.
Store layer:
- AdminUserSearchParams gains Role, Sort, Order, ActiveWithinDays,
HasWorkspaces, Disabled. Pointer types where tri-state ("no filter"
vs. "filter to false") matters.
- AdminUserListEntry wraps models.User with WorkspaceCount, StorageBytes,
Status — returned in AdminUserSearchResult.Users.
- SearchUsers SQL rewritten: LEFT JOIN against grouped subqueries so
one user owning N workspaces with M attachments each still produces
exactly one row (no aggregation explosion). Both subqueries filter
deleted_at IS NULL to match WorkspaceStorageUsage's existing
definition. Allow-listed sort clause prevents injection.
- computeAdminUserStatus exported for unit tests; precedence locked in
by TestComputeAdminUserStatus.
- TestSearchUsersAggregations covers workspace_count + storage_bytes +
status across a three-user fixture and each new filter/sort knob.
Model + scanner:
- models.User gains LastWriteAt. userColumns + scanUser updated; the
legacy callers (GetUser, ListUsers, etc.) inherit the new field for
free via the shared scanner.
Handler:
- handleAdminListUsers accepts the new params: role, disabled,
has_workspaces, active_within_days, sort, order. Tri-state bools
only fire when the query param is present. Response now embeds
workspace_count / storage_bytes / last_write_at / status.
Part of PLAN-1542. Frontend consumption lands in T1548 (cheap columns)
and T1549 (sort/filter UI).
* fix: address Codex review on TASK-1544
- Tri-state bool parsing in handler now uses strconv.ParseBool — accepts
the canonical truthy/falsy variants ("True"/"TRUE"/"t"/"1" and the
parallel falses), and silently ignores garbage values rather than
treating them as false. Closes the "disabled=TRUE silently means
enabled-only" surprise.
- SearchUsers count query no longer joins the storage aggregation when
HasWorkspaces isn't an active filter. The page query still needs both
joins (the row carries the data), but a typical "give me a count"
call no longer scans every live attachment. The workspace_count join
remains conditional on HasWorkspaces filtering.
Status threshold (>30d vs >=30d): the documented spec and impl both say
">30d" — no change.
|
||
|
xarmian
|
905baaa010 |
feat(oauth): backfill session.Extra into oauth_connections + switch read path (TASK-1522) (#583)
* feat(oauth): backfill session.Extra into oauth_connections + switch read path (TASK-1522) Phase C1 for PLAN-1519. Seeds existing OAuth grant chains into the new connection tables (Phase A) and switches /console/connected-apps to read from them, retiring the session.Extra parse on the read path. Backfill (internal/store/oauth_connections_backfill.go) - Walks oauth_access_tokens + oauth_refresh_tokens to find every distinct request_id chain (including refresh-only chains). - Picks the newest token row per chain — its session.Extra drives the seeded shape, so a chain whose user re-scoped recently reflects the latest decision. - Maps session.Extra shapes to the new tables per IDEA-1517 §2: no key → all_current=1; ["*"] → all_current=1; explicit slugs → all_current=0 + one join row per slug (added_by='user'). - Resolves slugs → workspace IDs; unresolved slugs (deleted / renamed workspace) are counted + logged at WARN, not fatal. - Idempotent on every INSERT (OR IGNORE / ON CONFLICT DO NOTHING) so re-running on every startup is a cheap no-op once stable. - Returns a BackfillOAuthConnectionsResult so the startup log reports chains_seen / connections_created / workspaces_added / unresolved_slugs — operators see fresh work and notice drift. Read-path rewrite (internal/store/connected_apps.go) - ListUserOAuthConnections projects AllowedWorkspaces from GetOAuthConnectionAccess (oauth_connection_workspaces JOIN workspaces) instead of parsing session.Extra strings. - Hydrates Name + MayCreate + AllCurrent + IncludeFuture from oauth_connections so Phase D's mutation UI has them. - Defensive fallback for chains without an oauth_connections row (any leftover the backfill missed): treats as legacy "any workspace, default-on flags" so the connection still renders. Backfill at startup keeps this branch unreachable in production. - Retires parseAllowedWorkspacesFromSession; the new extractAllowedWorkspacesFromSessionExtra helper in oauth_connections_backfill.go is the only consumer of the session.Extra shape on the store side. Model (internal/models/connected_apps.go) - Adds Name / MayCreateWorkspaces / AllCurrentWorkspaces / IncludeFutureWorkspaces. AllowedWorkspaces semantics stay stable (nil = "any"; explicit slugs = chip list) so the existing DTO + frontend continue working unchanged. Phase D exposes the new fields on the wire. Startup wiring (cmd/pad/main.go) - After srv.SetOAuthServer / SetClaimSecret, run the backfill once. Non-fatal on error (partial state is consistent and the next run completes). Quiet at the Debug level on steady-state re-runs; INFO when fresh work landed. Tests - 8 BackfillOAuthConnections cases: empty DB, pre-TASK-952 (no key), wildcard, explicit list, mixed resolvable/unresolved slugs, multi-row chain newest-row-wins, refresh-only chain, idempotent re-run (verified via post-run row count). - TestExtractAllowedWorkspacesFromSessionExtra replaces the retired parseAllowedWorkspacesFromSession test — covers all three IDEA-1517 §2 input shapes + malformed/non-array defensive cases. - TestListUserOAuthConnections_DeduplicatesChain + TestHandleListConnectedApps_DTOShapeAndAuditEnrichment updated to call BackfillOAuthConnections (the production startup hook) before asserting on AllowedWorkspaces — mirrors the real-world flow now that the read path no longer parses session.Extra inline. Parent: PLAN-1519. * fix(oauth): backfill counters reflect actual new rows per Codex review (round 1) PR #583 Codex review round 1 flagged that the backfill counters over-report on steady-state restarts: - wasFreshlyInserted compared updated_at vs created_at — true for every untouched existing row, so every restart counted every pre-existing connection as "created." - slugsAdded++ ran after AddConnectionWorkspace regardless of whether the INSERT OR IGNORE / ON CONFLICT DO NOTHING hit an existing row. Net effect: startup logs "backfill complete" with non-zero counts on every restart instead of the intended quiet "no-op" path — making real fresh work indistinguishable from steady-state. Fix: probe existence BEFORE the insert on both sides. - backfillOneChain reads GetOAuthConnection first; only sets created=true and runs insertOAuthConnectionIfAbsent on a miss. - Per-slug: IsConnectionWorkspaceAllowed pre-check; skip + don't increment when the row already exists. Two cheap PK / indexed lookups per chain. Pre-Phase-C deployments have small chain counts so the added cost is well below the scan already running. Removed the now-unused wasFreshlyInserted helper. Added an assertion in TestBackfillOAuthConnections_Idempotent that both ConnectionsCreated and WorkspacesAdded report 0 on the second run — the regression guard for this exact finding. Parent: PLAN-1519. * fix(oauth): backfill skips slug re-seed on existing rows per Codex review (round 2) PR #583 round 2 caught that the round-1 fix protected the parent oauth_connections row from re-seed but left the join table mutable from stale session.Extra: When a user removes a workspace from their connection's allow-list via Phase D's mutation UI (RemoveConnectionWorkspace), the next server restart would re-run the backfill, find the parent row intact, and re-INSERT the removed slug from the original session.Extra. The user's removal would silently revert every restart. Fix: backfill is a one-shot seed. Once the parent row exists, the new tables are authoritative — legacy session.Extra is frozen reference data, not a reconciliation source. The slug loop only runs when we just inserted a fresh parent row. Added TestBackfillOAuthConnections_DoesNotResurrectRemovedWorkspace as the regression guard: seeds two slugs, removes one, runs backfill again, asserts the removed slug stays gone and the kept slug is untouched. Parent: PLAN-1519. * fix(oauth): atomic per-chain backfill transaction per Codex review (round 3) PR #583 round 3 caught that round 2's "only seed slugs on fresh parent" gate introduced a permanent-partial-state risk: if the process crashes (or AddConnectionWorkspace errors) between inserting the parent row and finishing the slug loop, the next backfill sees created=false, short-circuits the slug seeding, and leaves the connection permanently scoped to a partial allow-list. Fix: per-chain transaction. Parent insert + every slug insert land in one BEGIN/COMMIT pair; any mid-loop failure rolls everything back. The next backfill then sees the chain as un-seeded and retries from scratch — preserving both round 2's "no-resurrection of user-removed slugs" (existence probe inside the tx) and round 3's "no permanent partial seed" (atomic commit). Scope: per-chain (small tx), not whole-backfill. The original no-transaction rationale was about lock-hold duration across thousands of chains; that doesn't apply at chain granularity (one parent + a handful of join rows = sub-millisecond hold). Removed the now-unused insertOAuthConnectionIfAbsent helper; the INSERTs live inline within the transaction. Added TestBackfillOAuthConnections_AtomicOnMidLoopFailure as the regression guard: forces a mid-loop INSERT failure via a duplicate slug in session.Extra (which violates the join table's PK on the second insert), asserts the parent row rolled back, then runs a clean retry and verifies full seed completion. Parent: PLAN-1519. * fix(oauth): surface store errors from backfill + list path per Codex review (round 4) PR #583 round 4 caught two silent-fallthrough paths that could leak partial/incorrect state instead of failing loudly: 1. Backfill slug loop: GetWorkspaceBySlug errors were treated the same as "workspace not found" — both incremented slugsMissed and continued. A real I/O error mid-loop would commit a partial allow-list, and the next backfill's parent-exists short-circuit would make that partial scope permanent. Fix: distinguish (nil, nil) "not found" from (nil, err) "real failure" — return the error so the per-chain transaction rolls back and the next run retries cleanly. 2. ListUserOAuthConnections hydration: GetOAuthConnectionAccess and GetOAuthConnection errors collapsed into the "no oauth_connections row" defensive-fallback branch, returning the legacy "any workspace, default-on flags" shape. On a real store failure that silently broadens a user's scope — e.g. a connection the user explicitly removed a slug from would render as "Any workspace" until the store recovered. Fix: surface store errors from both calls; the defensive fallback path is now exclusively for HasConnection=false, not for error masking. Both findings tighten the failure mode from "silently emit broadened/partial state" to "surface the error so retries happen against accurate data." Existing tests cover the happy paths; the failure paths are exercised by I/O errors against the same store interfaces (no new test added — the change is "return err instead of swallow it" and the assertion of NOT swallowing is the diff itself). Parent: PLAN-1519. |
||
|
xarmian
|
e59d3904c9 |
feat(server): refuse to mark item terminal while it has open children (IDEA-1494) (#571)
* feat(server): refuse to mark item terminal while it has open children (IDEA-1494)
Server-side guard inside handleUpdateItem that rejects a non-terminal →
terminal done-field transition when the item still has at least one
non-terminal child. Returns HTTP 409 with code=open_children plus a
structured details payload listing each blocking child's
{ref, title, status, collection_slug} so MCP-driven agents can
self-recover (ship the listed children, then retry) and the CLI can
render the same list verbatim.
Escape hatch: `--force` on `pad item update` / `pad item bulk-update`
and `force: true` on the MCP pad_item.action: update / bulk-update
inputs both forward into the same ItemUpdate.Force transport field
the handler consumes before any store mutation.
Trigger conditions are tight: the PATCH must change the done-field key
(resolved via TerminalValuesForDoneField against the parent's schema +
settings) AND the new value must be terminal AND the current value
must NOT already be terminal. Terminal → terminal and no-op terminal
transitions bypass the guard; only entering the terminal set is gated.
Per-child evaluation uses the child's own collection schema so
hierarchical workspaces with custom typed collections work without
extra plumbing.
Tests cover: rejection with one open child (with mutation-safety
assertion on the parent), no children, all-terminal children, --force
override, no-op terminal → terminal, terminal → terminal,
non-terminal → non-terminal, custom collection terminal_options
honored, and a parent task (not a plan) — IDEA-1494 optional extra #3.
MCP coverage asserts --force round-trips through both ExecDispatcher
and HTTPHandlerDispatcher and is omitted when force=false.
* fix(server): open-children guard round 2 — visibility, MCP pass-through, TOCTOU (IDEA-1494)
Three Codex round-1 issues, each fixed with the recommended shape:
P1 — visibility leak. The 409 response previously listed every blocking
child by ref/title/status, including children in collections the caller
couldn't see. The INVARIANT still evaluates against ALL children (it's a
data-integrity gate — a restricted user must not be able to close a
parent whose blockers they can't see), but the response payload now
filters to caller-visible children only. Hidden blockers surface as a
new `details.hidden_blocker_count` field plus an alternate human message
when every blocker is hidden ("blocked by N open children you don't
have access to"). Mirrors the visibility helpers (`visibleCollectionIDs`
+ `isItemVisibleToGuest`) used by the per-parent progress endpoint so
the two paths can't drift.
P2 — MCP code/details pass-through. The HTTP classifier was collapsing
409 into the generic `conflict` code and dropping `details`; the stdio
classifier was matching the human "cannot " message against the
validation regex and surfacing `validation_failed`. Both now surface
`open_children` with the structured details intact:
- HTTP: classifyHTTPStatusKind's 409 branch extracts the upstream
code; any non-empty, non-"conflict" code is passed through with
its `details` RawMessage. Generalizes beyond open_children — any
future structured 409 from a handler gets the same treatment.
- Stdio: the CLI writes a `pad-error: {json}\n` marker line on
stderr before the human-readable block (single source of truth for
both views), and classifyExecError detects the marker and lifts
the envelope verbatim. Marker is duplicated as a const between
internal/cli and internal/mcp to avoid pulling the cli package
into the classifier just for one string.
A new ErrOpenChildren error code constant + `Details json.RawMessage`
field on ErrorPayload back the wire shape.
P2 — TOCTOU. The guard previously ran in the handler before the store
transaction began; a concurrent child insert / child status flip could
slip between the children-list read and the parent's UPDATE. Fix:
- New `Store.UpdateItemWithPreCheck(id, input, precheck)` runs the
caller's invariant check inside the same tx, after acquiring the
workspace seq lock AND a new parent-children advisory lock keyed
on the parent ID. UpdateItem is now a thin wrapper passing nil.
- Every UpdateItem unconditionally acquires the parent-children
advisory lock for its own parent (if any) AND for itself-as-parent,
in a fixed order (parent first) so two updaters touching the same
parent always grab that key before the more-specific one — no
AB/BA deadlock.
- New `GetChildItemsTx` reads via the caller's tx; on Postgres the
advisory lock provides the snapshot guarantee (DISTINCT precludes
`FOR UPDATE`), on SQLite the global BEGIN IMMEDIATE write lock
serializes all writers.
- Handler now passes a precheck closure into UpdateItemWithPreCheck
at all three call sites (collab-snapshot path, applier-direct-write
path, main path). The guard's openChildrenGuardError sentinel is
unwrapped after each call so the 409 surfaces cleanly.
Tests:
- TestOpenChildrenGuard_VisibilitySanitizesPayload — restricted
editor sees parent + visible child, hidden child contributes to
hidden_blocker_count, no leak of ref/title/slug.
- TestOpenChildrenGuard_AllBlockersHiddenSurfaceGenericMessage —
open_children=[], hidden_blocker_count>0, message mentions "you
don't have access to."
- TestOpenChildrenGuard_TOCTOURace — 8 iterations of a child-flip
racing a parent-terminal update; asserts the forbidden outcome
(parent=completed AND child=open) never occurs.
- TestClassifyHTTPStatus_OpenChildrenPreservesCodeAndDetails +
inverse generic-409 test.
- TestClassifyExecError_OpenChildrenMarkerLiftsStructuredPayload +
no-marker-falls-through inverse.
* fix(server): open-children guard round 3 — 7 Codex findings closed (IDEA-1494)
P1 — visibility fail-closed. The handler was swallowing
visibleCollectionIDs errors, leaving visIDs==nil which the guard
treats as unrestricted, leaking hidden-child metadata. Now surfaces
the error as 500 BEFORE installing the precheck. Test:
TestOpenChildrenGuard_VisibilityLookupErrorFailsClosed closes the
store DB and asserts no 409+children leak.
P1 — link mutations acquire the advisory lock. SetParentLink,
ClearParentLink, CreateItemLink (when link_type ∈ childLinkTypes via
new isChildLinkType helper), DeleteItemLink (same condition), and
RestoreItem now take `pad:parent-children:<id>` in canonical sorted
order via new AcquireParentChildrenLocks helper. SetParentLink locks
BOTH old and new parents (re-parenting case). Race test
TestOpenChildrenGuard_LinkMutationRace asserts the forbidden
"link-committed-before-parent-flip AND parent flip succeeded" never
occurs by comparing link.created_at to parent.updated_at. Documented
semantics: status-wins + link-after-commit is legal under the
invariant "no open children EXIST AT THE MOMENT of transition" —
the post-condition variant ("no open child may EVER attach to a
terminal parent") is intentionally deferred.
P1 — MoveItem bypass closed. New MoveItemWithPreCheck mirrors
UpdateItemWithPreCheck — acquires workspace seq lock + parent-children
locks, re-reads in tx, runs caller precheck. handleMoveItem builds
the same guard closure using the DESTINATION schema for done-field
resolution (conservative — honors the schema the item moves INTO).
CLI gains `pad item move --force`, client gains MoveItemWithForce
that appends `?force=true` to the move endpoint. MCP catalog +
mapItemMove forward `force` through the route mapper. Tests:
TestOpenChildrenGuard_MoveItem_RejectsTerminalWithOpenChildren and
…_ForceOverrides.
P2 — pre-tx field-read TOCTOU. UpdateItemWithPreCheck and
MoveItemWithPreCheck now re-read the item via new getItemTx INSIDE
the tx (after locks) and pass that fresh snapshot to the precheck
closure; the precheck classifies the transition against the in-tx
view, not the handler-side pre-tx capture. Handler precheck closure
swaps `currentFieldsJS` from the in-tx snapshot. Test:
TestOpenChildrenGuard_PrecheckReadsInTxSnapshot stages a between-load
status mutation and asserts the precheck observes the post-mutation
fields.
P2 — bulk-update carries structured errors. cmd/pad/main.go's
updateFailure struct extended with Code + Details
(json.RawMessage). When client.UpdateItem returns *cli.APIError, the
row preserves the structured envelope. Human-text output also
renders the open-children list inline. Chose JSON-envelope route
over per-row stderr markers because bulk-update already produces a
structured envelope and ExecDispatcher returns stdout verbatim on
exit-0 — no classifier change needed. Test:
TestBulkUpdateStructuredFailuresCarryOpenChildrenDetails confirms
the wire shape the CLI lifts.
P3 — marker hardening. Marker bumped to versioned form
`pad-structured-error/v1:` (was `pad-error:`). cli.StructuredErrorMarker
+ mcp.structuredErrorMarker kept in lockstep with cross-references.
mcp.allowedStructuredErrorCodes whitelists known codes (currently
just open_children); unknown codes fall back to regex classification.
Marker must start the line after whitespace trim (embedded markers
ignored). Last-marker-wins to defeat pre-emption attacks. Tests:
TestClassifyExecError_{UnknownStructuredCode,OldMarkerVersion,
MarkerEmbeddedMidLine,LastMarker}.
P3 — soft-deleted collection schemas honored. New GetCollectionAnyState
mirrors childrenDoneFiltersForParent's inclusion rule; guard uses it
so a child still attached to a soft-deleted collection is evaluated
against ITS schema (custom terminal_options) instead of the default-
status fallback (which would mis-classify and false-block). Test:
TestOpenChildrenGuard_SoftDeletedCollectionSchemaHonored seeds a
custom collection, soft-deletes it while a child remains, and
asserts the terminal status is correctly recognized.
Comprehensive store-mutation audit results recorded in the PR
description (every method touching items.fields / items.collection_id
or item_links).
* fix(server): open-children guard round 4 — multi-parent locks, enum parity, PATCH atomicity (IDEA-1494)
Four Codex round-3 (blast-radius lens) findings, each fixed with the
recommended shape.
P1 — multi-parent lock set. acquireParentChildrenLocksForUpdate and
RestoreItem previously used `LIMIT 1` against item_links, so a child
with BOTH a `parent` link to P1 AND an `implements` link to P2 only
locked one of them. The other parent's open-children precheck could
race against the child's status flip and miss it.
Fix: new listParentChildLockKeys helper runs the same query
GetChildItems' inclusion rule uses (childLinkTypes), returns ALL
distinct parent target_ids, and feeds them into the canonical
multi-lock helper. Both UpdateItemWithPreCheck and RestoreItem now
acquire locks on {self} ∪ {all-parents-via-childLinkTypes}. Test:
TestOpenChildrenGuard_MultiParentChildLocksAll races a child status
flip against terminal-updates on both parents simultaneously.
P2 — lock-order asymmetry. The pre-fix codebase had multiple lock-
acquisition shapes: parent-then-self in acquireParentChildrenLocksForUpdate,
single-key in RestoreItem / CreateItemLink / DeleteItemLink /
ClearParentLink, and a sorted multi-key in SetParentLink. Two
concurrent callers using different ad-hoc orderings could AB/BA
deadlock.
Fix: removed the per-call-site AcquireParentChildrenLock helper
entirely. Every site now goes through AcquireParentChildrenLocks
(the canonical sorted multi-lock helper) — including ones that need
only one ID (the variadic call still sorts a one-element slice).
The helper's doc comment explicitly states the contract: "Ad-hoc
single-key acquisition outside this helper is FORBIDDEN — two call
sites taking distinct keys in different orders WILL deadlock."
Test: TestOpenChildrenGuard_NoDeadlockUnderReverseOrderConcurrency
runs reverse-order re-parents with a 5-second timeout; assertion
fails on hang.
P2 — HTTP/stdio code-surface parity. Round 2's HTTP pass-through
("any non-conflict upstream code") silently widened the ErrorCode
enum beyond stdio's allow-list (`open_children` only). Agents saw
different code surfaces depending on which dispatcher delivered
the response.
Fix: HTTP 409 branch in classifyHTTPStatusKind now consults the
same allowedStructuredErrorCodes whitelist stdio does. Codes
outside the set collapse to ErrConflict (no details), matching
what stdio does for an unknown-code structured marker. Doc on
allowedStructuredErrorCodes updated to make the dual-consumer
contract explicit: "Adding a new structured code is a TWO-WAY
change." Tests:
TestClassifyHTTPStatus_UnknownConflictCodeFallsBackToErrConflict
and TestStructuredErrorCodeParityAcrossTransports.
P3 — PATCH atomicity. A combined PATCH with `parent` + `status=terminal`
on an item with open children used to commit the parent-link change
INLINE (before the guard ran) and then reject the field write.
Caller saw 409 but the parent had already moved.
Fix: parent-link mutation is now DEFERRED — captured into outer-
scope vars during fields validation, executed AFTER
UpdateItemWithPreCheck succeeds. A guard rejection returns before
the link write block, so on rejection the link is untouched.
Documented choice: "reorder, don't tx-wrap" — wrapping SetParentLink
into the same store tx would require threading a *sql.Tx through
the SetParentLink API (which is also called from the
handler_item_links path); reordering is the smaller surgery and
gives the correct outcome on the failure direction. A residual
window remains in the OTHER direction (field write commits, link
write fails) — not made worse by the reorder, and called out
inline for a future tx-wrap pass.
Test: TestOpenChildrenGuard_PatchAtomicRejectionPreservesParentLink
sets up target → oldParent → openChild, sends PATCH {parent=newParent,
status=completed}, asserts 409 AND target.parent_link still points
at oldParent.
* fix(server): open-children guard — emit details.open_children as [] not null on hidden-only rejection (IDEA-1494)
|
||
|
xarmian
|
ec71903be7 |
feat(store): JSONB NOT NULL hardening on items/views + handler shape validation (IDEA-1486+1488) (#566)
* feat(store): NOT NULL hardening on items.fields/tags + views.config (IDEA-1486) Paired ship of IDEA-1486 (sibling-table JSONB NOT NULL hardening) and IDEA-1488 (handler-layer shape validation for ViewUpdate/CollectionUpdate). Generalizes the IDEA-1484 / collections.settings precedent (PR #562) to the remaining nullable JSON columns and closes the shape-validation gap that NOT NULL alone doesn't cover. Schema layer (IDEA-1486 floor): - migrations/056_items_jsonb_not_null.sql: rebuild items with fields TEXT NOT NULL DEFAULT '{}' and tags TEXT NOT NULL DEFAULT '[]', preserving all 7 indexes, recreating the 3 items_fts triggers, and rebuilding the FTS5 index. Foreign-keys-off / on bookends are lifted outside the IDEA-1485 atomic-tx wrapper. - migrations/057_views_config_not_null.sql: rebuild views with config TEXT NOT NULL DEFAULT '{}'. - pgmigrations/035 + 036: SET NOT NULL + SET DEFAULT on the three JSONB columns. Split per-table to mirror the SQLite per-table file granularity. Store layer (IDEA-1486 floor): - items.go UpdateItem and views.go UpdateView normalize "" -> "{}" / "[]" before writing. Same boundary pattern as CreateItem and the IDEA-1484 precedent at collections.go:248. - export.go ImportWorkspace coerces empty-string AND malformed JSON at import time on items.fields, items.tags, and collections.settings. Malformed input is coerce-and-log via slog.Warn (length only, never raw value) so legacy bundles don't fail-stop on one bad row. - remapFieldIDs early-returns "{}" on empty input so the second-pass UPDATE can't write "" verbatim. - Migrated the existing fmt.Printf at export.go:329 to slog.Warn for consistency. Handler layer (IDEA-1488 ceiling): - ViewCreate / ViewUpdate UnmarshalJSON via flexJSONToString with new ErrInvalidConfigType sentinel. - CollectionCreate / CollectionUpdate UnmarshalJSON with new ErrInvalidSettingsType sentinel. - handlers_views.go and handlers_collections.go surface both sentinels as 400 with the domain-level message (mirrors the BUG-1144 precedent at handlers_items.go:641). Tests: - internal/store/items_views_jsonb_test.go: store-coercion + import coercion + log-and-coerce-on-malformed + SQLite schema introspection (7 indexes + 3 FTS triggers + items_fts virtual table survival) + Postgres NOT NULL enforcement + migration re-apply idempotency + item_links round-trip after rebuild. - internal/server/handlers_views_collections_jsonb_test.go: PATCH/POST flexible-shape coverage for views.config and collections.settings, including domain-level 400 message assertions that the response does not leak Go unmarshal internals. Refs: IDEA-1486, IDEA-1488, IDEA-1484 (precedent), IDEA-1485 (substrate). * fix(store,models): codex R1 follow-ups for IDEA-1486 / IDEA-1488 Three concrete defects surfaced by codex R1 against the initial paired ship. All three close holes that defeated parts of the original contract. P1.1: migration 056 missed the playbook invocation_slug unique index. - migrations/056_items_jsonb_not_null.sql: recreate the partial UNIQUE index idx_items_invocation_slug_per_collection from migration 054 verbatim after the other 7 indexes. Without it, the application-layer pre-check in handlers_items.go:checkUniqueFields would be a TOCTOU race with no DB-level guard — the original index that 054 explicitly added as the actual uniqueness backstop would be silently dropped during the items rebuild. - items_views_jsonb_test.go: the schema-introspection test now asserts 8 indexes, not 7. Verified via `grep -rn "ON items(" migrations/` that no other items-touching indexes were missed. P1.2: flexJSONToString didn't validate inner content of JSON-encoded strings. Pre-fix, `{"config": "[]"}` / `{"settings": "not json"}` / `{"fields": "[]"}` / `{"tags": "{}"}` slipped past the shape validators because the `case '"'` branch unmarshalled the envelope and returned the inner string verbatim — bypassing the whole point of IDEA-1488. - models/item.go: after unmarshalling the JSON-encoded string, validate that the trimmed inner content's first byte matches expectedStart ('{' / '[') AND parses as JSON. Empty inner strings still pass through to the store-layer empty-string coercion (IDEA-1486 floor), so legacy "" → default normalization is preserved. - The pre-existing ItemUpdate fields/tags path inherits the same tightening because it routes through this helper — covered by new test file handlers_items_jsonb_inner_shape_test.go. - Parallel handler tests for views.config and collections.settings added to handlers_views_collections_jsonb_test.go. P2: coerceJSONForImport accepted JSON null as well-formed. - store/export.go: json.Unmarshal("null", &m) returns err=nil with m staying nil; the prior code returned the raw "null" string verbatim, which lands as JSONB null on Postgres (satisfies NOT NULL since SQL NULL ≠ JSONB null) or text "null" on SQLite. The non-nil check on the unmarshalled value routes JSON null to the existing log-and-coerce path with the rest of the malformed shapes. - items_views_jsonb_test.go: extended import test with an item carrying fields=null / tags=null; expects both coerced to "{}" / "[]" and the structured slog.Warn emitted. Verified: make test (SQLite) and the full ./... suite against the existing port-5445 Postgres container both pass cleanly. Refs: IDEA-1486, IDEA-1488, codex R1 review. * fix(store,server): codex R2 follow-ups for IDEA-1486 / IDEA-1488 Two defects surfaced by codex R2. P1 is a real ship-breaker; P2 closes a parity gap that R1 missed. P1: migration backfill normalized only SQL NULL, not malformed/wrong- shape JSON. The four new migrations originally wrote `WHERE x IS NULL`. Rows with fields = '' / 'null' / '[]' / 'not json' all survived the filter, then violated the post-migration NOT NULL+shape contract. Concrete ship- breaker on SQLite: 056 recreates the partial UNIQUE index on json_extract(fields, '$.invocation_slug') from migration 054, and json_extract errors on rows whose fields fails json_valid — a single bad row breaks CREATE INDEX mid-migration. Toggle-verified: with the NULL-only WHERE, the new SQLite test fails at exactly that CREATE INDEX with "SQL logic error: malformed JSON (1)". Widened the backfill clauses: - migrations/056: UPDATE items WHERE fields IS NULL OR json_valid(fields)=0 OR json_type(fields)!='object' (same trio for tags with 'array'). - migrations/057: same trio for views.config. - pgmigrations/035: WHERE fields IS NULL OR jsonb_typeof(fields)!='object'. JSONB rejects invalid JSON on write so the json_valid leg isn't needed on Postgres; only the shape check matters. - pgmigrations/036: same shape check on views.config. Regression tests in internal/store/items_views_jsonb_test.go: - TestItemsViewsJSONB_SQLiteBackfillRepairsMalformedShapes: applies migrations through 053 (skipping 054 which would itself error on malformed rows), seeds every observable shape pathology — SQL NULL, empty string, JSON null literal, wrong-shape JSON, non-JSON garbage — then applies 055/056/057. Asserts every malformed row is repaired AND the partial UNIQUE index actually fires on duplicate invocation_slug post-rebuild (proving the CREATE INDEX path executed end-to-end). - TestItemsViewsJSONB_PostgresBackfillRepairsMalformedShapes: parallel Postgres coverage; seeds JSONB null / array / primitive via direct ::jsonb cast and asserts the widened WHERE clause repairs each. P2: handleCreateItem didn't unwrap ErrInvalidFieldsType/ErrInvalidTagsType. R1's flexJSONToString tightening propagated the sentinels through every UnmarshalJSON path, but handleCreateItem (POST /items) still returned 'invalid JSON: <wrapped>' from decodeJSON. PATCH and the view/collection POST/PATCH handlers already unwrapped — POST was the outlier. - internal/server/handlers_items.go: mirror the PATCH-side errors.Is handling at the POST path. Brief, three-line diff. - handlers_items_jsonb_inner_shape_test.go: new TestCreateItem_ JSONEncodedStringInnerShapeValidated covers POST with fields=`[]`, fields=42, tags=`{}`, tags={"x":1}, plus a valid positive control. Asserts no "invalid JSON:" wrapper and presence of the sentinel message verbatim. Backfill-pattern audit (codex R2's grep prompt): only 055 / pg-034 (collections.settings, already shipped) exhibits the same NULL-only WHERE gap. Per the brief: NOT touched — retroactive repair belongs to a separate IDEA. Other NULL-only backfills (043/pg-023's oauth_providers, 044/pg-024's expires_at) handle their respective shapes correctly or aren't JSON columns. Verified: make test (SQLite) clean. Full ./... suite against the existing port-5445 Postgres container clean (one unrelated flake in internal/collab passed on rerun). Refs: IDEA-1486, IDEA-1488, codex R2 review. |
||
|
xarmian
|
438cb6180a |
fix(mcp): flexible JSON shapes on item create + clearer field surface (BUG-1431, BUG-1432) (#547)
* fix(mcp): flexible JSON shapes on item create + clearer field surface (BUG-1431, BUG-1432)
BUG-1432 root cause (real): models.ItemCreate.Tags is a Go string, so
the default unmarshaler rejected the natural JSON-array shape every
agent sends (`tags: ["foo","bar"]` → "cannot unmarshal array into Go
struct field ItemCreate.tags of type string", HTTP 400). On Postgres
the alternative — passing `tags: "foo,bar"` per the catalog's old
"Comma-separated tags" description — landed as a non-JSON value in
the JSONB column and surfaced as a generic HTTP 500. SQLite's TEXT
column silently accepted the corrupt value, which is why local repros
didn't show it.
Codex's independent investigation called out the asymmetry: ItemUpdate
already had a flexible UnmarshalJSON for `fields`/`tags` per BUG-1144,
but ItemCreate didn't. This PR mirrors that flexibility on the create
path and aligns the MCP surface description with reality.
BUG-1431 root cause (real, not the misdiagnosis the agent reported):
the dispatcher's `parseFieldKVP` only accepted the CLI-style array-of-
"key=value" shape, rejecting the JSON-native `field: {key: value}` map
shape with "expected array or string, got map[string]interface {}".
Agents naturally try the map shape and got a non-actionable error;
that drove the BUG-1409 agent to mis-blame status placement. Empirical
repro confirmed that `status` actually works in both top-level AND
inside-fields positions today (Tests 1, 4 in the investigation); the
real surface problem was the missing map shape on `field`.
Changes:
- internal/models/item.go: add UnmarshalJSON to ItemCreate mirroring
ItemUpdate's BUG-1144 pattern. Accepts `fields` as object or
JSON-encoded string; `tags` as array or JSON-encoded string; either
field absent / null leaves Go zero value. Wrong shapes surface
ErrInvalidFieldsType / ErrInvalidTagsType (existing sentinels) so
agents see clean domain errors instead of "Go struct field" leaks.
- internal/mcp/dispatch_http.go: parseFieldKVP now accepts
map[string]any in addition to the existing array/string shapes. Map
shape preserves non-string values verbatim (e.g. number from a typed
flag), matching the array path's existing pass-through for non-string
entries.
- internal/mcp/catalog_item.go: update `tags` description from
"Comma-separated tags" (wrong on both SQLite and Postgres) to
"Tags as a JSON array of strings, e.g. [\"v1\",\"frontend\"]". Update
`field` description to clarify it's the escape hatch for
SCHEMA-DECLARED custom fields, name the dedicated top-level params
agents should reach for instead (status/priority/category/parent/
role/assign/tags), and note the new map-shape acceptance. Tool-level
prose updated to match.
Tests:
- TestItemCreateUnmarshalFlexFields (mirror of
TestItemUpdateUnmarshalFlexFields): 9 cases covering array/string/
null/absent/wrong-shape tags + object/string/array fields, plus a
smoke test that other fields decode normally alongside the new
flex paths.
- TestParseFieldKVP_Variants: extended with 3 new map-shape cases
(basic map, empty-key-skipped, non-string-value preserved).
End-to-end verification: 5 input shapes via curl against the live
handler. Pre-fix `tags: ["foo","bar"]` returned HTTP 400; post-fix
returns HTTP 201 with `tags="[\"foo\",\"bar\"]"` in the column.
`tags: {x:1}` (wrong shape) now returns a clean
domain-level 400 instead of leaked Go internals. Existing back-compat
paths (JSON-encoded string forms) preserved.
Related: PR #546 (BUG-1430 rate limit) addressed the original 500
cascade that drove the agent's specific misdiagnoses in BUG-1409.
* fix(mcp): forward tags array on update + drop unsupported map-shape doc per Codex review (round 1)
Codex round 1 caught two issues:
[P1] dispatch_http_advanced.go's PATCH builder filtered on `string`
only when forwarding `tags`, so a schema-conforming
`pad_item.update tags: ["a","b"]` was silently dropped. Now forwards
verbatim like mapItemCreate does — the handler's ItemUpdate
flex-unmarshaler (BUG-1144) normalizes any shape downstream.
Regression test added.
[P2] The `field` description claimed `{key: value}` map shape was
accepted, but the schema Type stays `array<string>` so schema-following
clients won't send the map shape. parseFieldKVP's map-shape handling
(added in the previous commit) stays as defensive parsing for clients
that ignore the schema, but the description no longer promises a shape
the published schema doesn't advertise. Tool-level prose updated to
match.
* fix(mcp): revert speculative parseFieldKVP map-shape support per Codex review (round 2)
Codex round 2 [P2] pointed out the map-shape parseFieldKVP support
added in the first commit is dead code in practice:
1. The advertised schema for `field` is `array<string>` — no
schema-conforming client sends a map.
2. `BuildCLIArgs` rejects map-shaped repeatable flags before they
reach the HTTP dispatcher.
3. Even if a map did reach the dispatcher, `hasFieldChanges`
doesn't recognize map shapes as field changes — `pad_item.update
field: {effort: "l"}` would skip the merge and PATCH without
`fields`.
Either completing the support (fix hasFieldChanges + BuildCLIArgs +
ItemUpdate Unmarshal) OR reverting was the right call. Reverting
keeps the surface consistent with the schema and removes the
unreachable code; future agents who want to override fields can use
the documented `["key=value"]` array shape.
BUG-1431's functional fix lands as the catalog description tightening
(the empirical repro confirmed `status` placement already works in
both forms; the agent's misdiagnosis was rooted in unclear docs, not
broken code). BUG-1432's flexible JSON unmarshal on ItemCreate stays
— that's the real fix verified by the live-handler repro.
* fix(mcp): preserve empty-string tags no-op + table-driven test per Codex review (round 3)
Codex round 3 [P2] caught a regression introduced in round 1's fix: by
switching the tags forwarding guard from \`v.(string) && v != ""\` to
\`v != nil\` to support array shapes, the empty-string filter for tags
on update was lost. \`pad_item.update tags: ""\` would now forward an
empty string to ItemUpdate, which treats it as an explicit
empty-string write — corrupting the JSON/JSONB tags column (500 on
Postgres).
Fix: type-switch on tags. Empty string skips (matches pre-fix
behaviour); arrays (including empty array \`[]\`, the legitimate
"clear tags" case) and non-empty strings forward.
Tests: the single-shape array test is replaced with a table-driven
TestDispatchItemUpdate_TagsForwarding covering array, empty array,
empty string (no-op), and comma-separated back-compat. Each case
asserts the tags key's presence/absence and shape in the PATCH body.
|
||
|
xarmian
|
c38b3bf5cd |
feat(playbooks): add invocation_slug + arguments schema fields (TASK-1378) (#517)
* feat(playbooks): add invocation_slug + arguments schema fields (TASK-1378)
Foundational change for PLAN-1377 — playbooks become first-class invokable
procedures. Two new optional fields land on the Playbooks collection
schema:
- `invocation_slug` (text, kebab-case, unique-per-workspace among non-null
values): enables `/pad <slug>` direct invocation. Nullable so
trigger-only playbooks (e.g. on-release checklists) don't need one.
- `arguments` (json, array of {name, type, required, default, description}):
declares the playbook's argument contract; mirrors the body's
`## Arguments` section in queryable form.
Plumbing pieces:
- `models.FieldDef` grows two general-purpose options — `Pattern` for
regex validation and `UniqueScope` for collection-level uniqueness.
Both are opt-in; existing schemas are unaffected.
- `items.ValidateFields` learns the `json` field type (accepts any
JSON-decodable value) and applies `Pattern` to string-typed values.
- `handlers_items.checkUniqueFields` queries `Store.ListItems` to enforce
`UniqueScope == "workspace_collection"` on create + update.
- Two migrations (SQLite 054, Postgres 033) JSON-patch the playbooks
schema on existing workspaces so the new fields show up without a
workspace re-init.
- TypeScript `FieldDef` mirrors the Go side.
Parent: PLAN-1377.
* fix(playbooks): address Codex review round 1 findings (TASK-1378)
P1 — EditCollectionModal now round-trips opaque pattern/unique_scope
metadata. EditableField carries the new keys; the load + save paths
preserve them so re-saving the playbooks collection from the UI doesn't
strip server-side validation rules the modal doesn't yet expose
dedicated controls for. fieldFromDef mirrors the change for templates.
P2 — checkUniqueFields' pre-write ListItems check is now backed by a
partial unique index (idx_items_invocation_slug_per_collection,
SQLite + Postgres) scoped to non-empty, non-deleted rows. The pre-check
still gives users a friendly error message in the common case; the
index closes the TOCTOU race between two concurrent writers. The
create-conflict error message is now generic enough to cover both the
slug constraint and the new invocation_slug index.
P2 — `json` field type now rejects raw strings, numbers, and bools. Only
objects, arrays, and null are accepted, so a generic web text input
can't silently corrupt a structured field by emitting "[]" instead of
an actual array. FieldEditor.svelte routes `json` fields to a
read-only summary in both readonly and edit modes; dedicated editors
(like TASK-1384's playbook editor that owns `arguments`) own the
structured form.
P3 — invocation_slug regex now requires a minimum of two characters
(`^[a-z0-9][a-z0-9-]*[a-z0-9]$`) in the Go const, the SQLite migration,
the Postgres migration, and the validate tests. Single-letter slugs
would shadow plausible NL tokens (e.g. `/pad a ...`) and the doc
comment already claimed the two-char floor; this aligns code with
intent.
Parent: PLAN-1377.
* fix(playbooks): address Codex review round 2 findings (TASK-1378)
P2.1 — checkUniqueFields no longer passes IncludeArchived=true. The
application-layer pre-check now matches the partial unique index's
`deleted_at IS NULL` predicate so a soft-deleted playbook releases its
slug back to the pool and reclaiming it succeeds instead of 409'ing.
P2.2 — handleUpdateItem now maps UNIQUE constraint / duplicate key
errors from UpdateItem to HTTP 409, mirroring the create path. A true
concurrent-update race that slips past checkUniqueFields and trips the
partial unique index used to surface as a misleading 500.
(Not addressed in this round: Codex's third finding — concern about the
partial unique index applying to "every collection" — is, on close
reading, not what the index does. `ON items(collection_id, json_extract(...))`
scopes uniqueness to the (collection_id, slug) pair, so two items in
different collections with the same `invocation_slug` value coexist
fine. The migration-failure risk is theoretical: `invocation_slug` is
a brand-new field key, so no pre-existing items can have it set, and
no migration-time duplicates can exist. If a future custom collection
adopts the same field name, opting into per-collection uniqueness is
exactly the intended semantic of FieldDef.UniqueScope.)
Parent: PLAN-1377.
* fix(playbooks): map restore-path UNIQUE violations to 409 (TASK-1378)
Codex round 3: restoring an archived playbook can hit the partial
unique index on invocation_slug if a replacement item already claimed
the slug. Map UNIQUE constraint / duplicate key errors from RestoreItem
to HTTP 409 with a targeted message, matching the create + update paths.
Parent: PLAN-1377.
* fix(playbooks): map collab-snapshot UNIQUE violations to 409 (TASK-1378)
Codex round 4: the collab-snapshot PATCH branch under
`s.collab.UnderItemLock` ran its own UpdateItem call and fell through
to writeInternalError on any non-stale-snapshot error. A concurrent
edit racing the invocation_slug partial unique index would surface as
500 instead of 409. Mirror the main UpdateItem error mapping.
Codex's other round-4 finding — the partial unique index applying to
"every collection" — is not addressed because the index IS already
collection-scoped: `ON items(collection_id, json_extract(fields,
'$.invocation_slug'))`. Two items in different collections with the
same slug coexist; only same-collection duplicates conflict. Migration
duplicates are impossible because `invocation_slug` is a brand-new
field key with no pre-existing items setting it. A custom collection
that later adopts the same field name opts into per-collection
uniqueness, matching the FieldDef.UniqueScope="workspace_collection"
semantic.
Parent: PLAN-1377.
|
||
|
xarmian
|
7456b5aed6 |
feat(store): add workspace-scoped monotonic seq column to items (TASK-1352) (#492)
* feat(store): add workspace-scoped monotonic seq column to items (TASK-1352) Adds an `items.seq` column that bumps on every mutation (create/update/soft-delete/restore) as the cursor mechanic for the local-first read model's delta sync (PLAN-1343, DOC-1342 design decision #1). Each mutation stamps `MAX(seq) + 1 WHERE workspace_id = ?` inside the same transaction that performs the write, with a Postgres advisory lock keyed on the workspace serializing concurrent seq-bumping mutations. SQLite's single-writer rule covers the same guarantee there. Migration backfills existing rows with sequential per-workspace seqs in (updated_at, id) order so every workspace has a non-zero MAX(seq) floor immediately. Adds an idx_items_workspace_seq index supporting both the `/items-index` cursor read and the future `/items-changes` range scan. The Seq field is now populated through every items SELECT helper (GetItem, GetItemIncludeDeleted, ListItems, ListItemsIndex, listItemsFTS, SearchItems, ItemsModifiedSince, GetChildItems, ListStarredItems, ResolveItemIncludeDeleted, GetItemBySlugIncludeDeleted) and the workspace import path stamps it via the same MAX+1 subquery so imported rows don't all collapse to seq=0. Parent: PLAN-1343. Foundation for TASK-1353 (wire seq into /items-index cursor) and TASK-1354 (/items-changes delta endpoint). * fix(store): bump items.seq on role reorder, MoveItem, and field migrations per Codex review (round 1) Codex round 1 flagged that UpdateRoleSortOrder was rewriting items.role_sort_order without bumping the new workspace-scoped seq column — delta-sync clients would miss role-board reorders until a full refresh. The same gap applied to MoveItem (collection change) and MigrateItemFieldValues (bulk select-option rename), which are also user-visible mutations the cursor must surface. Each path now: - acquires the workspace seq advisory lock (no-op on SQLite) - stamps seq = MAX(seq)+1 inside the same transaction The bulk rename gives all rows affected by a single statement the same seq value (MAX+1 at statement start). That preserves the "no overlap, no gap" cursor contract — a client at cursor < MAX sees them all in one batch, at cursor >= MAX sees none. |
||
|
xarmian
|
bdcb62e902 |
fix(api): accept nested object/array for PATCH items fields/tags (BUG-1144) (#485)
The PATCH /api/v1/workspaces/{ws}/items/{ref} endpoint previously
demanded `fields` and `tags` arrive as JSON-encoded strings, because
models.ItemUpdate declares them as *string to mirror the storage shape.
Sending the natural nested-object shape any reasonable HTTP client
would produce returned HTTP 400 with a leaked Go unmarshal error
naming the internal struct field — confusing for anyone integrating
against Pad over HTTP (webhook reactors, custom dashboards, non-CLI
agents, third-party MCP bridges).
This is the symmetric input-side counterpart to BUG-991, which was
fixed at the MCP boundary in PR #364 with dual-emit normalization
rather than the full Plan-sized models.Item migration.
Fix: add a custom ItemUpdate.UnmarshalJSON that accepts either shape
on the wire and normalizes to the canonical string internally. The
struct field type stays *string, so the validation/storage/web/CLI
pipeline is untouched. All in-process Go callers construct ItemUpdate
literals (15 grepped call sites) and never hit UnmarshalJSON, so the
change is invisible to them.
Wrong shapes (e.g. `{"fields":42}`, `{"tags":{"x":1}}`) now return a
domain-level 400 — `"fields" must be a JSON object or a JSON-encoded
string` — surfaced via sentinel errors (ErrInvalidFieldsType /
ErrInvalidTagsType) that the handler unwraps from decodeJSON's
"invalid JSON: %w" wrapper.
Coverage:
- models/item_test.go: 10 sub-tests covering object, array, string,
null, absent, and wrong-type cases for both fields and tags.
- server/handlers_items_test.go: 6 PATCH integration sub-tests
asserting back-compat, the BUG-1144 repro now returns 200, and
that error responses no longer leak Go struct field names.
Smoke-tested against the live server with the exact repro curl from
BUG-1144 (HTTP 200), plus malformed (HTTP 400 with clean message)
and stringified-string back-compat (HTTP 200).
|
||
|
xarmian
|
18087463ce |
feat(collab): op-log cursor protocol — force-refresh + watermark advance (TASK-1319) (#472)
* feat(collab): op-log cursor protocol — force-refresh + watermark advance (TASK-1319)
Closes both holes left by TASK-1309:
1. Long-disconnected tab + external-write race. A reconnecting client
announces its highest applied item_yjs_updates.id via `?since=<id>`.
If that id is below MIN(id) for the item, rows it expected to
replay have been pruned and the server sends a `force_refresh`
control frame and closes the conn. Client recreates the Y.Doc
and lazy-seeds from items.content. Without this, Tab A's stale
state would silently overwrite an external CLI/MCP write on the
next 5s flush.
2. Browser-only-edited items never GC'd. Browser collab-snapshot
PATCHes now carry an op_log_cursor body field. The store advances
items.content_flushed_op_log_id only when the cursor matches the
current MAX(op-log.id) — proving the markdown captures every
persisted op. SQL CASE clause re-evaluates MAX at COMMIT time so
a peer op landing between client-side cursor capture and the
UPDATE leaves the watermark untouched (no over-advancement).
Combined cursor mechanism:
- Server attaches op_log_cursor JSON control frames after replay,
after every successful AppendYjsUpdate (originator), and to every
peer's binary fan-out (so all peers stay in lockstep without a
round trip).
- Client persists per-tab in sessionStorage (NOT localStorage —
avoids cross-tab cursor leakage that would force-refresh stable
sessions).
- Server's MIN(id) check + force_refresh fires only when a non-zero
`since` is below MIN; `since=0` is treated as a fresh client.
New store methods: MinOpLogID, MaxOpLogID. New ItemUpdate field:
OpLogCursor *int64. New control message types: op_log_cursor,
force_refresh. New OpEvent.OpLogID for cursor piggyback. Existing
collab tests updated to drain TextMessage cursor frames.
Tests cover: initial cursor frame after replay (populated + empty
op-log), force_refresh fires when since<MIN, delta replay when
since>=MIN, cursor broadcast to originator + peers on append, and
watermark advancement gated on cursor==MAX.
Parent: PLAN-1248. Builds on TASK-1309.
* fix(collab): skip stale-Ydoc flush on force_refresh teardown per Codex review (round 1)
A force_refresh tear-down means the local Y.Doc cursor is below the
server's MIN(item_yjs_updates.id) — its derived markdown is stale.
Without this guard the collab $effect cleanup runs flushCollabNow
on the way out and silently PATCHes that stale markdown back to
items.content, overwriting the canonical content the fresh provider
is supposed to lazy-seed from. Per Codex round 1 [P1] of TASK-1319.
* fix(collab): force_refresh on empty op-log + cancel pending flush per Codex review (round 2)
Two P1 fixes:
1. Manager.Join now force_refreshes when since>0 and the op-log is
empty (hasMin==false), not just when since<MIN. After
PruneAndApply wipes the entire op-log, MIN is undefined; the
original predicate would have admitted the stale tab and let its
on-open Y.encodeStateAsUpdate write resurrect the pre-prune
document.
2. The +page.svelte onForceRefresh handler now also clears
collabFlushTimer. Without this a 5s timer that armed before the
force_refresh frame arrived can still fire AFTER the cleanup
ran, PATCHing stale Y.Doc-derived markdown to items.content.
New test: TestRoomManagerForceRefreshOnEmptyOpLogWithSince covers
the empty-op-log branch.
Per Codex round 2 [P1] of TASK-1319.
* fix(collab): include forceRefreshNonce in Editor key so it remounts on force_refresh per Codex review (round 3)
The collab $effect cleanup runs on forceRefreshNonce bump, but the
<Editor> {#key} was `${item.id}:true` — itemID doesn't change, so
the keyed Editor wasn't unmounting. The Tiptap Collaboration
extension only binds in onMount, so the editor stayed wired to the
stale (destroyed) Y.Doc while a fresh provider+doc were set up
in parallel. Edits would either be unsynced or eventually flush
stale markdown again.
Adding forceRefreshNonce to the key forces the Editor to remount
in lockstep with the doc swap. Per Codex round 3 [P1] of TASK-1319.
* fix(collab): refetch item.content before lazy-seed on force_refresh per Codex review (round 4)
After force_refresh the collab $effect rebuilds the Y.Doc and the
lazy-seed (TASK-1261) seeds it from item.content. But item.content
was the cached page-state copy — possibly stale relative to the
server (the WS force_refresh can beat the SSE/visibility refresh
that would otherwise update it). Lazy-seeding stale content into
a fresh op-log re-introduces exactly the staleness force_refresh
was supposed to clear: the next 5s flush PATCHes that stale view
back to canonical items.content.
onForceRefresh now does an api.items.get() before bumping the
nonce so the rebuild's lazy seed reads server-fresh content. A
failed fetch falls through to the bump anyway (an editor on
possibly-stale content is still better than a broken editor).
Per Codex round 4 [P1] of TASK-1319.
* fix(collab): suppress cursor during replay + move force_refresh check before getOrCreate per Codex review (round 5)
Two more findings:
1. [P1] writeLoop sends op_log_cursor frames for live ops broadcast
during the replay window. A client disconnecting after one of
those cursors lands but BEFORE the rest of replay completes
would persist a cursor pointing past unreplayed rows. On
reconnect with since=that-cursor, server replays nothing — the
client's Y.Doc would be missing causally-required ops.
Fix: per-roomConn replayDone atomic.Bool. writeLoop suppresses
cursor frames while it's false. runConn flips it after the
post-replay initial cursor is on the wire. Live binary frames
continue to flow during replay (Yjs CRDT commutativity); only
the cursor metadata is gated.
2. [P2] Force-refresh path leaked an empty room. getOrCreate
inserted into m.rooms before the force_refresh bail-out left
an orphan entry that PruneSweep would later treat as 'active'
and skip indefinitely.
Fix: schema-rebuild + force_refresh checks now run BEFORE
getOrCreate. Both are store-only mutations and the per-item
lock is held throughout, so concurrency is unchanged.
New test: TestRoomManagerCursorSuppressedDuringReplay regression-
guards the cursor-suppression behaviour.
Per Codex round 5 [P1+P2] of TASK-1319.
* fix(collab): tighten initial cursor + sync-destroy provider on force_refresh per Codex review (round 6)
Two more P1 fixes:
1. runConn's empty-replay fallback used MaxOpLogID() to anchor
the initial cursor. A live op landing between replayTo
returning and the cursor write would be reflected in MAX
but its binary frame might not have flowed through this
conn's writeLoop yet — the cursor would advertise an id
the client hasn't received. Initial cursor is now strictly
max(highestReplayed, since); MaxOpLogID is removed from
the opLogStore interface.
2. Provider.handleControlMessage's force_refresh branch now
calls this.destroy() SYNCHRONOUSLY before invoking the
onForceRefresh callback. Previously the consumer's recovery
path (async items.get refetch) would race the provider's
own onClose-triggered reconnect, which would re-open with
since=0 and push Y.encodeStateAsUpdate of the stale Y.Doc
— recreating the corruption force_refresh was meant to
prevent. destroy() sets destroyed=true so scheduleReconnect
short-circuits.
Per Codex round 6 [P1] of TASK-1319.
* fix(collab): block flush scheduling during force_refresh recovery per Codex review (round 7)
Previously, after onForceRefresh fires:
1. Provider is destroyed synchronously.
2. Async items.get refetch is in flight.
3. forceRefreshNonce bumps after refetch resolves.
4. $effect cleanup runs, then rebuild.
But during steps 2-3 the editor component is still mounted with
the stale Y.Doc, and a local edit fires handleContentUpdate which
calls scheduleCollabFlush. clearTimeout earlier in onForceRefresh
only canceled the timer at THAT moment; a new edit during the
refetch window arms a fresh timer that fires before cleanup. That
PATCHes stale Y.Doc-derived markdown back to canonical content,
recreating the corruption force_refresh was meant to prevent.
Fix: forceRefreshInFlight flag set in onForceRefresh, blocks
scheduleCollabFlush, resets after the fresh provider is wired
(end of $effect run). Per Codex round 7 [P1].
* fix(collab): gate runCollabFlush itself on force_refresh in-flight per Codex review (round 8)
scheduleCollabFlush blocked the 5s timer path, but direct callers
of flushCollabNow / runCollabFlush (beforeunload handler,
rich-to-raw toggle) bypassed the guard. A page reload or raw
toggle DURING the force_refresh recovery window still PATCHed
stale Y.Doc-derived markdown to canonical items.content.
Pulling the guard into runCollabFlush covers every caller in one
spot and returns 'deduped' so the result-shape contract holds.
Per Codex round 8 [P1] of TASK-1319.
* fix(collab): distinct 'skipped' result for force_refresh path; raw-toggle aborts per Codex review (round 9)
runCollabFlush returning 'deduped' on the force_refresh-blocked
path was indistinguishable from a legitimate same-content dedupe.
The rich→raw toggle treats 'deduped' as 'server already has this
markdown' and seeds rawSeedMarkdown from it — letting the user's
next raw edit overwrite canonical items.content with content
derived from the stale Y.Doc.
Add a distinct 'skipped' result for the force_refresh path. Raw
toggle aborts on it (with a 'try again in a moment' toast); other
callers fall through unchanged because no other call site
behaviorally depends on 'deduped' vs 'skipped'.
Per Codex round 9 [P1] of TASK-1319.
* fix(collab): server-side gate + post-await client guard against stale collab-snapshot per Codex review (round 10)
A force_refresh frame can arrive WHILE a collab-snapshot PATCH is
already mid-flight to the server. The client-side
forceRefreshInFlight check at PATCH-start can't catch this race;
the request lands at the server with stale Y.Doc-derived markdown.
Two-pronged fix:
1. Server: handler now checks op_log_cursor against MIN(op-log.id)
for collab-snapshot PATCHes and returns 409 Conflict when
cursor < MIN. Such cursors prove the flushing tab's Y.Doc was
built on rows that have been pruned (PruneAndApply, schema
rebuild, dormant GC). The markdown is, by construction, stale.
2. Client: post-await check on forceRefreshInFlight returns
'skipped' instead of 'flushed' so saveStatus / lastFlushedContent
don't seed from a known-stale base even if the server happened
to accept the PATCH (e.g. MIN advanced after handler validation).
New tests: TestCollabSnapshotRejectsCursorBelowMin (gate fires),
TestCollabSnapshotAcceptsCursorAtOrAboveMin (negative path).
Also de-leak an unused slice in the round-5 cursor-suppression test
so staticcheck stays clean.
Per Codex round 10 [P1] of TASK-1319.
* fix(collab): reject collab-snapshot when cursor>0 and op-log empty per Codex review (round 11)
The HTTP-layer gate I added in round 10 mirrored only PART of the
WS-upgrade force_refresh predicate. Round 5 had already taught us
that 'op-log entirely pruned' is a separate stale path from
'cursor below MIN' (PruneAndApply, schema rebuild, dormant GC all
leave hasMin=false), and the WS check now uses
`since > 0 && (!hasMin || since < minID)`. The HTTP gate had
only the second clause.
Mirror the WS predicate at the handler so a stale collab-snapshot
PATCH against an empty op-log gets a 409 too. New regression:
TestCollabSnapshotRejectsCursorOnEmptyOpLog.
Per Codex round 11 [P1] of TASK-1319.
* fix(collab): reject collab-snapshot cursor=0 on non-empty op-log per Codex review (round 12)
Round-11 gate accepted cursor=0 unconditionally. But a stateful tab
whose previous session disconnected BEFORE receiving the
post-replay cursor frame (network blip during the writeMu burst
between replay binaries and the cursor) ends up with sessionStorage
cursor=0 + a non-empty Y.Doc populated by prior replay binaries.
On reconnect with since=0 the server treats it as fresh, replays
nothing if the op-log was meanwhile pruned, and the client's
on-open Y.encodeStateAsUpdate resurrects pre-prune ops. The next
flush carries cursor=0 + stale-derived markdown.
The gate now refuses any incompatible cursor:
- cursor>0 + empty op-log (prior rule)
- cursor<MIN + non-empty op-log (prior rule, now naturally
catches cursor=0 too because 0 < any positive MIN)
The WS replay path is unchanged — full replay from since=0 is
the recovery for clients that genuinely lost their cursor; the
corruption manifested through the flush PATCH which we now gate.
New test: TestCollabSnapshotRejectsCursorZeroOnNonEmptyOpLog.
Per Codex round 12 [P1] of TASK-1319.
* fix(collab): close cursor=0 client/server gaps + lock validation+write atomically per Codex review (round 13)
Four P1 issues addressed:
1. Client always sends op_log_cursor (including 0) so the server
gate sees the field. Previously cursor=0 was omitted, which
silently bypassed the server's stale-snapshot rejection.
2. Provider construction now resets sessionStorage cursor to 0
when the Y.Doc is empty. The Y.Doc isn't persisted across
page reload, so a stored cursor=N + fresh empty Y.Doc would
announce since=N to the server and miss rows 1..N from
replay (server only replays id > N).
3. onOpen skips Y.encodeStateAsUpdate when lastOpLogID === 0.
A populated Y.Doc + cursor=0 is the network-blip-during-cursor-
write failure mode; pushing that state can resurrect ops the
server has pruned. Server replay + lazy-seed handle recovery
without our push.
4. Server gate now runs INSIDE the per-item collab setup lock
(new RoomManager.UnderItemLock helper) so a concurrent prune
(PruneAndApply, schema rebuild, dormant GC) cannot land
between the MIN check and the items.content write. Without
this, a tight race let stale snapshots overwrite canonical
content the prune just installed.
Per Codex round 13 [P1] of TASK-1319.
* fix(collab): gate handleDocUpdate on cursorAnchored to close stale-Ydoc edit path per Codex review (round 14)
Round 13 fix skipped on-open send for lastOpLogID===0, but local
edits via handleDocUpdate still propagated. A populated Y.Doc +
no-cursor-yet client could type, the edit would land in the
op-log with id N, server would send originator cursor=N, and
the next 5s flush would carry an 'anchored' cursor that passed
the server's MIN check — overwriting items.content with stale-
Y.Doc-derived markdown.
Add a cursorAnchored boolean. Set on first op_log_cursor frame
receipt (including cursor=0 against an empty op-log — that's a
legitimate 'server has nothing' signal). handleDocUpdate refuses
to send before this. Local edits buffer in the editor; once the
cursor arrives (or force_refresh rebuilds the provider), the
existing reconnect/edit paths catch them up.
Per Codex round 14 [P1] of TASK-1319.
* fix(collab): buffer + flush pre-anchor local updates per Codex review (round 15)
Round 14 silently dropped local Yjs updates fired before the
first op_log_cursor frame anchored the session. Yjs updates are
incremental: a dropped keystroke leaves later ops referencing
structs no peer can resolve, breaking convergence.
Buffer pre-anchor updates in a Uint8Array[] (capped at 1000 to
prevent unbounded growth in pathological 'anchor never arrives'
scenarios — overflow triggers force_refresh-style recovery).
On the first cursor frame, flush the buffer in order so the
server gets every causally-required struct before any post-
anchor updates land.
Per Codex round 15 [P1] of TASK-1319.
* fix(collab): destroy provider before force_refresh on pre-anchor buffer overflow per Codex review (round 16)
Round 15 overflow path called onForceRefresh but didn't destroy
the provider synchronously. A late op_log_cursor arriving before
the page-level rebuild (the recovery callback is async — refetches
items.content) would flip cursorAnchored=true, the partially-
populated buffer would flush, but the DROPPED prefix (the
overflowed entries) would leave server-side ops causally
incomplete — exactly the bug the buffer was supposed to prevent.
destroy() sets destroyed=true, removes message listener,
short-circuits scheduleReconnect, closes the socket. Late cursor
frames can no longer anchor a doomed provider.
Per Codex round 16 [P2] of TASK-1319.
* fix(collab): refuse rebuild on refetch fail + broaden on-open gate to cursorAnchored per Codex review (round 17)
Two findings:
[P1] force_refresh recovery bumped forceRefreshNonce in finally
even when the item.content refetch failed. The rebuild then
lazy-seeded from the cached (possibly-stale) item.content, and
the next flush would PATCH that stale view back to the server.
Move the bump into .then() so a failed refetch surfaces a
'please reload' toast and leaves the editor effectively
read-only (forceRefreshInFlight stays true, blocking flushes).
[P2] Send-on-open gate was lastOpLogID > 0, which silently
dropped local edits made during a brief offline window after a
legitimate 'cursor=0' anchor (empty op-log session). Switch to
cursorAnchored — the boolean specifically distinguishes
'unanchored' (stale Y.Doc + no server confirmation) from
'anchored at cursor=0' (legitimate empty op-log).
Per Codex round 17 [P1+P2] of TASK-1319.
* fix(collab): force_refresh on cursor=0 against non-empty Y.Doc per Codex review (round 18)
cursor=0 means the server's op-log is currently empty. A
non-empty Y.Doc at first-cursor receipt implies the ops came
from an earlier connection within this provider's life that
never reached its post-replay cursor frame, followed by a
server-side prune (PruneAndApply, schema rebuild, dormant GC)
during our disconnect. Anchoring at cursor=0 in that state
would mark a stale Y.Doc as authoritative; the next on-open
state push or flush would resurrect pre-prune state and
overwrite canonical items.content.
Detect the configuration via Y.encodeStateVector length and
invoke the same force_refresh-style recovery the explicit
server frame triggers: destroy provider, clear sessionStorage,
fire onForceRefresh so the page rebuilds from items.content.
Per Codex round 18 [P1] of TASK-1319.
* fix(collab): gate cursor=0 force_refresh on remoteSyncApplied per Codex review (round 19)
Round 18 force_refreshed the provider whenever cursor=0 arrived
against a non-empty Y.Doc. But local pre-anchor edits (user typed
before the initial cursor=0 of a legitimate empty-op-log session
arrived) ALSO populate Y.Doc — yet those edits live in
preAnchorUpdates and were supposed to flush on anchor. The
predicate spuriously triggered force_refresh, dropping the
buffered local edits.
Track remoteSyncApplied (set when readSyncMessage applies
anything to Y.Doc — replay binary or live peer op). Only force_
refresh on cursor=0 when remoteSyncApplied is true: that's the
true 'remote replay landed but server now reports empty op-log
=> mid-session prune' signature.
Per Codex round 19 [P1] of TASK-1319.
* fix(collab): repair brace mis-merge in wsProvider cursor=0 guard
The round-19 patch overlapped the round-18 inner block, producing
an extra brace + over-indented body. Collapsing into a single
clean block restores parseability without changing semantics
beyond what round 19 already documented.
* fix(collab): gate syncStep2 reply on cursorAnchored per Codex review (round 20)
readSyncMessage writes an inline syncStep2 reply when it receives
a peer's syncStep1. That reply embeds our current Y.Doc state.
If a peer's syncStep1 arrives before our first op_log_cursor
(pre-anchor window), the reply path bypasses handleDocUpdate's
cursorAnchored gate and lets potentially-stale Y.Doc state reach
the server before the cursor=0 + remoteSyncApplied force_refresh
recovery has a chance to fire.
Suppress the reply while unanchored. Peer state propagation
still works: the buffered preAnchorUpdates flush on anchor, and
the lazy-seed rebuild after a force_refresh seeds canonical
content from items.content.
Per Codex round 20 [P1] of TASK-1319.
* fix(collab): fold mid-replay live op ids into post-replay cursor + remoteSyncApplied only on apply per Codex review (round 21)
Two more findings:
[P1 server] writeLoop suppresses cursor frames during replay to
prevent the client persisting a cursor past unreplayed rows.
But binary frames for those live ops still go through
(commutativity), so the client APPLIES them to its Y.Doc. The
post-replay initial cursor only covered max(highestReplayed,
since), leaving the cursor below the highest applied op. On
empty-replay sessions this trips the client's
'cursor=0 + remoteSyncApplied' force_refresh path and discards
buffered pre-anchor edits.
Track maxLiveOpLogIDDuringReplay on the roomConn (atomic
compare-and-swap) and fold it into the post-replay cursor.
[P1 client] remoteSyncApplied was set on every MESSAGE_SYNC,
including syncStep1 (which only carries a state vector — it
doesn't apply state). A peer's syncStep1 arriving pre-anchor
would falsely flag remote-sync-applied and trip the cursor=0
force_refresh on legitimate empty-op-log sessions. Set the
flag only after readSyncMessage returns, and only for
syncStep2 / update subtypes.
Per Codex round 21 [P1] of TASK-1319.
* fix(collab): widen writeMu critical section + drop omitempty on op_log_id per Codex review (round 22)
Two more P1s:
[P1 server] writeLoop's mid-replay record-max happened OUTSIDE
writeMu, so runConn's post-replay read could race the record:
runConn loads → writeLoop's atomic store of higher value →
runConn sends cursor below the live id. Move the entire
per-event sequence (binary write + replayDone observation +
record-or-send) inside writeMu, and have runConn acquire
writeMu around its read+cursor-write+replayDone-flip. The lock
serializes the two paths cleanly: writeLoop events that ran
first have already recorded; events that arrive after replayDone
flips emit their own cursor frames.
[P1 protocol] OpLogID had `omitempty` JSON tag — a legitimate
cursor=0 (empty op-log session) serialized as
`{"type":"op_log_cursor"}` with no op_log_id field. The
client's strict-type check then rejected it as malformed,
leaving the session unanchored and local edits buffered
forever. Drop omitempty so 0 is wire-visible. Other control
types (applier_request/ack) carry an extra op_log_id:0 in
their JSON, which their client dispatches ignore.
Per Codex round 22 [P1] of TASK-1319.
* fix(collab): route originator cursor through writeLoop FIFO per Codex review (round 23)
readLoop sent the originator's op_log_cursor directly via
sendOpLogCursor right after AppendYjsUpdate, bypassing the bus/
writeLoop ordering. With a peer op already queued in rc.bus, the
sequence on the wire could be:
1. originator cursor=N (newer local op)
2. peer binary (older op)
3. peer cursor=M < N (rejected by client's max-take logic)
Client persists cursor=N. If the client then disconnects before
applying the peer binary, reconnect with since=N replays nothing
(server has nothing > N) and the older peer op is lost forever
to this client's Y.Doc.
Fix: writeLoop now processes self events too — skipping the
binary echo (the originator already has Y.Doc state) but routing
the cursor frame through the same FIFO bus channel as peer ops.
The originator's cursor=N now arrives strictly AFTER all
older-id peer events on the same channel.
Per Codex round 23 [P1] of TASK-1319.
|
||
|
xarmian
|
191b887e23 |
feat(versions): VersionSource attribution + collab coexistence (TASK-1267) (#465)
The collab 5s-flush PATCH (TASK-1260) sends
`?source=collab-snapshot` with a body of just `{ content }`. Without
a handler-side stamp, `Store.UpdateItem`'s default coerced empty
input.Source to "web" on the version row and the per-(actor, source)
throttle suppressed every collab-driven snapshot following the user's
last manual web edit — version-diff effectively went silent during
co-edit sessions.
Adds:
- ItemUpdate.VersionSource: overrides per-version-row Source
attribution WITHOUT mutating items.source. The latter feeds
WorkspaceHasAgentActivity's `source IN ('cli', 'mcp')` filter,
so a CLI/MCP-created item the user opens in the editor would
otherwise silently flip out of the agent-activity tally on every
auto-flush.
- Store.UpdateItem prefers VersionSource over Source for version
row creation; falls back to Source then "web" if neither set.
- handlers_items.go stamps `input.VersionSource = "collab-snapshot"`
for `?source=collab-snapshot` PATCHes (when not already set).
Tests:
- internal/store/items_collab_versions_test.go: store-level
reverse-patch reconstruction over a CLI→web→collab-snapshot
edit sequence; verifies IsDiff=true on at least one row.
- internal/server/handlers_items_collab_versions_test.go: full
HTTP-level test of the route; asserts a collab-snapshot version
row is created AND that items.source stays "cli".
Four rounds of Codex review.
|
||
|
xarmian
|
04514817ae |
feat(store): add Yjs op-log table + store methods (TASK-1252) (#450)
* feat(store): add Yjs op-log table + store methods (TASK-1252)
Persistence groundwork for the dumb-relay WebSocket server in PLAN-1248.
The item_yjs_updates table records every Yjs binary update (browser
edits, future designated-applier conversions of CLI/API content
changes) so reconnecting peers can replay updates since their last
known cursor and cold rooms can rebuild their in-memory Y.Doc.
Schema (mirrored across SQLite + Postgres):
- id monotonic — INTEGER PRIMARY KEY AUTOINCREMENT (SQLite)
/ BIGSERIAL (Postgres). Never reused, even after
deletes; serves as the cursor every reconnecting
client compares against.
- item_id FK with ON DELETE CASCADE so item deletion reclaims
op-log space automatically.
- update_data raw Yjs binary update — BLOB / BYTEA. Opaque to the
server.
- schema_version stamped per row. Mismatch on connect drives
TASK-1268's snapshot-and-rebuild flow.
- created_at ISO8601 UTC TEXT, matching pad's cross-dialect
timestamp convention (see migrations/047_attachments).
Drives PruneYjsUpdatesBefore.
Store API (internal/store/yjs_updates.go):
- AppendYjsUpdate — validates non-empty itemID/data/schemaVersion,
inserts and returns the new monotonic id (RETURNING on Postgres,
LastInsertId on SQLite). Empty-zero-byte updates are rejected at
the Go layer rather than relying on NOT NULL — they're a no-op
that would only pollute the log.
- LoadYjsUpdatesSince — strict id > sinceID filter, ordered by id
ascending. sinceID=0 returns everything (cold-room rebuild path).
Tolerates either RFC3339 or "YYYY-MM-DD HH:MM:SS" timestamp formats
on read so any future operator-written / CURRENT_TIMESTAMP-style row
doesn't blow up the load path.
- PruneYjsUpdatesBefore — created_at < cutoff, scoped to itemID.
Returns rows-affected count. Used by the eventual GC sweeper
(out of scope for this task).
Tests cover: append + monotonic ids, load-since-cursor filtering,
input validation, prune scoped to itemID, and ON DELETE CASCADE on
parent item removal. Pass on SQLite locally; Postgres mirror migration
+ store methods are dialect-agnostic.
Parent: PLAN-1248. First task of Phase 1 — Backend foundation.
* docs(store): document AppendYjsUpdate per-item serialization contract per Codex review (round 1)
P1: Postgres BIGSERIAL ids are allocation-ordered, not commit-order.
Concurrent appends to the same item could in theory produce a cursor
gap — a slower transaction can hold a smaller id while a faster one
commits a larger id first, and a reader that advances past the visible
larger id would later miss the smaller id when it commits.
The dumb-relay room manager (TASK-1255) is the sole writer per item by
design — there's exactly one goroutine appending per Y.Doc — so the
hazard does not manifest in practice. The fix is at the API contract
level: the doc comment now spells out the serialization requirement,
why the room manager satisfies it, and the multi-replica re-enforcement
note for the future Redis-fanout IDEA. We do not take an internal
advisory lock because that would be paid by every append even though
the caller already holds the per-room mutex.
No code change — contract is at the doc comment.
|
||
|
xarmian
|
f9d3244660 |
feat(connected-apps): user-facing OAuth connection management page (TASK-954) (#390)
* feat(connected-apps): user-facing OAuth connection management page (TASK-954)
Adds /console/connected-apps where a logged-in user can see every
OAuth grant chain they've authorized via the MCP consent flow
(Claude Desktop, Cursor, …) and revoke any of them. Joins to the
DCR client metadata for the display name + logo, and to the MCP
audit log (TASK-960) for the "last used" + "30-day calls" columns.
Pieces:
- internal/store/connected_apps.go — ListUserOAuthConnections walks
oauth_access_tokens + oauth_refresh_tokens, dedups by request_id,
hydrates client metadata, parses session_data for the workspace
allow-list, classifies granted_scopes into a coarse capability
tier. RevokeUserOAuthConnection verifies ownership (ErrConnection
NotFound for stranger's chains — anti-enumeration; same shape as
for unknown chains) then calls the existing RevokeRefreshTokenFamily
+ RevokeAccessTokenFamily so the next /mcp call gets 401.
- internal/models/connected_apps.go — OAuthConnection + CapabilityTier
models.
- internal/server/handlers_connected_apps.go — REST endpoints:
GET /api/v1/connected-apps (list) + DELETE /api/v1/connected-apps/{id}
(revoke, idempotent, 204). Wrapped in requireCloudMode group.
List enriches with MCPConnectionStatsForUser (audit aggregates) —
soft-fails on the audit lookup so a broken audit table degrades
to "no last-used data" instead of a broken page. Revoke records
an "oauth_connection_revoked" entry in audit_trail via the
existing CreateActivity path.
- web/src/routes/console/connected-apps/+page.svelte — list with
per-app card (logo, name, capability badge, workspace chips with
+N expander, connected/last-used relative times, 30-day count),
Details expander showing scope_string + workspace list + redirect
URIs, Revoke button → confirm modal → optimistic refresh, friendly
empty state linking to /connect.
- web/src/routes/console/+layout.svelte — Connected Apps nav link
(cloud-mode-gated, between Settings and Billing).
- web/src/lib/api/client.ts + types/index.ts — typed client +
ConnectedApp interface.
Tests cover:
- Store: chain dedup across rotation siblings, subject filtering
(Bob can't see Alice's), inactive chains excluded, ownership
check on revoke, idempotent re-revoke, capability tier mapping,
session-data allowed_workspaces parsing (both []string and JSON
[]interface{} round-trips).
- Handler: cloud-mode gate (404 outside), owner-only filtering,
DTO field shape + audit enrichment populating last_used_at +
calls_30d, revoke ownership 404 (not 403 — anti-enumeration),
idempotent 204, audit_trail row written.
`make check` clean (lint + go test ./... + svelte-kit build).
Parent: PLAN-943.
* fix(connected-apps): point empty-state link at getpad.dev (Codex review round 1)
Codex caught: the empty-state link to /connect 404s because /connect is a
pad-web (marketing site) route, not a docapp route. From inside the
authenticated console at app.getpad.dev, the right target is the
absolute https://getpad.dev/connect URL — same pattern the +error.svelte
page uses for its "Back to getpad.dev" + "/docs" links.
* fix(console nav): exclude /console/connected-apps from Workspaces active match (Codex round 2)
Codex caught: the Workspaces nav predicate `isActive('/console') && !isActive('/console/settings') && ...` was missing the new /console/connected-apps prefix, so both Workspaces AND Connected Apps lit up when viewing the connected-apps page.
Same shape as the existing exclusions for settings / billing / admin.
|