mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-11 13:28:57 +00:00
402f79e016d8dfd8ef8d3ee6b7583fe4359bb336
104 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xarmian
|
402f79e016 |
feat(store,server,web): collection kernel traits — de-hardcode conventions/playbooks slugs (TASK-2657, BUG-2702) (#1171)
Implements SPEC-5 §Collection traits (approved v1.1) — the first unit of
PLAN-2656 phase 0. Three kernel behaviors were keyed on the literal collection
slugs "conventions" and "playbooks": what the agent bootstrap loads, which
items route by invocation slug, and which items export as portable artifacts.
Collections now DECLARE those behaviors and the kernel resolves them from the
declarations.
Fixes the KERNEL half of BUG-2702, which stays open for the rest (see below).
A slug is not a stable identifier —
UpdateCollection re-slugs on any name change, and renaming a collection is a
documented onboarding step (TASK-1510) — so renaming either collection silently
detached all three behaviors from it, with the items still present and no error
anywhere. Measured on origin/main before the fix: conventions and
convention_index dropped 1 -> 0, playbooks 1 -> 0, and GET /playbooks/{slug}
went 200 -> 404, so `/pad ship` stopped resolving with no sign the playbook
still existed. Both halves are locked by regression tests observed failing on
unfixed code.
BUG-2702 is NOT fully closed here, deliberately. Every kernel behavior follows
the trait, and library activation on the MCP dispatcher and CLI was converted
too — but the pack's own dedicated web routes (/conventions, /playbooks list and
detail, /library) still address their collection by literal slug and render
empty after a rename. Filed as BUG-2705 with the route paths and the likely fix
shape; 2702 closes when that lands. Degradation there is bounded: no data loss,
and the collection stays usable at its own /[collection] route and in the
sidebar.
SPEC-5 was amended to v1.1 BEFORE any code, per the spec tree's own discipline:
bootstrap_include becomes a LIST of {mode, filter, key} because v1.0 could not
express convention_index at all; the conventions filter is now normative and
includes status=active, which v1.0's shorthand omitted and which the
implementation does enforce (implementing v1.0 literally would have leaked
draft conventions into every agent's boot payload); v1 filters are field-
equality maps with query/1 named as the widening path, since SPEC-2 is phase 1
and PLAN-2656 forbids growing toward it; and invocation_field is constrained to
the literal `invocation_slug`, because any other field name falls outside the
partial unique indexes in migrations/054 and pgmigrations/033 that are the real
uniqueness guard.
Traits get their own column rather than a key inside the schema JSON. The
schema column is overwritten wholesale on update and every client rebuilds it
fields-only, so a traits key stored there is destroyed by one ordinary
collection edit — measured during this task, not assumed. Trait authority
cannot rest on a value an unrelated UI save deletes. UpdateCollection writes
traits only when explicitly supplied, so pre-existing clients leave them alone;
an explicit "{}" still clears.
Bootstrap keeps its three payload keys as first-party views fed from the
declarations, and gains a generic bootstrap_includes array for any other
declared key — so the boot surface is genuinely generic rather than three
hardcoded payloads, and no consumer breaks.
Existing workspaces are backfilled slug-keyed in both dialects, guarded on
traits='{}' so a re-run cannot clobber a workspace's own declarations. The
backfill inherits today's blind spot (a workspace that renamed the collection
before upgrading is not reached) but cannot do worse than the status quo, which
is itself slug-keyed; from the backfill forward the hazard is structurally gone.
Malformed declarations are refused at create and update rather than stored:
an unparseable blob degrades to "declares nothing", which is silently the wrong
behavior instead of a loud error (SPEC-0 L6).
Web groups agent-facing collections by bootstrap_include presence, replacing a
hardcoded two-slug array repeated at five call sites.
Not done, deliberately: no MCP catalog change (traits are first-party kernel
declarations, no agent needs to set them, and the separate column means
pad_collection.update passes through harmlessly — no ToolSurfaceVersion bump);
bootstrap's collections[] projection does not carry traits (PLAN-1410 trimmed
that payload and nothing consumes them there); prefix.go's NormalizeSlug is
untouched (a pure function with no workspace context, and de-hardcoding two of
its six slugs would make it less coherent, not more).
Eight Codex review rounds found nineteen real defects, all fixed here. The
serious one:
bootstrap_include filter keys FAIL OPEN. The item store's field-filter path
drops any key its sanitizer rejects, removing the predicate rather than matching
nothing, so a declaration filtering on `"stat us"` would narrow nothing and ship
every convention — drafts included — to every agent at boot, defeating the
status=active guarantee this change makes normative. Filter keys are now
validated against the store's own sanitizer shape and pinned by a cross-package
agreement test, since models cannot import store and a future divergence would
silently reopen it. SPEC-5 amended to v1.2 with the rule and its fail-open /
fail-closed asymmetry. Also fixed: an unknown declared artifact_kind reached
artifact.Encode and surfaced as a 500 (now a 400 at the export boundary, since
SPEC-5 permits unknown kinds as legal non-round-tripping declarations); and
workspace import validated traits as JSON only, so an archive could persist a
declaration that degrades to "declares nothing" (now validated, degrading to
"{}" with a warning rather than refusing an import that may be the only copy).
Later rounds found more, and several were defects this change itself created.
A hidden collection could SHADOW a visible one: resolution used to name exactly
one collection, so with several declaring, resolving across all of them and
rejecting afterwards on visibility made a visible playbook unreachable behind a
hidden one — candidates are now filtered by visibility before selection, in both
playbook resolution and artifact import. Importing a pre-traits archive produced
an INERT workspace: the migration backfill cannot reach rows inserted long after
it ran, so conventions/playbooks arrived declaring nothing, and canonical
declarations are now inferred from the slug when a collection declares none
(never overriding declarations that survived the round trip). The generic
include path had no L4 boot budget and is now capped with an overflow count.
Trait parsing claimed to be strict but json.Decoder ignores trailing bytes, so
`{...} garbage` parsed cleanly. First-party payload keys are now mode-pinned,
since their projections have fixed shapes and declaring the other mode would be
silently ignored. Duplicate artifact_kind / invocation_field declarations are
refused at the collection API, and a conflicting archive warns on import.
Agent-facing text was updated with the rest, not after it: SKILL.md,
instructions.md and the MCP catalog said the literal slugs, which is exactly the
artifact an agent acts on. ToolSurfaceVersion 0.24 -> 0.25 for the
pad_library.activate behaviour change.
Trait uniqueness is a documented BEST-EFFORT gate, not an invariant, by lead
ruling. The gate reads then writes without a lock, import bypasses it, and a
rename can mint a duplicate without touching that path. The database-level
enforcement (partial unique indexes on the extracted traits) cannot ship first:
existing deployments can already hold duplicates via rename-then-reseed, so the
index would fail the migration on precisely the databases that most need
repairing. TASK-2710 carries the de-duplication pass and the indexes; SPEC-5
v1.3 records the deferral and the reason. L6's requirement that conflicts fail
loud is met by the refusal plus the warning — the mechanism is deferred, the
principle is not.
Gates: build · make lint 0 issues · go test ./internal/... · make test-pg ·
svelte-check 0 errors · vitest 99 files / 1734 tests. Mutation-verified across
four matrices, 20 mutations, 19 caught; the survivor is a seeding path whose
trait-vs-slug difference is unreachable today (SeedCollectionsFromTemplate
creates any missing template collection before it seeds items), recorded on the
task trail rather than papered over with a test that proves nothing.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
|
||
|
xarmian
|
449ac109e9 |
fix(server,mcp,cli): refuse system-metadata keys in fields_patch + retry-hostile error code (BUG-2627 part 2, BUG-2675) (#1166)
* fix(server,mcp,cli): refuse system-metadata keys in fields_patch + retry-hostile error code (BUG-2627 part 2, BUG-2675) Part 2 of BUG-2627 closes the door that mints the defect parts 1 and 3 dealt with: `--field implementation_notes=<json>` stored the entries as a JSON-ENCODED STRING, which is invisible to every reader and — since part 3's guard — disables `pad item note` on that item until the row is repaired. Refused SERVER-SIDE in `fields_patch`, not at the CLI as the item's scope line proposed. The deviation is deliberate and recorded on the trail: the CLI is one of three clients, and all three lower a user field-setter into the same key (`pad item update --field` at cmd_item.go, the MCP `field` param via dispatch_http_advanced.go on remote, and stdio by shelling out to that CLI). One gate closes all three; a CLI-only refusal would have left remote MCP writing the key. Both call sites were read, and the CLI's lowering is now pinned by a test rather than left as an assumption. Scope, stated because it is deliberate: this closes UPDATE only. The full `fields` blob stays open because that door is SHARED — `pad item note` / `decide` / `github link` send one, and so does convention activation via BuildConventionItemFields -> ItemCreate. Closing it would break the system writers the gate exists to protect. Item create therefore remains a mint site, tracked with the rest of that surface in BUG-2685. The refusal message is per-key: implementation_notes -> `pad item note`, decision_log -> `pad item decide`, github_pr -> the GitHub link flow, and `convention` refuses WITHOUT naming a command, because none writes it. PATTE-135 wants a remedy that works in the failing state; a single "use pad item note" line would have been wrong for three of the four keys. BUG-2675 rides along on one ToolSurfaceVersion bump, as ruled. The append refusal from part 3 reached MCP agents as `server_error` — not our fault, and not transient, so agents could reasonably retry a failure that is deterministic forever. New closed-set code `stored_state_unreadable`, emitted on BOTH transports: HTTP classifies the sentinel error directly, stdio via a `pad-structured-error/v1:` marker the CLI now writes for its own local refusal (the first marker generated without an upstream APIError). v0.16-then-v0.17 is what a one-transport fix costs. Also here: - items.ReservedOverrideKeys -> ReservedFieldKeysIn. The second caller passes a patch, not an override map, and the old doc comment said fields_patch was an open exposure — true until this commit. - `Extract* returns nil for THREE reasons` -> FOUR. The comment listed four; the count was corrected everywhere except the code. - Consumer-read artifacts updated where the claim is ACTED on, not only where it is documented: instructions.md (incl. a "do not retry this code" section), the catalog `field` param description, `pad item update --help`, README. Gates: build · make lint · go test ./... · make test-pg · Codex. Eleven-mutation matrix run against the new tests; every one killed by an assertion (two were rewritten after killing by compile error / surviving, which proves nothing). Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(server,mcp): honest remedy when the stored value is already unreadable; name the MCP-facing code (Codex round 1) Three findings from the pre-push review, all real: P2 — the refusal named `pad item note` unconditionally, but on an item whose stored value is ALREADY undecodable that command refuses too (part 3's guard). The caller was routed in a circle: field write refused -> run the note -> refused -> back again. That is exactly the failure PATTE-135 exists to prevent, and my own trail had reasoned the remedy was safe on the strength of the HEALTHY case only. The message now inspects the item's stored value and, when the key is unparseable, says so and points at the one action that works in that state (inspection), noting that the repair needs a full `fields` write no CLI flag exposes. P2 — two doc claims were false where an actor reads them. The catalog said reserved keys are refused "on every action that accepts field", which includes CREATE, and create is deliberately NOT gated; and both the catalog and instructions.md named `validation_error` (the HTTP code) where an MCP client actually receives `validation_failed`. Both corrected, and the create exception is now stated rather than implied by omission — an agent that reads only "refused on update" will otherwise assume create is fine, which is how a hole gets used. nit — the destructive-downstream sentence claimed every reserved key becomes unreadable and trips an append guard. True only for the two append-backed keys; github_pr and convention are simply overwritten. The clause is now per-key, because a confident wrong explanation is worse than a vague right one. Two more mutations run against the new branch: always-readable (the circular remedy returns) and never-readable (the working remedy disappears) — both killed by assertions. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(models,mcp,cli): one appendability predicate, per-key docs, stdio hint parity (Codex round 2) Five findings, all real. P2 — the message's readability check and the guard it describes were two different decodes. Mine unmarshalled into []json.RawMessage; the guard uses []ItemImplementationNote. A stored `[1]` passed mine and fails the guard, so the message would again have prescribed a command that refuses — the same circularity round 1 caught, through a narrower door. Replaced with models.StructuredFieldIsAppendable, which ASKS the guard rather than re-deriving it, plus an agreement test over 12 shapes x 2 keys that compares the predicate against the real Append* helpers. Verified by restoring the RawMessage version: the table catches it on `[1]`. P2 — stdio lost the new code's hint. Remote MCP told the agent retrying is pointless and how to inspect; stdio got the code with an empty hint, because the CLI's marker envelope carried none and the classifier parsed none. Both fixed, with the hint hoisted into paired constants (the same duplication StructuredErrorMarker already uses) and the test comparing the two TRANSPORTS' envelopes rather than either against a literal. P2 — doc text was still false for `convention`: the catalog, the instructions and `--help` all said reserved keys are maintained by note/decide/the GitHub flow, which is true of three of the four. Each key now names its own writer, and `convention` names library activation. Also dropped the `malformed_override` advertisement — that is the SERVER's code; an MCP client sees validation_failed for both refusals. nit — the classification test called structuredAppendErrorResult directly, so deleting either dispatcher call site left it green. Added dispatcher-level tests driving the real server + store, asserting the code, the hint, and that the item's stored fields are byte-identical afterwards. Mutation-verified by reverting the note call site. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(items,models,mcp): github_pr stays writable through fields_patch; no nil-map panic (Codex round 3) P1 — the gate refused `github_pr`, and that was wrong. My model was "system writers use the full fields blob, user setters use fields_patch", which holds for three of the four reserved keys and fails for this one: `pad github link` needs a local git checkout and the `gh` CLI, so it is excluded from remote MCP BY NAME, and internal/mcp/dispatch_http.go's noRemoteEquivalent map tells remote agents in so many words to use `item update --field github_pr=...` instead. For that audience the patch door is not a bypass of the writer — it IS the writer. So the refusal deleted a documented capability from remote agents, and answered with a message naming a command they cannot run: the same circular remedy round 1 caught, aimed this time at the people the gate was meant to help. items.PatchRefusedFieldKeysIn now exempts the key and records the rule being applied — refuse a raw write where a real writer exists — rather than the list it produces. Whether remote agents should get a proper PR-link action, so the key can be closed too, is a product question and is left as one. P2 — the hint told agents to read the bad value with `pad_item action=get`. They cannot: stripDuplicatedFieldsKeys removes implementation_notes and decision_log from every MCP response's fields blob, and the top-level arrays come from the extractor, which returns nil for exactly this shape. The value is invisible on the whole surface. The hint now says so and routes to a human, who can read it with `pad item show --format json`. P2 — `fields` holding a literal `null` unmarshals into a NIL map with no error, and both Append* helpers assign into what they get back, so `pad item note` PANICKED ("assignment to entry in nil map") instead of appending. Reproduced, fixed in parseMutableItemFields, and pinned by a test that fails on a panic rather than taking the process down. An absent blob and a null blob mean the same thing to every caller. Pre-existing, but it sits in the function family this bug is about and the message was about to recommend the command that panics. nit — README claimed a "closed eight-code taxonomy" (17 codes, and I had just added one) and read as if create lowers into fields_patch. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(models,mcp): predicate matches the append on malformed blobs; stop promising a broken workaround (Codex round 4) P1 — round 3 exempted `github_pr` from the update gate on the strength of noRemoteEquivalent's documented workaround. That workaround does not work: ingestFieldKVP (remote) and parseFieldFlag (CLI, and so stdio) both store a `field` value as a STRING, so the PR data lands double-encoded and no link appears — the BUG-2627 shape one key over. Filed as BUG-2696 with the three candidate fixes; NOT folded in, because the narrowest of them changes how every field value is typed. The exemption stands regardless: refusing would leave remote agents with strictly less than a broken door. What changes is what we may PROMISE. The catalog, instructions.md, version.go and README said "this is how you link a PR"; they now say the door is open and broken, and to hand PR linking to a human. Advertising a capability that isn't there is the failure mode this whole unit keeps circling. P2 — StructuredFieldIsAppendable returned TRUE when the whole fields blob was unparseable, on the reasoning that a broken outer blob is a different problem. True of the cause, irrelevant to the caller: the Append* helpers bail on that same parse, so the message again named a command that fails. It now returns false, which is simply the honest answer to the question asked, and the agreement table grew a malformed-outer-blob leg — the gap that let the disagreement through. P2 — the message claimed a raw field write always stores something Pad cannot read back. That holds for the CLI and MCP (a `--field` value is typed by schema lookup and these keys are in no schema) but not for a direct REST caller sending a valid array, who is refused for ownership reasons alone. Reworded to say both parts. nit — a misplaced parenthetical in the README read as if item CREATE lowers into fields_patch. It does not; it sends the full blob. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(mcp,models): stop the remote hint advertising the broken PR workaround; classify an unparseable blob as retry-hostile (Codex round 5) P1 — I corrected four artifacts that pointed agents at the github_pr field write and missed the fifth: noRemoteEquivalent's own text, which IS the message a remote agent receives when it calls `github link`, and which Codex had quoted at me in round 3 to establish the workaround existed. The nearest artifact to the actor was the one I did not open. Both entries now say there is no working remote path and name BUG-2696, with a test pinning the negative so a future edit cannot quietly reinstate the advice while the write is still broken. P2 — a fields blob that will not parse at all produced a bare parse error, so `note` / `decide` reached agents as `server_error`: transient- looking, and therefore retried, for a failure that is as deterministic as the per-key one BUG-2675 exists for. Both Append* helpers now wrap that parse failure in ErrStructuredFieldUnreadable, which both transports already classify, and the malformed-blob test asserts the sentinel rather than just an error. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * docs(mcp,cli): qualify what an agent can actually see when the state is unreadable (Codex round 6 nit) Round 5 widened stored_state_unreadable to cover a fields blob that fails to parse outright, which made half of its own hint false: MCP's normalization strips a broken structured KEY (so `get` hides it), but leaves an unparseable BLOB as a raw string (so `get` shows it). The hint and instructions.md asserted the first case for both. Now stated per layer, in the two paired constants and the instructions. The reason it is worth the words rather than being cut: an agent told 'you cannot see this' does not look, and would have missed a value that was in fact right there in the response it already had. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(mcp): classify the move/copy reserved-key refusal as validation on stdio too (Codex round 7) P2 — carried over from v0.22, surfaced because THIS bump documents the two reserved-key refusals as agreeing across transports. The move/copy message ("Field(s) reserved for system metadata and not settable here") matched none of the stdio validation patterns, so the same deterministic 400 arrived as validation_failed on remote and server_error on stdio — and server_error reads as transient, so an agent retries a refusal that can never pass. One pattern added, plus a test that drives both real classifiers with the real server message text for both refusals, so a reworded message that stops matching fails here rather than in the field. nit — the github_pr exemption is UPDATE-only; move and copy still refuse it, because there the argument is BUG-2674's (an override reintroduces the key the migration just dropped), not this one's. The catalog and instructions said "not refused" without that qualifier. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * fix(mcp): cover the copy path's own refusal wording in the stdio classifier (Codex round 8) P2 — round 7 fixed the MOVE wording; the copy path words the same class of refusal differently ("Destination collection has no field(s): ..."), so it kept arriving as server_error on stdio and validation_failed on remote. Third message in one family, and the round-7 test used the move text for every case, which is why it missed this. The parity table now carries all three real messages plus a control leg using one the pattern list already covered — without it the table could pass by matching everything. Recorded in the pattern list's comment rather than left implicit: matching prose is a stopgap, the structural fix is the pad-structured-error/v1 marker that carries the code instead of inferring it, and until a refusal emits one, this test is where a new wording has to be added. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * test(mcp): use the real upstream codes in the parity fixtures (Codex round 9 nit) The copy legs carried `validation_error` where the handlers actually emit `malformed_override` and `invalid_override`. The 400 branch ignores the body code today, so the test passed either way — which is exactly why the fixture mattered: it was quietly recording a wrong contract, and a future code-aware classifier would regress against a table that agrees with it. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * docs(mcp): the upstream code is not forwarded to MCP clients (Codex round 10 nit) The catalog said the server's own code (validation_error / malformed_override) appears in the MCP message. It does not: the 400 branch emits code=validation_failed with a fixed "Validation failed." message and the server's text in the HINT, discarding the finer-grained code. Reworded to say what an agent actually receives, and to say that telling the two refusals apart means reading the message. Also carried the update-only qualifier on the github_pr exemption into the README, matching the catalog and instructions. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V * docs(items): state the exemption predicate, not the exemption list (lead ruling) The lead's ruling on the github_pr reversal: make the REASON what the code says, so the next key added to reserved metadata is evaluated against 'does this audience have a real writer?' rather than pattern-matched onto a list that happened to be wrong for one key. Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V |
||
|
xarmian
|
de96cce900 |
fix(items,server,web): reserved metadata survives a move; referential metadata travels only within its context (BUG-2674) (#1165)
* fix(items,server): reserved metadata survives a move, and dropped fields are reported (BUG-2674)
Moving an item destroyed its implementation notes, decision log and linked-PR
metadata. Well-formed data, on a routine documented operation, silently, with a
success message.
Reproduced before the fix: a note written through `pad item note` — correct
shape, visible on every surface — was gone after `pad item move`, leaving
fields as `{"status":"new"}`.
## Why it happened
items.MigrateFields drops every key absent from the TARGET schema. The reserved
keys — implementation_notes, decision_log, github_pr, convention — are system
metadata that NO collection schema declares; each renders from its own dedicated
surface rather than as a generic field. So they are absent from every targetDefs
and were dropped on every move.
That blindness is structural, not incidental: any code path reasoning about
fields BY CONSULTING A SCHEMA cannot see these keys. It is the shared root of
this bug and of BUG-2627, where the CLI types a --field value by schema lookup
and these keys fall through to a raw string.
## The enumeration comes first, deliberately
Before this there were four constants and exactly ONE non-test consumer treating
them as a set — an inline || chain in a CLI display path. Naming the set inline
again here would have created the SECOND hand-maintained list, which is the
generator pattern behind both bugs reproduced inside its own fix: the next
reserved field lands in the constants, gets wired into whichever surface
prompted it, and silently misses the other.
So models.IsReservedItemField is now the single place that knows, MigrateFields
consults it, and the CLI's || chain is converted to it — the only way it is
provably THE list rather than A list. (formatChangeValue keeps its per-key
switch: it needs to know WHICH reserved key it has, to say "notes" vs "entries",
not whether the key is reserved.)
`convention` is IN the set, settled with evidence rather than by the principle
alone: 35 of 36 conventions in a live workspace do not store the key at all, and
the one that does holds a blob that is a redundant mirror of the alias keys
beside it. No user types a `convention` object — ApplyItemConventionMetadata
writes it, via library activation and the web form. System-stamped.
## Contract
System-minted non-referential data carries; anything dropped is reported.
PLAN-2357 DR-17 settled the analogous case — tags carry because "there is no
workspace-scoped foreign key to break, so dropping them would lose information
for no safety reason". These are the same shape: inert JSON with nothing that
could dangle in a destination. The plan's carry list simply never considered
them, so there was no deliberate semantics to defer to. DR-17's own heading is
"None of this may be silent."
## The reporting half
MigrateResult.Dropped has always existed and the single-move handler has always
thrown it away, so the only record of a field disappearing was the field being
gone. It now rides the move's audit metadata — not the response body, which is
the bare item and would break every consumer, and the activity timeline is where
someone asking "what happened to my item" looks. Joined into one string because
that map is map[string]string and a raw array renders as a Go map literal in the
timeline (BUG-2628).
## Verified
Unit: reserved keys carry with their payload INTACT (asserted on the value, not
merely the key — a carry that re-encoded or zeroed it would pass a presence
check), and bypass schema matching entirely, so a target declaring
`implementation_notes` as `text` cannot coerce them. Mutants run: guard removed
-> both new tests fail; carried-but-also-reported-dropped -> the not-dropped
assertion fails; carry-everything -> the control leg fails alongside three
pre-existing tests.
Live, against a server built from this branch: the note survives the move
byte-identical, and the move's activity metadata carries
`dropped_fields: "priority, status"` for the values the target schema genuinely
has no home for.
## Known scope limit
The BULK move path still discards its Dropped list — a reporting gap only, since
the carry-through lives in MigrateFields and bulk inherits it. Threading the list
out crosses two function boundaries whose signatures serve every bulk operation,
so it is a refactor of the bulk dispatch's return contract rather than a line.
Filed as BUG-2683 rather than smuggled in here.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
* fix(server,web): close the four gaps Codex round 1 found on the carry-through (BUG-2674)
Round 1 raised no P1 and four P2s. Three were real defects introduced or exposed
by the carry-through; one was a genuine overclaim in the previous commit. All
four closed here, each mutation-verified rather than asserted.
## A schema may no longer declare a reserved key
MigrateFields carries these keys by identity, but every caller then validates
against the target schema — and ValidateFieldsDetailed iterates schema.Fields,
so it DOES see a declared key. A target declaring implementation_notes as `text`
would receive the carried array and reject it, turning a move that previously
destroyed the notes into one that fails outright. That is a worse failure than
the one being fixed: loud, but it blocks an operation that used to work.
The gate already existed — validateNoReservedFieldKeys, with its
grandfathering — and listed only parent/plan. The four metadata keys join it,
sourced from models.ReservedItemFieldKeys() so the two lists cannot drift.
Forbidding the declaration is the honest fix; coercing the value, or skipping
validation for a key the schema genuinely declares, would be guessing at which
meaning the author wanted.
The web's RESERVED_FIELD_KEYS gains the same four, preserving the existing
deliberate asymmetry (the client lowercases and is therefore stricter than the
server's exact match) so the UI steers authors away before the 400.
## The copy preflight no longer under-reports
`carried` is built by walking the DESTINATION SCHEMA, and these keys are declared
by no schema anywhere — so after the carry-through they appeared in NEITHER
bucket. A copy of an item whose content is its notes would report "nothing
carries over" while in fact retaining them. Before the carry-through they at
least showed under `dropped`, accurately. Reporting in neither is a regression
in the preflight's honesty, which is the same defect class as the move that
reported nothing.
They are now appended to `carried` after the schema-ordered entries, marked
`type: "system"` with a rendered label since they have no author-supplied one.
The bucket's doc comment says so: a client must no longer assume every `carried`
entry resolves to a destination FieldDef.
## The audit report now reaches a human
The previous commit claimed the activity timeline is where someone asks "what
happened to my item" — true, and the timeline renderer ignored the key, so the
report existed only for API and CLI consumers. Stored-but-invisible is not
reported. TimelineActivityCard renders the dropped keys on a move.
## Test aliasing
The "untouched" assertions compared the result against the SAME objects passed
in, so an in-place mutation would change both sides and DeepEqual would stay
true. The expectations are now independent deep copies — the only thing that
makes "untouched" mean untouched.
## Mutants, each run
Preflight pass removed -> the carried assertion fails. Timeline block disabled
-> the render assertion fails. Timeline action guard dropped -> the non-move
negative leg fails (a presence-only test would have passed it). Reserved-set
helper returning everything -> the IsReservedItemField control leg fails.
## Not fixed here
Codex's remaining observation — that a cross-workspace copy now carries
github_pr into a workspace whose repository it does not describe, and leaves a
convention blob detectable on an item outside the conventions collection — is a
product question about what a copy MEANS, not a defect in this mechanism. Raised
for a ruling rather than decided inside a bug fix.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
* fix(items,server): referential system metadata travels only within its context (BUG-2674)
Lead ruling on the copy-semantics fork Codex round 1 raised. It does not add an
exception to the carry rule — it applies the qualifier the rule already had.
The contract was "system-minted NON-REFERENTIAL data carries". github_pr is
referential: it names a repository that is a property of the SOURCE workspace's
project, and it hydrates into code_context and renders as a live PR link. Carried
into another workspace that link is a false statement about the destination's
project, not preserved information. implementation_notes and decision_log
describe the item's own history and are true wherever the item is.
So the rule stays one sentence: non-referential system data carries everywhere;
referential system data carries only where its referent's context still holds.
## Scope is a required argument
MigrateFields takes items.MigrateScope. Required rather than defaulted because
BOTH wrong answers lose something: SameWorkspace on a cross-workspace copy
carries a PR link into a workspace it does not describe, and CrossWorkspace on
an ordinary move DROPS metadata from an item whose repo context never changed. A
caller that must name its scope cannot pick one by omission.
The two move handlers pass SameWorkspace as a property of the endpoint, not a
guess — a move changes an item's COLLECTION and cannot change its workspace.
The copy and its preflight COMPUTE it by comparing workspace ids rather than
assuming cross-workspace, because that endpoint accepts a target_workspace equal
to the source; hardcoding would drop a github_pr from a same-workspace duplicate.
Both sides use the same helper, or the preview promises a carry the copy drops —
the DR-6 divergence the shared endpoint exists to prevent.
## The drop is reported, with a reason that explains itself
PLAN-2357 DR-17: "None of this may be silent." It would be perverse to
reintroduce a silent drop inside this fix's own new branch.
The preflight reports it as `referent_not_portable` rather than the generic
`no_target_field`. That generic reason would be actively misleading here: no
schema declares these keys ANYWHERE, so "the destination has no such field" is
equally true of the source and explains nothing about why the value is being
left behind.
## Verified
Mutants run: scope ignored (always carry) -> the cross-workspace leg fails;
generic reason on the preflight drop -> the reason assertion fails. The
same-workspace leg and the non-referential-sibling leg are what stop an
implementation that ignores scope in EITHER direction from passing — each half
alone is satisfiable by a constant.
Gates re-run for THIS commit: lint 0 · go test ./... 0 · make test-pg 0 (3282).
Web gates NOT re-run and not claimed: this commit touches no web file (the web
half of BUG-2674 shipped in
|
||
|
xarmian
|
6a2910d45a |
fix(models): refuse an append that would destroy an unreadable structured field (BUG-2627, part 3) (#1164)
* fix(models): refuse an append that would destroy an unreadable structured field (BUG-2627, part 3)
AppendImplementationNote and AppendDecisionLogEntry rebuilt their entry slice
from Extract*, then assigned it over the field key UNCONDITIONALLY. When the
stored value was something Extract* could not decode, Extract* returned nil and
the assign overwrote that value with a one-element slice -- reporting success.
Observed live, not hypothesised: an item whose implementation_notes held a
JSON-ENCODED STRING lost its stored notes to a single `pad item note` call, with
no warning on any surface. Reproduced on a scratch item with two notes; both
were gone and the command printed "Added implementation note".
The guard cannot key on Extract* returning nil, which is the obvious shape and
the wrong one. Extract* returns nil for three different reasons:
1. the key is absent -- the first append on an item. Must proceed.
2. the key holds an empty array -- well-formed, just empty (Extract* has an
explicit len == 0 -> nil). Must proceed.
3. the key holds a value that does not decode -- the defect. Must refuse.
`if Extract(...) == nil { refuse }` passes every refusal test and breaks every
first append. So assertStructuredFieldAppendable tests decodability against the
raw value in the fields map, which is the only check that separates (3) from (1)
and (2). Applied to both helpers; ErrStructuredFieldUnreadable is exported so
callers can match on it.
The refusal message deliberately does NOT name an append path. Per PATTE-135 a
suggested remedy has to work in the state where the message appears, and every
append path is precisely what is being refused; `pad item show --format json` is
read-only and does surface the raw value, so it is the one action safe to
suggest. A test asserts the message never names `pad item note`.
Tests are mutation-verified per assertion. Each mutant was run and the killing
assertion recorded: guard removed -> the refusal legs; the plausible-wrong
`Extract(...) == nil` guard -> the empty-list and explicit-null CONTROL legs
(it passes all three refusal tests, so without those controls the wrong
implementation ships green); returning mutated fields alongside the error -> the
`fields != ""` assertion, which an errors.Is check alone would not catch;
message naming an append path -> the message assertion.
Verified end to end against a binary built from this branch: the trace that
destroyed two notes now exits non-zero and both notes read back byte-identical,
while a healthy item still takes a first note and appends onto an existing one.
This is part 3 of BUG-2627 and ships FIRST by design. Parts 1 (repair the
affected row) and 2 (refuse --field for structured keys at the CLI) follow,
because part 2's error message names a remedy that destroys affected rows until
this guard exists.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
* test(models): close the coverage asymmetry Codex round 1 found (BUG-2627, part 3)
Round 1 accepted the guard condition and the error path, and was right that the
tests did not carry the bite the commit message claimed. All four gaps closed,
each mutation-verified rather than assumed:
- decision_log had no control legs of its own. The two helpers carry INDEPENDENT
guard calls, so coverage on the notes side says nothing about the log side --
an Extract-nil guard on AppendDecisionLogEntry passed every existing test.
Mutant run: it now fails the empty-list and explicit-null legs.
- No ordering assertion, so a helper that PREPENDED satisfied every length
check. Mutant run: prepending now fails "existing entry is preserved".
- The non-string refusal shapes (wrong element type, list of strings, bare
number, incompatible nested value) were untested, and the object case asserted
only the error, not the empty fields return.
- No test covered a sibling reserved field surviving a successful append.
Mutant run: rebuilding fieldsMap fresh instead of mutating the parsed one now
fails, where it previously left every notes assertion green. github_pr is the
witness because it shares the fields blob.
Codex also reported two findings that are NOT fixed here, deliberately:
P1 -- moving an item drops implementation_notes / decision_log / github_pr
entirely, because items.MigrateFields drops any key absent from the target
schema and these are reserved metadata that no schema declares. Verified by
reading migrate.go and then reproduced live: a well-formed note written through
`pad item note` was destroyed by `pad item move`, silently, with a success
message. That is worse than the defect this part guards -- it destroys VALID
data on a routine operation -- but it is a different mechanism at a different
door, so it is filed as BUG-2674 rather than folded in.
P2 -- generic field writers (--field implementation_notes=...) still reach the
fields patch and overwrite. That is BUG-2627 part 2, which ships after part 1 by
the ordering already recorded on the item.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
* test(models): pin the guard's type parameter against a wrong-but-compiling swap (BUG-2627, part 3)
Codex round 2, and it is a real hole rather than a coverage complaint. The guard
is generic, so instantiating it with the WRONG entry type still compiles:
assertStructuredFieldAppendable[ItemImplementationNote](m, ItemFieldDecisionLog)
Every test written so far passes under that swap. The two structs disagree only
on shapes nothing exercised: `{"decision":{"nested":"object"}}` is ACCEPTED by
ItemImplementationNote (unknown key, ignored by encoding/json) and REJECTED by
ItemDecisionLogEntry (Decision is a string). So the guard would permit an append
that ExtractItemDecisionLog then reads as empty -- silently destroying the stored
entry. That is precisely the guard/extractor divergence this change exists to
prevent, reintroduced one type parameter away.
Both directions are now pinned, each mutation-verified:
- decision-log cases only ItemDecisionLogEntry rejects (a `decision` holding an
object, a `rationale` holding a list). Mutant run: the notes type parameter on
the log guard fails both.
- the mirror for notes (`summary` holding an object, `details` holding a list).
Mutant run: the log type parameter on the notes guard fails both, plus the
pre-existing incompatible-nested-value case.
Correcting the previous commit message: it said round 1's four gaps were "all
closed", which was overstated -- the malformed-entry matrix still ran only
against AppendImplementationNote, which is how this hole survived it.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
* docs(models): correct the guard's own comment — four nil cases, and pin the type-parameter warning where it is read (BUG-2627, part 3)
Codex round 3 returned CLEAN with three nit-level accuracy notes against my
commit messages. Two are already self-corrected in a later message; the third
matters because the SAME undercount sits in the code comment, which is the
artifact a maintainer actually reads.
- Extract* returns nil for FOUR reasons, not three: absent key, empty array,
explicit JSON null, and an undecodable value. The code always handled null
(its own branch), the comment just did not count it.
- The type-parameter hazard round 2 found now lives in the function's doc
comment rather than only in a test name. A wrong-but-compiling instantiation
is silently DESTRUCTIVE, not merely wrong, and the next person to add a third
structured field will reach for this function without reading the tests first.
Comment-only; no behaviour change.
Claude-Session: https://claude.ai/code/session_017jD6t1zjxGSq47SQpZfp1V
|
||
|
杨成锴
|
22c5a858a1 |
fix(server): stop counting disabled conventions as completed work (#1152)
Merged after two codex review rounds (converged) on top of the community-loop supply-chain/static review. Review found two narrow follow-ups — the guest item-grant leg of the grouped terminal query keeps pre-PR over-matching semantics, and the standup/changelog display layer hardcodes `status` — both pre-existing edges, filed internally as follow-up work. Thanks @asjdf for a well-tested fix, and for honoring the per-collection terminal_options contract on both the CLI and server paths. |
||
|
xarmian
|
6f16003199 |
fix: surface implementation notes + decision log in the item timeline (BUG-2301) (#1144)
* fix(server): merge implementation notes + decision log into the item timeline (BUG-2301) `pad item note` and `pad item decide` have written structured entries since |
||
|
xarmian
|
ec7fd027fc |
feat(server,cli): watches, user-scoped event stream, plugin monitor command — PLAN-2469 Phase 1 (TASK-2533) (#1082)
* feat(store): race-free status/assignment mutation signal (TASK-2533)
Adds models.Item.LastMutation (ItemMutationSignal), populated inside the
SAME transaction that already writes status_transitions / assigned_user_id
in UpdateItemWithParentLink and MoveItemWithPreCheck. This is the
foundation for TASK-2533's watch-notification pipeline: a before/after
snapshot taken in the HTTP handler layer would race concurrent writers of
the same item, so the signal is computed where the authoritative diff
already happens, in-transaction.
* feat(store): watches table migration, both drivers (TASK-2533)
watches(id, workspace_id, user_id, item_id, predicate, created_at) per
DOC-2479's subscription-table design: durable, server-side subscriptions
that survive both the plugin-monitor process and a padd restart.
uq_watches_user_item makes `pad watch <ref>` idempotent (re-watching
upserts the predicate). Wires watches into the workspace-purge child-delete
list, mirroring item_stars.
* feat(watchevents): add in-process notification bus (TASK-2533)
New package: a global (not per-workspace) in-process pub/sub bus carrying
watch-worthy Notifications (status-change / assignment / comment; ask
reserved in the enum with no producer yet — see the follow-up server
commit). Bus is an interface specifically so a Redis-backed implementation
can slot in later without touching any caller; only MemoryBus exists today.
Package doc comment states the single-process/multi-instance limitation
explicitly, mirroring internal/events' shape.
* feat(store): watches CRUD (TASK-2533)
models.Watch + Store.CreateWatch (upsert on user+item)/GetWatchByUserItem/
ListWatchesForUser (unscoped by workspace — a watch is personal, and the
event-stream handler needs every watch a caller holds across all their
workspaces)/DeleteWatch.
* feat(server): watch/nudge event stream + CRUD endpoints (TASK-2533)
GET /api/v1/events/stream (DOC-2479): a user-scoped, cross-workspace SSE
stream, filtered server-side to the caller's watches (with optional
--until field=value predicate) plus "addressed to you" — narrowed to
assignment-to-you only for Phase 1, confirmed with the dispatcher: this
codebase has neither a Collection.Kind field nor any user->active-role
binding to ground DOC-2479's "human-gate-shaped collection targets your
active role" half mechanically. watchevents.KindAsk stays in the wire
enum with no producer. `pad session register` is the natural future hook
for a session-carried role identity.
POST/DELETE .../items/{slug}/watch, GET /api/v1/watches (unscoped,
mirrors /auth/tokens' shape for a personal, not workspace, resource).
Producer wiring (TASK-2533 audit) publishes from every live mutation path
that can produce a LastMutation signal or a new comment: handleUpdateItem
(incl. its collab sub-paths and the comment-attached-to-update path, which
bypasses handleCreateComment entirely), handleMoveItem, handleCreateComment,
item creation with an initial assignee, and the bulk-items loop (covers
archive/restore/move/set-priority/tag/untag/assign uniformly via one call
site). Named, not silent, bypasses: import bundle, status_transitions
backfill, workspace restore/purge — none are live human-facing mutations.
Known Phase-1 tradeoff, flagged not fixed: bulk mutations are NOT batched
into one notification the way the existing SSE/webhook bulk path is — a
bulk-assign of N items surfaces N individual notifications. Each is still
correctly scoped by the recipient's own watches/addressed-to-you filter
(a narrower audience than the workspace-wide SSE firehose the existing
batching protects), so this is a noise-discipline tradeoff, not a leak.
* feat(cli): pad watch + pad session register (TASK-2533)
pad watch <ref> [--until field=value] creates/upserts a durable watch;
pad watch list / pad watch remove <ref> are the hygiene companions the
dispatcher asked to be included explicitly rather than silently added.
pad watch --stream --for-session is the plugin-monitor command: one
stdout line per matching event ("PAD TASK-214 -> kind (actor): summary"),
silent on startup with no .pad.toml (hourly retry) or an unreachable padd
(backoff retry) per DOC-2479's noise-discipline contract. The retry/
backoff math and line formatting are pure, unit-tested functions; the
actual sleep loop is not (per the dispatcher's ask).
pad session register writes ~/.pad/sessions/<pid>.json (pid, cwd,
CLAUDE_CODE_MESSAGING_SOCKET when set) -- forward-looking infra for
Phase 3's live-sessions/presence surface; nothing consumes it yet in
Phase 1/2.
* fix(server): comment replies never published a watch notification (TASK-2533)
Codex round 1 finding 2 (verified real, not a false positive):
handleCreateReply is a SEPARATE code path from handleCreateComment — it
calls store.CreateComment directly via POST .../comments/{id}/replies,
not POST .../comments — and was missing the watch-notification hook
entirely. A reply to a comment on a watched item produced zero
notification. Same kind=comment publish as the top-level path, plus a
regression test covering the reply route specifically.
* fix(server): re-check current access before serving/delivering watches (TASK-2533)
Codex round 1 finding 1: ListWatchesForUser filtered only by user_id — a
watch row survives a revoked workspace membership or grant (nothing
deletes it), so GET /api/v1/watches and the event-stream's notification
filter could keep leaking item title/ref, workspace slug, actor, and
summary for access the caller no longer has.
Adds Store.ListWatchesForUser's ItemCollectionID column (needed for the
visibility check) and server.filterWatchesByCurrentAccess, which mirrors
computeSSEVisibility's RBAC resolution (handlers_events.go) — admin
bypass, VisibleCollectionIDs for member/guest full-collection access,
GuestVisibleResources for item-level grants — grouped by workspace since
a caller's watches can span many, unlike a single SSE connection scoped
to one. Fails closed on any lookup error.
Wired into handleListWatches here; the event-stream's loadWatchPredicates
call site picks up the same filter in the next commit, which also
restructures that function's Subscribe/replay sequence and therefore
touches the same lines.
* fix(watchevents): atomic ID assignment + subscribe-and-replay (TASK-2533)
Codex round 1, findings 3 and 4 (same subsystem, fixed together):
Finding 4 — sequence assignment and replay-buffer insertion happened
under SEPARATE locks in MemoryBus.Publish. Two concurrent Publish calls
could append to the ring buffer out of ID order, corrupting since()'s
ordering assumptions (it walks the ring oldest→newest assuming monotonic
IDs). Fixed by unifying seq assignment, buffer append, and the
subscriber-list snapshot under one lock; the (already non-blocking)
fan-out send still happens after releasing it.
Finding 3 — GET /api/v1/events/stream called Subscribe() and, later
(when resuming via Last-Event-ID), EventsSince() as two separate calls.
A Notification published in the window between them landed in BOTH the
replay result and the live channel, double-delivering it. Bus gains
SubscribeAndReplaySince(sinceID), which atomically subscribes and reads
the replay buffer under the SAME lock; the stream handler now uses it
whenever a Last-Event-ID is present (this commit carries that call-site
change, plus the finding-1 loadWatchPredicates filter wiring from the
previous commit — both land in the same lines of this function).
Adds a concurrent-publish ID-ordering test and a subscribe-then-
concurrent-publish no-duplicate test, both run with -race.
* fix(cli): monitor silent-start ordering + sync_required handling (TASK-2533)
Codex round 1, findings 5 and 6:
Finding 5 (P1) — runWatchMonitor called getClient() once, before the
loop and before the .pad.toml check. getClient() -> getConfiguredConfig()
os.Exit(1)s when unconfigured with no TTY, or launches an INTERACTIVE
configuration wizard when one is attached — either way a direct violation
of DOC-2479's silent-start contract, which requires "not ready yet" to be
a silent retry, never a crash or a prompt. Adds monitorClient(), which
builds the client the same way but returns a plain error instead of
exiting or prompting; client construction now happens INSIDE the loop,
after the .pad.toml gate, on every iteration, and its failure folds into
the existing padd-unreachable backoff path.
Finding 6 (P2) — streamWatchEvents ignored "sync_required" (the server's
signal that the requested Last-Event-ID was evicted from its replay
buffer), so a stale cursor got resent on every reconnect forever. Now
clears the cursor on sync_required so the next reconnect is a fresh,
non-resuming subscription instead.
Both covered by tests that assert the goroutine returns promptly on
context cancellation (proving no os.Exit / no blocking prompt was hit,
since the test process itself is still running to observe the return)
and that streamWatchEvents clears/re-tracks the cursor correctly around
sync_required.
* fix(server): uniform current-access gate for watch AND addressed-to-you delivery (TASK-2533)
Codex round 2, findings 1 and 2 — same subsystem (watch/nudge delivery
access control), fixed together; finding 2 explicitly falsifies finding
1's fix's own admin-bypass argument, so this replaces that reasoning
rather than patching around it.
Finding 1 (confirmed real): VisibleCollectionIDs / GuestVisibleCollectionIDs
deliberately over-widen for navigation — a collection ID is included if the
caller has an item grant on ANY item inside it, explicitly leaving
item-level narrowing to the caller (their own doc comments say so).
computeWatchAccessVisibility used that over-wide set directly as the
"fully visible" gate, so a guest granted item A was treated as having full
access to A's WHOLE collection, including an ungranted sibling item B.
Fixed by building the "genuinely full access" set from
GuestVisibleResources' fullCollectionIDs (populated only from direct
collection_grants, never widened by an item grant) + GetMemberCollectionAccess
/ ListSystemCollectionIDs for an actual member — exactly computeSSEVisibility's
own fullCollSet construction, not an approximation of it.
Finding 2 (confirmed real): the addressed-to-you (KindAssignment) branch in
watchNotificationVisible returned true unconditionally, with NO access
check. validateAssignmentScope (internal/store/items.go) only checks
WORKSPACE membership, never collection access, so an item can be assigned
to a "specific"-access member whose granted collections don't include it
at all — an ordinary assignment, no revocation timing required. Fixed by
gating EVERY notification kind — watch-matched and addressed-to-you alike —
through the SAME watchAccessVisibility check before either branch runs.
watchevents.Notification gains CollectionID so the check has what it needs
without a second lookup; the stream handler resolves it lazily per
workspace via a small connection-scoped cache (workspaces aren't known in
advance for addressed-to-you the way watch workspaces are), cleared on the
same reval tick that reloads the watches map.
This also required replacing computeWatchAccessVisibility's admin-bypass
argument, not just its code: "every call site filters the caller's OWN
watches" stopped being a sufficient justification once addressed-to-you
(which is fundamentally about *this* caller's own assignment activity
across every workspace) shares the same gate — a bearer-borne admin token
unconditionally trusted for that is exactly BUG-1616's blast radius. Now
mirrors computeSSEVisibility's cookie-vs-bearer distinction exactly.
Tests: guest-with-item-grant no longer sees a sibling item's watch or
stream notification (filter-level and HTTP/SSE-level); an assignment
outside a restricted member's granted collections is denied at both
levels; addressed-to-you is proven still gated (denied with no access,
visible once granted) as a pure unit test.
* fix(store): always re-read existing under lock, not just for precheck/patch updates (TASK-2533)
Codex round 2 finding 4, verified real: updateItemWithParentLinkOnce's
`existing` snapshot was only refreshed under the write lock when precheck
!= nil, ExpectedUpdatedAt != "", or FieldsPatch != nil — any update
touching none of those (e.g. a plain title-only PATCH) kept the STALE
pre-tx `existing` for the rest of the function, including the
LastMutation assignment-delta comparison added in TASK-2533's first
round. A concurrent OTHER transaction's assignment change landing between
this transaction's pre-tx read and its lock acquisition would get
misattributed to THIS transaction: a title-only update could report a
spurious, wrongly-attributed AssignmentChanged for a transition it never
made, duplicating the one the other transaction already reported
correctly (or missing a real one, depending on interleaving).
The status-transition capture already defended against exactly this with
its own separate conditional re-read; the assignment-delta capture added
later did not replicate that guard. Fixed by making the re-read
unconditional — once, right after the locks are held, before any SET-
clause building or the UPDATE itself — so every existing.* comparison in
this function is race-free by construction, not by each caller
remembering to guard itself. Also removes the now-redundant duplicate
re-read the status code had of its own.
Reproduces the exact race deterministically using UpdateItemWithPreCheck's
precheck hook as a synchronization point (TX2's assignment change blocks
mid-transaction while TX1's title-only update races its own pre-tx read
against it) — the new test fails reliably against the pre-fix code and
passes reliably (including under -race, and in Postgres mode) against
the fix.
* fix(watchevents): send under the same lock Unsubscribe/Close use (TASK-2533)
Codex round 2 finding 3, confirmed real and high-severity: Publish
snapshotted subscriber channels under the lock, released it, and only
then sent to them. A concurrent Unsubscribe or Close could close one of
those channels in the window between the snapshot and the send — a send
on a closed channel PANICS in Go, which crashes the whole padd process,
not just one subscriber's connection. The reasoning for releasing the
lock before sending ("a slow subscriber would stall everyone else") didn't
hold up: the send is already non-blocking (select/default — a full
channel is dropped-and-logged, never awaited), so holding the lock
through it costs nothing and closes the window structurally.
Adds a hammer test (many iterations of concurrent Publish / Subscribe /
Unsubscribe / Close, short-lived churned channels, recover()-wrapped so a
regression fails cleanly instead of crashing the whole `go test` run) that
reproduces "send on closed channel" dozens of times per run against the
pre-fix code (plus an independent -race detection) and passes cleanly,
repeatedly, against the fix.
* fix(server): re-fetch the user, not just the vis map, on each reval tick (TASK-2533)
Codex round 3, confirmed real: watchVisCache captured *models.User ONCE
at connect time (newWatchVisCache) and never re-fetched it; reset()
cleared only the per-workspace visibility map. computeSSEVisibility's own
doc comment explains why it re-fetches the user fresh on every call —
"so mid-stream role changes (admin demotion, user.disabled flips) take
effect on the next tick" — and the round-2 commit claimed to mirror that
"exactly," but only carried over the collection/bearer logic, not the
re-fetch itself. Net effect: a demoted or disabled admin kept fullAccess
on an open stream (both watch-matched and addressed-to-you delivery,
since both go through this same cache) until reconnect.
Adds watchVisCache.refreshUser, called by both the constructor and
reset() so the cadence matches computeSSEVisibility's actual cadence in
handlers_events.go (that function is invoked once at connect and again
only on each membershipCheck tick — never per event — so "per cache
reset" here is the same cadence, not a narrower one). Deliberately fails
CLOSED (not open-to-stale like computeSSEVisibility's own transient-error
fallback) on a fetch error, a deleted user, or a disabled user — a nudge
stream's wrong failure mode is delivering a fact to someone who
shouldn't see it, not a dropped UI update, so this trades
computeSSEVisibility's availability-leaning fallback for a stricter one
and says so in the comment rather than repeating the "mirrors exactly"
claim the fix falsified.
Tests: a unit-level pair (mirroring handlers_events_revalidation_test.go's
existing admin-demotion/disable coverage of the analogous SSE gap
exactly) proves an admin loses fullAccess after a demotion + reset(),
and a disabled user is denied outright; an HTTP/SSE-level test proves a
live stream stops delivering entirely once its connected user is
disabled and a reval tick passes. All three reproduce the bug reliably
against the pre-fix code and pass cleanly against the fix.
The HTTP-level test deliberately runs serially (not t.Parallel()): it
mutates the package-level watchListRevalInterval var, which every other
parallel watch-stream test in this package also reads via its own
ticker — writing to it from a t.Parallel() test raced against those
reads under -race (misattributed by the race detector to a whole
cluster of unrelated concurrently-running tests before this was
diagnosed). Full server package -race pass is clean after the fix.
* fix(server): decouple vis-cache reset from watch-list reload success (TASK-2533)
Codex round 4, confirmed real: on a reval tick, if ListWatchesForUser
errored, the handler's `continue` skipped visCache.reset() entirely —
the two were coupled, with reset() only reachable on the reload's
success path. A demoted or disabled user's stale identity/visibility
(round 3's fix) stayed live for exactly as long as that UNRELATED query
kept failing, so the round-3 leak reopens for the duration of any
watch-list reload error.
Fixed by running visCache.reset() first, unconditionally, before
attempting the watch-list reload. On a reload failure, the stale watch
list is kept (its own staleness is already bounded by
watchListRevalInterval's "eventually consistent" contract) but is now
gated by the FRESH visCache regardless — a demoted/disabled user is
denied via visCache even while the watch list itself lags a tick.
Chose this over dropping all delivery for the tick (the other option the
finding offered) because tying stream availability to an unrelated
query's transient health seemed like the wrong tradeoff; the comment at
the call site states this choice explicitly.
Adds a watchPredicatesLoadFault test seam on *Server (mirrors the
existing restoreAckFault pattern) so the reload failure can be forced
deterministically without breaking the DB connection for the whole test.
Reproduces the exact bug: forces the reload to fail on every tick while
concurrently disabling the connected user, and asserts addressed-to-you
delivery (which depends only on visCache, never the watch list) is
denied anyway. Fails reliably against a reverted (pre-fix, coupled)
version of the reval branch and passes cleanly against the fix.
Full server package -race pass, full suite (SQLite + Postgres) pass,
lint clean — this is the pre-PR verification matrix; round 5 will be a
narrow re-verify of this fix only.
* fix(server): bound stale watch set under persistent reload failure; atomic test seam (TASK-2533)
Codex round 5, two P2s, both confirmed real:
Finding 1 — `watches = fresh` only ran on the reload's success path, so
under a PERSISTENT (not single-tick) reload failure the watch set stayed
live indefinitely: a dead watch (removed, item deleted) kept matching
forever, and a watch created during the outage was silently missed
forever — visCache (round 4) gates current ACCESS, not whether a watch
still legitimately exists, so it couldn't catch this on its own. Fixed
by tracking consecutive reload failures and clearing the watch set once
maxConsecutiveWatchReloadFailures (3 ticks) is crossed, failing closed
on watch-matched delivery specifically while addressed-to-you delivery
(visCache-only, unaffected either way) continues throughout. Updated the
tradeoff comment at the call site so the "eventually-consistent" claim
now matches the bounded, not unbounded, behavior it actually describes.
Finding 2 — the watchPredicatesLoadFault test seam was a plain `func()
error` field, written by a test AFTER the SSE stream's background
goroutine was already running and reading it on every reval tick:
genuinely racy, unlike restoreAckFault's own use of the identical field
shape, which is set once, synchronously, before the single HTTP request
that reads it — goroutine creation's happens-before edge makes THAT
usage safe without any extra synchronization. Verified restoreAckFault
does not share the flaw and left it untouched. Fixed the watch seam with
atomic.Pointer[func() error] instead.
Test for finding 1: forces maxConsecutiveWatchReloadFailures+1
consecutive reload failures via the (now-atomic) fault seam and asserts
watch-matched delivery is suppressed once the bound is crossed while
addressed-to-you keeps delivering, then clears the fault and confirms
watch-matched delivery resumes on the next successful reload — a bounded
outage response, not a one-way ratchet. Fails reliably against the
bound disabled, passes cleanly restored.
This is the (re-run) pre-PR verification matrix per the dispatcher:
SQLite + Postgres + full-suite -race + lint + gofmt, all clean. Round 6
is a narrow re-verify of these two fixes only.
* test(store): bound the concurrent mutation-signal test's wait (TASK-2533)
CI-triage follow-up: PR #1082's plain Postgres step hit go test's default
10-minute per-binary timeout. Investigated whether any store test added by
this branch scales with runner slowness (lock-wait defaults, sleep-based
polling, transaction-hold durations):
- Watches CRUD tests (8): 0.63-0.80s each under Postgres, isolated and in
the full 741-test package run.
- Mutation-signal tests (6), including the precheck-hook two-transaction
race test: 0.48-0.80s each; the race test held at 0.48-0.51s across 10
consecutive runs (no variance) and across the full-package run.
- Full store package under Postgres: 279.17s and 277.12s across two runs
on this branch, matching the ~275s/297s baseline team-lead measured
locally and on PR #1081 — no reproducible slowdown from anything this
branch adds.
No pathological test found locally. The one test with genuine
cross-goroutine DB lock contention (TestLastMutation_AssignmentDelta_
NotMisattributedUnderConcurrentWrite) had an unbounded wg.Wait() as its
only unbounded wait — TX2's release was already unconditional (fixed 50ms
sleep, not gated on TX1's progress), so there's no deadlock risk, but
there was no ceiling on how long legitimate lock contention could
stretch it under a slow/shared runner. Replaced with a bounded 10s wait
that fails fast with a diagnostic instead of silently consuming
test-binary budget if it's ever exceeded. Verified the regression test
still fails reliably (5/5) against a revert of the round-2 fix it guards.
Could not reproduce the CI timeout locally; likely the pre-existing
~297s CI baseline (already noted as close to the 10-minute ceiling)
plus environmental variance on the shared runner, not a specific test
this branch adds.
|
||
|
xarmian
|
1eb1c9eda6 |
feat(server): expose ACL-gated moved-to pointer on item GET (TASK-2359)
An item MOVED to another workspace — copied, then archived — can now say where it went. GET on a single item gains an optional `moved_to` block naming each destination in displayable terms (workspace slug + item ref + title + collection slug), so a consumer can render a link without a second call. No HTTP redirect, no resolver change. The ACL gate is the point. A destination is revealed only after the caller independently passes AuthorizeCrossWorkspaceRead (TASK-2358) with an ITEM scope on the destination item itself. Workspace-level access is not sufficient: a restricted member of the destination workspace, or a guest holding one unrelated item grant there, has a role in that workspace while having no right to the copied item's collection. A caller who fails that check sees NO hint a destination exists. The key is omitted entirely — not a null, not an empty array, not a boolean — so the response is byte-identical to an archived item with no move record at all. A structurally distinguishable response is itself the leak. Restore decision: the block is OMITTED for a non-archived source. Restoring a moved-out source leaves two live items with the same content in two workspaces, which is legitimate, but at that instant the source has not moved anywhere and the response must stop asserting that it did. Past-tense provenance is the back-pointer question and applies equally to plain copies, which this field must never claim as moves. Also honored: DR-2a (only archived_source rows feed the pointer; plain copies are back-pointer material only), per-destination filtering over the forward lookup's SET with no short-circuit on the first hit or first denial, newest-first ordering, a scan bound on the per-GET authorization cost, and deliberate isolation of the hand-rolled public share-link DTO — pinned by an explicit negative test that freezes its key set. Claude-Session: https://claude.ai/code/session_01E2fRi12n8rARczvdEa2LYT |
||
|
xarmian
|
bf14c1168a |
feat(store): add item_workspace_moves provenance table (TASK-2356)
Phase 1 of PLAN-2357. Durable record of "this item was copied/moved from workspace A to workspace B", backing the forward redirect (TASK-2359) and the destination's back-pointer. Implements DR-2 / DR-2a. Paired, dual-dialect, forward-only migrations (migrations/077 + pgmigrations/055). archived_source distinguishes a move from a plain copy (INTEGER on SQLite, BOOLEAN on Postgres, written through dialect.BoolToInt). source_seq is a NULLABLE per-source move ordinal that exists solely so two moves inside the same second are orderable — created_at is second-precision RFC3339, so archive -> restore -> move again would otherwise resolve to an arbitrary destination. Partial index (source_item_id, source_seq DESC) WHERE archived_source, deliberately NOT unique: restore-then-move-again legitimately repeats. The back direction IS uniquely indexed — a destination item is created by exactly one copy, in the same transaction that writes its provenance row, so a duplicate there would silently change which source the back-pointer names. Cascade is asymmetric on purpose, inverting item_collection_moves: the archived source is precisely the row whose pointer must survive, so source_item_id carries no FK at all; target_item_id cascades, because a pointer at a vanished destination is worse than no pointer. Store accessors: a tx-taking insert helper (no self-committing variant — the row must land in the copy transaction), a forward lookup returning a SET newest-first, and a back lookup. The insert rejects an archived row with no seq and a copy row with one, so DR-2a's ordering invariant is enforced at the write boundary rather than assumed. NULL ordering is normalized with COALESCE because SQLite and Postgres disagree on DESC NULL placement. Also wires workspace purge, which the two-workspace shape requires: both workspace columns are RESTRICT references, so a purge clearing only one direction would fail outright when the purged workspace sits on the other end. Tests cover insert-in-tx, forward lookup with multiple destinations ordered newest-first and scoped to one source, back lookup, rollback leaving no row, and both DR-2a criteria. The ordering and scoping tests use fixed row IDs whose lexical order contradicts the expected answer, so deleting the ordering term or the WHERE clause under test fails them on every run rather than half the time; verified by mutating the production query. Claude-Session: https://claude.ai/code/session_01E2fRi12n8rARczvdEa2LYT |
||
|
xarmian
|
40f88052cd |
fix(collab): version restore via prune+reseed (BUG-2264) (#990)
Version restore didn't reconcile the live Y.Doc: peers kept editing a Y.Doc built on pre-restore ops, and their next collab-snapshot flush clobbered the restored items.content. Reworked restore to prune+reseed — the restored content becomes canonical and every peer converges on it (unflushed edits are discarded, which is exactly restore semantics), replacing the earlier applier/epoch/watermark routing. handleRestoreItemVersion drives RoomManager.ForceRefreshRoom under the per-item lock. Hardened across Codex xhigh review rounds: - Atomicity: pre-prune MAX(op-log), the items.content write, the "Restored from…" version, the op-log wipe, AND both durable restore boundaries all run in ONE store transaction. A failed commit rolls back all of it — no divergent state, no fail-open boundary. - Unambiguous commit signal: UpdateItem reads the updated row WITHIN the tx (getItemTx) before commit, so a read failure can't make a committed update look failed and the returned seq is this restore's. - Restore freeze: conns are paused via a dedicated rc.frozen flag (NOT canWrite) so the auth-revalidation loop can't thaw the freeze mid-restore or promote a viewer; pickApplier + the applier-ack handler reject frozen conns so a concurrent external PATCH can't falsely succeed. - Stale-flush boundary: pre-prune MAX+1 fences in-flight snapshot cursors under the same item lock. - force_refresh fan-out deadlock: per-conn timer-close so a wedged writeLoop can't hang the fan-out + item lock. - Stale-SEED clobber: the client announces the item.seq it seeded from (?content_seq=) on every (re)connect; Join force_refreshes any seed that predates the last restore. Residual #1 (restart-durability) CLOSED durably, for BOTH stale vectors — the in-memory fences didn't survive a restart, so a surviving cursor-0 pre-restore browser tab wasn't fenced on reconnect. Two nullable per-item columns (migration 075 SQLite / pg 053), both stamped in the restore's own tx (atomic with the content write + op-log prune): * items.last_restore_seq — the content generation. Join's stale-seed fence reads it (via store.ItemLastRestoreSeq) when the in-memory fast-path misses (after a restart); if that read errors, Join fails CLOSED via a RETRYABLE plain close (not a force_refresh, which would discard the Y.Doc and spin an unbounded refresh loop) so the client reconnects with backoff, Y.Doc intact. * items.restore_boundary_op_id — the op-log-id boundary. The collab-snapshot flush gate reads it (via store.ItemRestoreBoundaryOpID) when the in-memory RestoreBoundary misses (after a restart), failing closed (409) on a read error, so a surviving tab's stale HTTP flush is fenced too. No SCHEMA_VERSION bump — durable columns are not a Y.Doc node-spec change. Deferred to BUG-2276: (a) a Postgres commit whose ack is lost is treated as rolled-back (needs commit-outcome reconciliation; SQLite unaffected); (b) a restore rollback racing an in-flight external-applier ack can drop the ack and retry/fall back (needs the applier flow serialised under itemLock at a 30s-stall cost). NOTE(BUG-2270): ForceVersion can mint same-second version rows; the item_versions ordering tie-breaker is tracked separately. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra |
||
|
xarmian
|
1dbe04399a |
fix(store): optimistic concurrency + sibling broadcast for collection settings (BUG-2265) (#989)
* fix(store): optimistic concurrency for collection settings writes (BUG-2265) Collection-level settings (e.g. quick_actions) were written by reconstructing the whole settings JSON from a caller's local Collection snapshot, and UpdateCollection replaced the column with no concurrency check. Two ItemDetails in the same collection (full-page pane host master + pane) hold independent snapshots and clobbered each other. Mirror the item optimistic-concurrency pattern (IDEA-1480): add CollectionUpdate.ExpectedUpdatedAt; when set, UpdateCollection re-reads updated_at atomically under the workspace write lock (SQLite BEGIN IMMEDIATE / Postgres advisory xact lock) and returns CollectionUpdateConflictError on a mismatch. Empty token keeps the legacy last-write-wins path unchanged for CLI/MCP/API callers. No DB migration — reuses collections.updated_at. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * feat(server): collection.updated broadcast + 409 conflict mapping (BUG-2265) - handleUpdateCollection boundary-validates expected_updated_at (400 on a malformed token) and maps store.CollectionUpdateConflictError to the shared update_conflict envelope (HTTP 409) — byte-identical wire shape to the item path, via the extracted writeUpdateConflictEnvelope helper. - Add the collection_updated EventBus type and publish it after a successful update so sibling ItemDetails / collection pages refresh their independent Collection snapshot proactively, shrinking the 409 window. Routed by Collection (slug) through the existing SSE visibility filter. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(web): 409-aware collection settings writes + sibling refresh (BUG-2265) - CollectionUpdate carries expected_updated_at; add isUpdateConflictError. - QuickActionsMenu sends the token and, on a 409, refetches the collection, re-appends the new action onto the FRESH settings, and retries once — no silent loss, no user-visible error. - EditCollectionModal captures the token at open-time (edge-gated seed so a concurrent broadcast can't wipe in-progress edits) and shows a non-destructive "changed elsewhere, reload" message on 409 rather than auto-merging a full-form edit. - Subscribe to collection_updated over SSE: ItemDetail and the collection page refresh their own Collection snapshot (gen/slug-fenced against the persistent pane host's no-remount switch), so siblings converge before the next save. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(store): harden collection optimistic concurrency + web fetch ordering (BUG-2265, Codex round 1) Address Codex review findings: - [P1] same-second clobber: now() is one-second precision, so two guarded writes in the same second kept an identical token. The accepted write now advances updated_at strictly past the token (only when now() hasn't already moved on), making a stale-token replay deterministically conflict. Add a same-second regression test. - [P1] tokenless-writer race on Postgres: the advisory lock only serialized writers that also took it. Replace it with a `FOR UPDATE` row lock on the in-tx re-read (Postgres) — SQLite's BEGIN IMMEDIATE already serializes every writer — so a concurrent tokenless UpdateCollection can't slip between the re-read and the UPDATE. - [P2] rename broadcast: only publish collection_updated when the slug is unchanged. A rename's old-slug event would make siblings refetch a dead slug (404) and a new-slug event can't reach old-slug visibility snapshots; renames are handled by the existing navigation path. - [P2] out-of-order refreshes: ItemDetail and the collection page now use a dedicated monotonic refresh counter so two rapid collection_updated fetches can't resolve out of order and clobber newer state (loadSeq/loadGeneration only bump on route/item loads). Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(store): make collection updated_at strictly monotonic for ALL writes (BUG-2265, Codex round 2) The previous same-second advance ran only on guarded updates, so a tokenless UpdateCollection could write the current second over a forced expected+1s, regressing the concurrency token and letting a stale guarded client clobber newer data (Codex P1). Route every collection update through one small transaction that re-reads updated_at (FOR UPDATE on Postgres; SQLite BEGIN IMMEDIATE covers it) and derives the new timestamp atomically: strictly advance past the row's current value when now() hasn't already moved on. This makes updated_at a reliable concurrency token for guarded AND tokenless writers. Add a tokenless-monotonic regression test. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: close remaining collection-concurrency gaps (BUG-2265, Codex round 3) - [P1] Board column reordering (handleGroupReorder) rebuilt the full schema from a stale local snapshot and wrote it with no token — a lost-update path identical to the bug being fixed. Now sends expected_updated_at and, on 409, refetches, re-applies the reorder onto the fresh schema, and retries once. - [P2] The workspace settings page seeded EditCollectionModal from a page-load-time collections list, so a change that predated editing produced a false 409. It now refreshes the list on collection_updated (seq-guarded). - [P2] collection.updated is now delivered to item-grant-only SSE subscribers for collections they can see — it's itemless but leak-free (only the slug), so guests' ItemDetail schema/settings snapshots converge too. Filter test extended. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(web): switch-safety + conflict-merge fixes for collection writes (BUG-2265, Codex round 4) - Board column reorder now ABORTS on a 409 (with a "reorder again" toast) instead of replaying a stale option order onto the fresh field, which would silently drop a concurrent option add/remove/rename. Reordering is cosmetic; never worth clobbering a real schema edit. Also captures ws/slug/base before the await and fences the write against a route switch. - QuickActionsMenu captures workspace + collection identity BEFORE the first await, so a mid-save navigation can't make the 409 refetch/retry target the wrong collection (no guaranteed remount). - Settings-page SSE refresh captures the workspace and drops the result if the workspace changed while fetching, so a slow refresh for workspace A can't overwrite workspace B's freshly loaded collection list. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(store): sub-second collection updated_at token, no future drift (BUG-2265, confirming pass #6) The same-second monotonic advance manufactured whole-second FUTURE updated_at values; sustained >1 write/sec on one collection drifted arbitrarily ahead of wall-clock. collections.updated_at is TEXT on both dialects and never compared lexically (only via time.Equal + display), so switch the update write to sub-second nowNano(): same-second collisions become near-impossible, so the token advances naturally. Keep a strict-monotonicity guard but step by a single NANOSECOND on the (now near-impossible) coarse-clock/step-back collision, so any drift is bounded to nanoseconds. Dual-dialect; covered by make test-pg. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(server): sanitize + always-broadcast collection event (BUG-2265, confirming pass #2,#3) - #2 (P1): collection_updated is delivered to item-grant guests, but the event carried ActorName/Source, leaking the owner's identity + edit source. Strip them — publishCollectionEvent now emits workspace + slug (+ new_slug) only. - #3 (P2): always broadcast (including on rename), routed by the OLD slug and carrying the NEW slug via a new Event.NewSlug field, so remote tabs on the old slug can re-target instead of silently 404ing on their next action. Tests: assert no actor/source leak on a settings update; assert a rename routes by old slug + carries new_slug. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(web): decisive switch-safety + rename handling for collection writes (BUG-2265, confirming pass #1,#3,#4,#5) - #1 (P1): EditCollectionModal captures the target collection id/slug/name/ws + updated_at when the form is SEEDED, and handleSave/handleArchive now operate on that captured identity (not the live props). The seed effect re-seeds when the collection IDENTITY changes (not on a same-id broadcast refresh), so a reused route can't leave A's form saving/deleting to B. - #3 (P2): on a rename event the collection route navigates to the new slug (preserving the pane query) and ItemDetail refetches by new_slug; the SSE event type carries new_slug. - #4 (P2): the reorder-conflict path refetches the collection (reseeds the token) before prompting, so a missed SSE event doesn't make every retry 409 forever. - #5 (P2): QuickActionsMenu only invokes oncollectionupdated when the live workspace/slug still match the captured identity, so a reused route can't assign an old response to the newly-navigated page. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(server): nano token round-trip, rename visibility, publish-before-migration (BUG-2265, confirming pass 2) Address the round-2 confirming-pass findings (server-only): 1. (P2) The shared update_conflict envelope formatted actual_updated_at with second precision (time.RFC3339), truncating the now sub-second collection token so the client's retry token never matched — a permanent 409 loop. Format with time.RFC3339Nano. Item tokens are zero-nanosecond, so RFC3339Nano emits no fractional part — the item 409 wire shape is byte-identical and the item path still compares via time.Equal. Added a test that the returned token round-trips as a usable retry token. 2. (P2) Rename events are routed by the OLD slug, but a subscriber that revalidated after the rename only has the NEW slug in visibleSlugSet, so the visibility check dropped the event before the new_slug branch. Accept a rename when EITHER the old slug or the (authorized) NewSlug is visible; downstream item-grant gating uses whichever slug is visible. Filter test extended. 3. (P2) The collection_updated event was published only after field migrations succeeded, but UpdateCollection already committed (updated_at advanced). On a migration failure clients got a 500 and no refresh, leaving siblings with stale tokens that 409 blindly. Publish on the commit (before the migration), so siblings always resync regardless of migration outcome. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: atomic collection update+migration; modal same-id rename retarget (BUG-2265, confirming pass 3) Address the round-3 confirming-pass findings; defer cross-tab rename RE-NAVIGATION to BUG-2272 (placeholder) per coordinator. 1. (P1) Migration atomicity. UpdateCollection committed the schema + concurrency token BEFORE MigrateItemFieldValues ran, so a migration failure returned 500 with the row already changed → the retry was guaranteed to 409 and item values were left inconsistent with the committed schema. Made the two ATOMIC: extracted applyFieldMigrationsTx and run it inside UpdateCollection's own transaction (after taking the workspace seq lock), so a migration failure rolls back the schema AND the token — nothing changes, the retry works. The handler now passes migrations through instead of running them separately, and publishes the event only after the fully-atomic commit. store/tx work → make test-pg run green. 2. (P2) EditCollectionModal same-id rename. The round-1 identity capture ignores same-id prop refreshes (to preserve edits), but a concurrent RENAME changes the slug (not the id), so handleSave/handleArchive PATCHed a dead slug → 404 before the token could 409. On a same-id prop change whose slug changed, the seed effect now retargets the endpoint slug + re-captures the token WITHOUT reseeding the form (in-progress edits preserved). Deferred (BUG-2272, TODO comments added, already broken on main — no regression): - ItemDetail full-page item URL/collSlug not retargeted after a remote rename. - Collection route chained-rename events during SSE replay landing on a dead intermediate slug. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(web): keep seeded token on same-id rename retarget (BUG-2265) On the EditCollectionModal same-id rename branch, retarget the endpoint (slug/name/ws) only — drop the token re-capture. Re-capturing let a later handleSave succeed against the renamed collection and apply the modal's stale pre-rename full form, silently REVERTING the concurrent rename (the exact stale-snapshot clobber BUG-2265 prevents). Keeping the seeded token means a concurrent rename correctly yields a 409 → the non-destructive "collection changed, reload" message. Slug-retarget without token-recapture gives both: no 404 (right URL) and no clobber (409 fires). Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: lock-order deadlock + unified collection-snapshot fences (BUG-2265, confirming pass 4) 1. (P1) DEADLOCK regression. UpdateCollection's atomic migration path took the collection-row FOR UPDATE lock and THEN the workspace seq lock, but item creation takes them in the reverse order (workspace advisory lock first, then the collection-row FK lock on INSERT) — a concurrent item-create + schema-migration ABBA-deadlocks on Postgres. Fix: acquire the workspace seq lock BEFORE the collection-row FOR UPDATE (matching item-create's order). Every store path that locks both now takes them workspace-seq → collection-row (tryCreateItem, UpdateItem, MigrateItemFieldValues, UpdateCollection). Added a concurrency regression test (item-create racing schema-migration); make test-pg green. 2+3. (P2) Cross-generation fence gap. The SSE collection refresh and route/item loads used SEPARATE counters, so a stale in-flight load could complete after a fresh SSE refresh and revert the collection + its concurrency token. Unified to a SINGLE monotonic collection-snapshot generation in BOTH the collection route and ItemDetail — every collection-snapshot write (loadCollection/loadData, the SSE refresh, reorder, and the quick-action/edit-modal callbacks) bumps it on start and gates its assignment on "still latest". ItemDetail's load keeps a switch-escape so a stale refresh for the OLD collection can't block loading a NEW one. Settings page unified the same way over its collections-list writes. 4. (P2) Settings page fed a stale editingCollection to the edit modal after a remote rename (its prop never changed → the same-id-rename retarget never fired → 404). The unified refresh now re-points editingCollection at the refreshed object for the same id, so the modal's retarget fires. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: emit item-changes signal on field migration; QuickActions retry-by-id (BUG-2265, confirming pass 5) 1. (P1) A collection update that runs a field migration mutates item `fields` JSON and advances item `seq`, but only collection_updated was published — open item views refreshed collection METADATA and returned without reconciling the migrated items, so clients kept stale field JSON under the new schema and a later full-fields item update could UNDO the migration (a clobber). UpdateCollection now returns the migrated-item count; when > 0 the handler ALSO emits the existing bulk item-mutation signal (items_bulk_updated, Op=migrate) so open views reconcile via /items-changes. Fires only when the migration touched >= 1 item — a pure settings/quick-actions update emits nothing extra. No store SQL/locking change (Go signature + count plumbing only); make test / make test-pg both green. 2. (P2) QuickActionsMenu's 409 retry GET-by-slug 404s if the competing update renamed the collection. Resolve the fresh collection by STABLE id (list + find by id) before re-appending + retrying, mirroring EditCollectionModal's identity approach; the result-propagation guard is now id-based too so a rename doesn't spuriously drop it. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: uniform sweep of item-grant delivery, rename routing, and 409/404 retries (BUG-2265, confirming pass 6) One pattern-sweep instead of per-site patches. Audited every event this PR publishes and every client retry path, applying three patterns uniformly: PATTERN A (item-grant SSE reconcile) + B (old-slug rename routing): instead of a SEPARATE items_bulk_updated migration event (which carries op/count for items an item-grant subscriber can't see and isn't rename-routed), FOLD a SANITIZED `items_changed` bool onto collection_updated — already item-grant-delivered (round 3) and already old-slug-routed with new_slug (round 2). On it the client triggers a /items-changes deltaSync (server-filtered to the caller's grants) and ItemDetail refetches its open item, so item-grant EDITORS reconcile migrated field JSON — closing the clobber where a stale full-fields update would UNDO the migration. Leak surface: "a collection you can see items in changed [+ renamed + had item changes]" — a bool, no per-item data. Removed the round-5 items_bulk_updated publish. The pre-existing items_bulk_updated (archive/move) is untouched and correctly stays suppressed for item-grant users. PATTERN C (409 AND 404 in retries): a competing RENAME can 404 a slug-targeted write before it can 409, bypassing recovery. Added isNotFoundError / isConflictOrNotFound helpers; every write/retry path now treats BOTH: QuickActions save resolves-by-id and retries on either; board reorder reseeds-by-id and aborts on either; EditCollectionModal save shows the reload prompt and archive resolves-by-id and retries on either. Tests: server asserts collection_updated sets items_changed on migration (not on settings-only) and stays sanitized; the SSE-filter test asserts the migration variant reaches item-grant subscribers for a visible collection; web unit tests assert 404/409 classification and a real component-driven not_found -> resolve- by-id -> retry in QuickActionsMenu. make test / make test-pg / npm run test all green. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: stable collection-ID identity for collection events + request-based items_changed (BUG-2265, confirming pass 7) 1. (P1) Collection events were identified only by MUTABLE, reusable slugs, and events replay — so a stale rename event's old slug, once re-owned by a DIFFERENT collection, could pass a slug-based match and misroute a client (navigate away / load the wrong schema) or leak the new slug. Fix at the ROOT: carry the STABLE CollectionID on collection_updated (Event.CollectionID) and match by ID everywhere: - Server visibility: sseEventVisibleFor matches collection_updated on a new visibleCollIDSet (built from the same VisibleCollectionIDs), not the slug — so an event for a collection the subscriber can't see by ID is dropped even if its (reused) slug is in visibleSlugSet. Filter test proves the slug-reuse drop. - Clients: ItemDetail and the collection route match `event.collection_id === <their collection>.id`, not slug. Slug(s)/new_slug stay only for the rename-navigation URL. Settings refreshes its whole list (already id-safe). 2. (P1) items_changed was keyed off the affected-ROW count, delivered to item-grant subscribers → a subscriber whose own items were unaffected could infer that HIDDEN items matched the migrated value. Now keyed off whether a field MIGRATION WAS REQUESTED (len(input.Migrations) > 0), independent of row count — leaks nothing about hidden item values. Reverted round-5's UpdateCollection count-return (no longer needed). Test: a migration matching ZERO items still sets items_changed. Deferred with markers: - NOTE(BUG-2273) at ItemDetail's reconcile-skip AND updateField: the web editor's full-fields field write lacks item-level OCC (never adopted IDEA-1480/v0.14), so the migration reconcile is best-effort. - TODO(BUG-2272) at the reorder 404 reseed: it refreshes `collection` but not the route `collSlug` (renavigation, deferred). Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix: archive OCC (no destructive wrong-target) + settings load fence (BUG-2265, confirming pass 8) 1. (P1) EditCollectionModal handleArchive resolved the target by stable id but the server DELETE re-resolves by the MUTABLE slug — a rename that re-owned that slug before the delete landed would archive the WRONG collection. Close the TOCTOU with an expected_updated_at OCC on the delete, mirroring the update OCC: DeleteCollection re-reads updated_at under a lock (FOR UPDATE on Postgres) and 409s on mismatch; the handler validates the token + maps the 409; the client sends it as a query param; handleArchive passes the seeded token (and the fresh token on the resolve-by-id retry). A reused slug or a concurrently-changed target now yields a clean 409 → the reload message, never a wrong-collection archive. Server test: stale token 409s (and the collection survives); current token 204s; malformed 400s; no token 204s. 2. (P2) settings load(): the generation was bumped AFTER awaiting setCurrent, so a slow load for workspace A could resume after B's load and clobber B's name/context/collections/members. Capture a dedicated loadGen at load() ENTRY (before any await) and fence EVERY continuation on it; the collections write additionally respects collectionsGen so it can't revert a fresher SSE refresh. Using a dedicated loadGen (not the SSE-shared collectionsGen) means an SSE collections-refresh mid-load doesn't drop the name/members writes. Deferred: TODO(BUG-2272) at the collection route's rename-navigation site — the global collectionStore (sidebar/pickers) isn't refreshed and the workspace layout ignores collection_updated, so the sidebar keeps the dead slug. Layout- level renavigation, deferred. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra * fix(web): dedicated item-snapshot fence + id-based rename comparisons in ItemDetail (BUG-2265, confirming pass 9) One comprehensive ItemDetail async-snapshot fence sweep so this file's item/ collection fencing is uniform and ID-based. 1. (P1) The migration item-refetch and loadData shared loadGeneration, so the refetch could apply migrated fields and then a stale loadData response overwrite them (a later full-fields edit then undoes the migration). Added a DEDICATED itemGen (separate from loadGeneration and collectionGen), bumped at the start of BOTH loadData's item load AND the migration refetch, and gated BOTH `item = ` writes on "still latest itemGen" — neither can stale-overwrite the other. Swept the other PASSIVE item snapshot-refreshes onto itemGen too (SSE item_updated/archived/restored, onSync deleted/incremental/full, the collab refresh) so they're ordered against each other and the migration/load. 2. (P2) A settings update that follows a rename before the rename fetch completes requested the OLD slug and bumped collectionGen, cancelling the valid rename fetch. Fetch slug is now `event.new_slug || event.collection || slug`. 3. (P2) The loadData collection fence-escapes compared the stale load's SLUG vs the freshly-renamed snapshot's slug (they differ on a rename → escape let the stale result overwrite). They now compare stable collection IDs; the SSE refresh's post-fetch identity check is id-based too. Audit (site -> generation -> id?): every PASSIVE snapshot-refresh (loadData item+collection, migration refetch, SSE x3, onSync x3, collab) bumps the correct dedicated gen (item->itemGen, collection->collectionGen) and compares identity by id. The DELIBERATE user/action writes (title/field/tag/assignee/role/content/ link/version/restore saves) keep loadGeneration + item-id switch-safety; their item-snapshot concurrency vs the migration refetch is the deferred item-OCC gap (BUG-2273, best-effort) — reordering them last-started-wins is orthogonal to that. Claude-Session: https://claude.ai/code/session_01EZ6yr6pAUFb1uffan912ra |
||
|
xarmian
|
c72fe5a663 |
feat(items): add unparented filtering contract (#926)
* feat(items): add unparented filtering contract * fix(items): preserve unparented projection state * fix(views): preserve reserved filter on reset * fix(items): resync projection scope changes * fix(items): address PR 926 review findings - localIndex: fetch snapshot before clearing store/cache in resyncProjectionScope (no data-loss window on fetch failure) - items: degrade to committed item when post-parent-link readback fails instead of 500 - items: treat unparented=<non-true> as a field filter so a schema field named unparented still filters - persistence: delete dead persistCursor - mark validateUnparentedListRequest canonical; cross-reference the 3 early-feedback copies * fix(items): resync race + purge safety per Codex review (round 1) - resyncProjectionScope: merge-reconcile instead of blunt clear so a higher-seq upsert/delta racing the snapshot fetch is preserved (not erased) and the cursor never regresses below it - recheck generation after persistWipe so a sign-out/403 purge during the wipe can't resurrect purged rows via persistDelta - snapshot rows authoritatively replace local copies (drop is_unparented on downgrade); mergeRow's projection-preservation is bypassed for resync * fix(items): sanitize projection bit on preserved racing rows per Codex review (round 2) When a projection resync lands a restricted snapshot, strip is_unparented from any racing higher-seq row kept by the seq guards — the old scope no longer grants it. Keep the row itself (dropping it would reintroduce the racing-mutation data loss; server 403 enforces real visibility). * fix(items): transactional cache replace in resync per Codex review (round 3) Replace wipe()+persistDelta() in resyncProjectionScope with a single persistReplace() transaction (clear + write in one tx). Avoids the deleteDatabase() onblocked cross-tab hang where a pending delete stalls the following reopen+write indefinitely, wedging the resync promise. wipe() stays for the sign-out / schema-mismatch full-teardown paths. * fix(items): drop-and-replay resync reconciliation per Codex review (round 4) Rework resyncProjectionScope: drop every row absent from the authoritative snapshot (not just older-than-cursor ones) and pin the cursor to the snapshot cursor. A post-snapshot mutation the client can still see is re-fetched by the next /items-changes?since=cursor under the NEW scope, so visible rows return and old-scope-hidden rows stay gone — no old-scope row survives the resync, and nothing is permanently lost. Present-in-snapshot racing edits are still kept (is_unparented stripped under a restricted scope). * fix(items): continue delta poll after resync so replay actually fires (round 5) The drop-and-replay resync (round 4) pins the cursor to the snapshot cursor so post-snapshot mutations re-fetch under the new scope — but both poll loops broke out / returned immediately after the resync, so the replay never ran until an unrelated sync/reload. Both callers now continue the loop from the pinned cursor; resync already aligned the scope so the branch can't re-fire, and the existing 50-iteration cap bounds it. * fix(items): keep pendingResync set until replay catches up (round 6) resyncProjectionScope cleared pendingResync after installing the snapshot but before the pinned-cursor replay drained. If that replay later failed or hit the 50-page cap, pendingResync stayed false and the next bootstrap() no-opped with racing mutations still missing. Let the reconcile loop's caughtUp logic own the flag instead. * fix(items): set pendingResync when any resync begins (round 7) Round 6 removed the premature clear but only the bootstrap path pre-sets pendingResync; a page deltaSync resync ran with it false, so a failed/capped replay there wouldn't trigger a bootstrap resume. Set pendingResync=true at the start of resyncProjectionScope so any caller marks catch-up pending; the reconcile loop clears it on caughtUp. * fix(items): fence stale optimistic writes + epoch-guard resync catch-up (round 8) Adds a resync-epoch + fenced-id mechanism to close the last two race classes: - fencedIds: a resync records the ids it dropped (hidden under the new scope). upsert() refuses a fenced id, so a stale old-scope create/update response resolving after the resync can't resurrect a now-hidden row that no new-scope delta would evict (P1). An authoritative applyDelta re-add un-fences; the next resync recomputes the set (re-upgrade clears it). Self-contained in the store — no epoch threading through the optimistic callers. - scopeEpoch: bumped when a resync installs a new snapshot. Both reconcile loops capture it before each /items-changes and skip treating a response that raced a concurrent resync as caught-up, so a stale in-flight delta can't clear pendingResync without validating the pinned cursor (P2). Regression test covers fence → reject stale upsert → authoritative re-add un-fences → later edits accepted. * fix(items): bump scope epoch before resync fetch (round 9 P2) scopeEpoch advanced only after listIndex() returned, so a reconcile response racing the fetch saw the old epoch and could clear the pendingResync the resync set at start. Bump the epoch before the network await instead. |
||
|
xarmian
|
bfa32dde5a |
fix(security): encrypt webhook HMAC secrets at rest, mask in responses (BUG-2057) (#915)
Webhook signing secrets were stored plaintext in the webhooks.secret column and echoed back in every API response. Encrypt them at rest (reusing the existing AES-256-GCM store helpers, same pattern as TOTP secrets) and return the raw secret ONLY in the creation response; list responses now mask it and expose a has_secret flag instead. - store: encrypt on CreateWebhook, decrypt on Get/ListWebhooks so the dispatcher still signs with the plaintext secret. Reuses the secret column with the "enc:" prefix — no new column/migration. Keyless self-host stays a no-op fallback (encrypt returns plaintext; decrypt passes legacy rows through unchanged). - BackfillEncryptWebhookSecrets encrypts pre-existing plaintext rows on startup once a key is configured (idempotent), mirroring the TOTP backfill. - model: add HasSecret so masked responses still signal presence. - handlers: mask secret on list; document raw-only-on-create. - tests: encrypt-at-rest round-trip + HMAC validity, list decrypt, plaintext backfill/back-compat, and the API mask-except-on-create contract. Claude-Session: https://claude.ai/code/session_015yuBJQYfDj95cgX3DaD8SF |
||
|
xarmian
|
3f69b76b06 |
feat(security): enforce session UA binding under strict mode (TASK-2056) (#912)
Session IP/User-Agent binding was log-only by default, so a stolen session token granted durable any-origin access. IP-change enforcement already existed behind PAD_IP_CHANGE_ENFORCE=strict; this extends the same single toggle to also enforce the User-Agent-hash binding. When strict enforce is ON, a request whose client IP OR User-Agent hash no longer matches the session's stored binding now revokes the session (DeleteSessionIfExists) and rejects the request (401 for API, revoked-passthrough for public/browser paths), killing the stolen token. When enforce is OFF (default), behavior is unchanged: UA mismatch is logged (slog only, no new audit row) and the request proceeds, so existing self-host users see no behavior change and routine client churn (browser/WebView updates, DevTools emulation, mobile-app rebuilds) is tolerated. The UA hash is stable within a real session, so UA-mismatch enforce carries fewer false positives than IP enforce (mobile roaming, VPN toggles, carrier NAT) — documented in the handler comment. Adds the ActionSessionUAChanged audit action, emitted only in strict mode. No DB migration: reuses the existing IPChangeEnforce config flag and the existing session store primitives. Claude-Session: https://claude.ai/code/session_015yuBJQYfDj95cgX3DaD8SF |
||
|
xarmian
|
bed933d7fd |
feat(items): field-level PATCH + conflict envelope + read-only version history (TASK-2022) (#876)
* feat(items): field-level PATCH + conflict envelope + version history Adds three related item-update primitives (TASK-2022 / IDEA-1480): - Field-level merge: PATCH `fields_patch` shallow-merges onto the item's current fields INSIDE the write transaction (null deletes a key), so concurrent single-field updates no longer clobber each other via the full-blob read-modify-write. `pad item update` and the MCP `pad_item.update` action now send only the changed keys. - Optimistic concurrency: optional `expected_updated_at` on update; on mismatch the store returns *UpdateConflictError and the handler emits the pad-structured-error/v1 conflict envelope (HTTP 409, code=update_conflict). Surfaced on CLI (`--expected-updated-at`) and MCP (`expected_updated_at`). - Read-only version history: `pad item history <ref>` (alias `versions`) and MCP `pad_item.history`, reusing the existing item_versions store + versions endpoint (no new store, no schema change). MCP ToolSurfaceVersion bumped 0.9 -> 1.0 (new action + param; update behavior change). No migration required. Claude-Session: https://claude.ai/code/session_019knGmnHcx5rrgWXQ8V8DZS * fix(items): address Codex review — dispatcher fields_patch, OCC ordering, date/required guards Round 1+2 review fixes for TASK-2022: - HTTP MCP dispatcher (dispatch_http_advanced.go) now sends fields_patch (only changed keys) instead of a client-side merged full fields blob, and forwards expected_updated_at — remote MCP callers get the same race-free merge + optimistic concurrency the CLI/HTTP paths do. - ValidatePartialFields rejects null-deleting a schema-declared REQUIRED field (would otherwise persist a blob the full-update validator rejects). - Open-children guard on the fields_patch path merges the patch onto the IN-TX locked row inside the precheck (not a stale pre-lock preview), so a priority-only patch can't false-fire the guard. - Optimistic-concurrency check now runs BEFORE the open-children precheck in the store, so a stale expected_updated_at yields update_conflict (not open_children) — single in-tx re-read shared by both. - Date auto-population on the patch path only fills an EMPTY current date; an existing end_date the caller isn't touching is preserved. Tests added for each fix. Claude-Session: https://claude.ai/code/session_019knGmnHcx5rrgWXQ8V8DZS |
||
|
xarmian
|
c846cff4fd |
feat(project): agent-accessible activity feed (pad project activity + MCP action) (#877)
* feat(project): agent-accessible activity feed (pad project activity + MCP action)
Add a non-streaming, bounded activity query so agents can catch up on
what other agents/users changed since they last worked — the query
counterpart to the live `pad project watch` SSE stream.
- CLI: `pad project activity [--limit N] [--actor user|agent] [--since DATE]`
backed by the existing GET /workspaces/{ws}/activity feed.
- MCP: `pad_project.activity` action (passThrough) + cloud HTTP route.
- Extend the activity endpoint with a server-side `since` date filter
(handler parse + store SQL clause) so limit/actor/since behave
identically across CLI, stdio MCP, and cloud HTTP transports.
- Bump ToolSurfaceVersion 0.11 -> 0.12 (drift guard, README, CLAUDE.md,
instructions.md) and add a SKILL.md querying-guidance line.
Tests: store since-filter test, HTTP dispatch test, catalog action test.
Claude-Session: https://claude.ai/code/session_019knGmnHcx5rrgWXQ8V8DZS
* fix(mcp): mark pad_project.activity read-only in tool surface
Add activity to readOnlyActions so the serialized MCP tool surface emits
read_only:true (missing entries default to write). Spot-check it in
tool_surface_test.go to guard against regression.
Claude-Session: https://claude.ai/code/session_019knGmnHcx5rrgWXQ8V8DZS
|
||
|
xarmian
|
48104a5eff |
perf(server): collapse dashboard/bootstrap N+1 into set-based queries (BUG-2002) (#847)
The dashboard builder (also reused by the bootstrap endpoint and every pad_set_workspace) ran ~1000 queries on a large workspace: one GetItemLinks + per-link GetItem for every non-done item (blocked attention + suggested_next filter), a GetChildItems per active plan (progress + suggested_next), a GetItemIncludeDeleted per recent-activity row, a GetCollection per visible collection, and a per-collection COUNT via ListCollections whose result the dashboard never uses. Replace every per-item/per-row loop with a set-based query: - GetBlocksEdges: one workspace-wide JOIN of blocks-links -> blocker essentials, ordered created_at DESC to preserve the old first-active- blocker selection. Drives both blocked attention and the suggested_next blocked-filter (retires itemBlockedByActive). - GetChildItemsForParents: one IN query grouping all active-plan children by parent (progress + suggested_next; no-content projection). - GetItemsByIDsIncludeDeleted: one IN query batch-hydrating recent-activity items (include-deleted). - ListItemsParams.NoContent: skip loading full markdown bodies on the count/summary scans (allItems, plans, stalled, orphaned). - ListCollectionsMinimal now also selects slug; the dashboard uses it in place of ListCollections, dropping the unused per-collection COUNT N+1 and the GetCollection-per-visible-id loop. Per-item N+1s are gone; query count is now constant in workspace size. Verified byte-identical dashboard + bootstrap JSON against three live workspaces (docapp/claude/apm); dashboard latency ~376ms -> ~198ms on the 1907-item docapp workspace. New store methods are unit-tested. |
||
|
xarmian
|
0aa431f132 |
fix(server,cli,mcp): default item list to per-collection non-terminal filter (BUG-2001) (#845)
The CLI's default `pad item list` (no --status/--all) sent a hardcoded ~20-status allowlist as the status filter. Collections with custom status vocabularies (blog: drafting/scheduled; human-tasks: todo) fell outside the list and had their open items hidden. MCP inherited the same bug via the CLI default and the HTTP route table's mirrored allowlist. Replace it with a server-side `non_terminal` filter: ItemListParams.NonTerminal resolves each collection's terminal set from its schema's terminal_options (falling back to DefaultTerminalStatuses) and keeps only items NOT in that set — reusing the existing doneFiltersForWorkspace + buildChildrenDoneExpr machinery, applied in both the normal and FTS query paths. The CLI default and both MCP dispatch paths (ExecDispatcher via the CLI, HTTPHandlerDispatcher via mapItemList) now send non_terminal=true. --status X and --all semantics are unchanged. |
||
|
xarmian
|
b0eeef16ce |
feat(store): email_verification_tokens + SendEmailVerification + token reaper (TASK-1936) (#806)
Wave 2 of PLAN-1933 — verification-token infrastructure (pure infra; no endpoint consumes it until Wave 3). - Migration 071 (SQLite) / 049 (Postgres): email_verification_tokens table, cloning the password_resets shape (id/user_id FK/token_hash/expires_at/ used_at/created_at + token_hash + user_id indexes), per-dialect created_at default. - Store email_verification.go: 256-bit crypto/rand token, padver_ prefix, SHA-256-at-rest, non-destructive Lookup, atomic UPDATE...RETURNING Consume. Deltas from password_resets (DR-2): 24h TTL, keep invalidate-prior-on-mint (resend burns the old link), consume side-effect sets users.email_verified_at (RFC3339-with-Z, same format Wave 1's migration used) in one transaction — no password reset, no session mint. - Email SendEmailVerification: clones SendPasswordReset, "1 hour" -> "24 hours". - Token reaper (DR-5): lifecycle-safe background sweep (mirrors orphanGC/opLogGC — self-registers on Server.bg, context-cancellable via stop channel, started only from cmd/pad/main.go so unit tests don't leak goroutines) calling the four previously-unwired CleanExpired* methods (email verifications, password resets, sessions, CLI auth sessions) hourly. Adds CleanExpiredEmailVerifications. - Audit consts ActionEmailVerified + ActionEmailVerifiedByAdmin. Gates: make check + make test-pg green (store + migration on both dialects). Claude-Session: https://claude.ai/code/session_01HxBkAMiFBtCRJ2tKSCt3ST |
||
|
xarmian
|
6a63fba188 |
feat(store): add users.email_verified_at column + model plumbing (TASK-1935) (#805)
Wave 1 of PLAN-1933 (email verification). Pure infra — nothing reads the column until Wave 3, so this is behaviourally a no-op and mergeable early. - Migration 070 (SQLite) / 048 (Postgres): add nullable email_verified_at TEXT, mirroring disabled_at. UNCONDITIONALLY backfill every existing row to verified (RFC3339 'Z'-suffixed) so no existing / OAuth / self-host account is write-locked on deploy (inverted vs password_set's conditional backfill). SQLite ALTER without IF NOT EXISTS; Postgres with it. - SAFE default = verified (DR-3): CreateUser / CreateOAuthUser write a verified timestamp unless UserCreate.Unverified is explicitly requested (only the future cloud self-serve branch will set that). A missed call site fails SAFE (verified), not write-locked. - models.User.EmailVerifiedAt + IsEmailVerified() (mirror IsDisabled). - Update userColumns + BOTH scan sites (scanUser AND the inline SearchUsers scan) so the admin user list keeps working. - Expose derived email_verified bool in sessionUserPayload for a later wave. Gates: make check + make test-pg both green (dual-dialect verified). Claude-Session: https://claude.ai/code/session_01HxBkAMiFBtCRJ2tKSCt3ST |
||
|
xarmian
|
010af13abd |
fix(ci): gofmt internal/models/workspace.go to unbreak Go job (BUG-1911) (#784)
The mid-struct doc comment added in #781 split the Workspace struct into two gofmt alignment groups; the file landed without re-running gofmt, leaving golangci-lint red on main and every PR since. Claude-Session: https://claude.ai/code/session_01CL1pBjNpPUX6SWkuAuYXHS |
||
|
xarmian
|
584ac9a806 |
fix(web): treat CLI/MCP-created workspaces as agent-connected (BUG-1557) (#781)
`pad init` connects an agent (installs the skill, stores credentials) and creates a workspace, but the web UI still showed the "connect an agent" banner and onboarding launchpad. The only signal for "agent connected" was has_agent_activity — an item existing with source cli/mcp — and a fresh pad-init workspace has zero items, so the UI nagged to connect an agent the user already had. Give the server a truthful signal: a workspace created through an agent surface already has an agent wired up before it creates its first item. Add a `source` column to workspaces (web/cli/mcp), attributed authoritatively server-side from the request auth shape (actorFromRequest) — never from the request body, so a web client can't spoof "cli" to self-suppress the prompts. The dashboard ORs source in (cli,mcp) into has_agent_activity when the cheap item check comes up empty. - migrations 069 (sqlite) / 047 (postgres): workspaces.source NOT NULL DEFAULT '' (legacy rows stay "unknown", never treated as agent-created) - models.Workspace.Source + WorkspaceCreate.Source (json:"-", server-set) - thread source through the CreateWorkspace INSERT + all 7 workspace scan sites (workspaces.go, workspace_members.go) - handleCreateWorkspace derives source from actorFromRequest - OnboardingLaunchpad step 1 collapses to "Agent connected" when the agent is already wired up, shifting emphasis to "tell it to set up" Web modal and cloud-signup auto-create flows are unchanged and still correctly prompt to connect (source web / empty). Tests: store source round-trip across reads; dashboard reports agent-connected for a cli-created workspace with zero items; web-created stays not-connected until an agent item exists; a web body-spoofed source is ignored. Claude-Session: https://claude.ai/code/session_01HxBkAMiFBtCRJ2tKSCt3ST |
||
|
xarmian
|
915f7e66c5 |
fix(web): show issue ID and status pills in activity log (BUG-1748) (#776)
Activity rows (the dedicated Activity page and the dashboard's Recent
Activity list) showed only the item title, never the issue ID. Add the
ref (e.g. BUG-1748) as a leading monospace badge on both surfaces.
The ref rides on the per-row item lookup that already runs to populate
the title, so there are no new DB queries — enrichActivities and the
dashboard recent-activity builder now also copy item.Ref after
ComputeRef(). New item_ref field on models.Activity, DashboardActivity,
and the TS Activity / recent_activity types.
The Activity page now renders field changes as structured pills
("status: open → fixing") instead of a raw string, via a new shared
parseFieldChanges util that also replaces the private copy in
TimelineActivityCard.
Claude-Session: https://claude.ai/code/session_01HxBkAMiFBtCRJ2tKSCt3ST
|
||
|
xarmian
|
3704cc2c9f |
fix(store): cast jsonb metadata to text for Postgres LIKE + gofmt (BUG-1702) (#693)
The status-transition backfill query used `a.metadata LIKE '%→%'`, but activities.metadata is jsonb on Postgres where LIKE (~~) is undefined, failing TestBackfillStatusTransitions(_SeedSeqBelowHop) and erroring in any Postgres deployment. Cast to ::text on Postgres (dialect-guarded), matching AttachmentReferenced. Also gofmt comment.go + the share-links test that were tripping golangci-lint. |
||
|
xarmian
|
076fb9b2e7 |
feat(comments): comment editing backend — user_id, UpdateComment, PATCH, SSE (TASK-1663) (#665)
* feat(comments): comment editing backend — user_id, UpdateComment, PATCH, SSE (TASK-1663)
Foundation for comment editing (PLAN-1662). No migration — comments.user_id
already exists (012_users.sql) but was never written or exposed.
- Populate user_id on create/reply: CreateComment takes an explicit userID
param (passed from currentUserID by the handlers, not via the request body
so it can't be spoofed). Expose user_id on models.Comment + all comment
SELECTs/scans. The workspace export path is left as-is — imported comments
keep NULL user_id (admin-only edit), matching the pre-identity fallback.
- Store.UpdateComment(id, body): replaces body + bumps updated_at; the
comments_fts_update trigger re-indexes.
- PATCH /workspaces/{ws}/comments/{commentID}: author-or-admin only
(canEditComment), rejects empty body. Editing is an authorship op, distinct
from delete (item editors). NULL user_id → admin-only.
- comment_updated SSE event: broadcast from the handler; added to the web
sse allowlist + ItemTimeline refresh set.
- web: api.comments.update(), Comment.user_id type.
Tests: author edits own (200), non-author non-admin (403), admin edits
anyone (200), empty body (400), NULL-user_id comment is admin-only.
Parent: PLAN-1662.
* fix(account): detach authored comments on account deletion per Codex review (round 1)
Now that TASK-1663 populates comments.user_id (FK to users.id),
DeleteAccountAtomic would fail on the FK for any user who authored a
comment. Null comments.user_id for the user before deleting the row —
comments live on in soft-deleted/other workspaces; the display-name
author is preserved and the comment just becomes admin-only to edit.
Regression test added.
|
||
|
xarmian
|
1b1068537c |
feat(tags): workspace tag enumeration endpoint + cross-collection filter (TASK-1653) (#658)
* feat(tags): workspace tag enumeration endpoint + cross-collection filter (TASK-1653)
Foundation for the tags feature (PLAN-1652 / IDEA-1649). The write path and
per-collection ?tag= filter already existed; this adds tag enumeration and a
verified cross-collection read so a single tag can group items of any type.
- store: dialect.JSONArrayElements unnests a JSON text-array column
(json_each on SQLite, jsonb_array_elements_text on Postgres);
Store.ListWorkspaceTags returns distinct tags + item counts, ordered by
count desc then tag asc, with the same collection/item ACL filters as
ListItems so counts never leak hidden items.
- server: GET /workspaces/{ws}/tags (handleListTags), respecting collection
visibility + guest item grants.
- models: TagCount{tag,count}.
- cli: client.ListTags + `pad tag list`.
- web: api.tags.list + TagCount type (items.list already forwards `tag`).
- tests: store-level (cross-collection aggregation, collection scoping,
non-nil-empty = empty, archived excluded) and handler-level (a Task + an
Idea sharing one tag; GET /tags counts + ordering).
Parent: PLAN-1652.
* fix(tags): count distinct items per tag, not tag occurrences per Codex review (round 1)
COUNT(DISTINCT i.id) so an item with duplicate tags (e.g. ["ux","ux"]) is
counted once — the write path doesn't enforce per-item tag uniqueness.
Adds a regression test.
|
||
|
xarmian
|
eeff78118b |
feat(insights): per-user layout customization + persistence (TASK-1634) (#645)
* feat(insights): per-user layout customization + persistence (TASK-1634)
Let users personalize the Insights surface, persisted per-user per-workspace:
toggle which metric cards show, and remember the window + collection filter.
Backend:
- migrations 064/043: user_report_layouts (user_id, workspace_id, config JSON,
PK(user_id,workspace_id), ON DELETE CASCADE) — dual-dialect.
- models.ReportLayout (hidden_cards/default_window/default_collections) +
ReportCardIDs/ValidReportWindow validation.
- store.GetReportLayout / SaveReportLayout (ON CONFLICT upsert, both dialects).
- GET/PUT /workspaces/{ws}/report/layout — per-user; PUT sanitizes window +
filters hidden_cards to the known card set. web client + TS type.
Frontend (Insights page):
- loads the saved layout, hydrates window/collections/hidden cards
- a "Customize" panel toggles each card (SvelteSet-backed); each section gated
on !hiddenCards.has(id); Totals always shown
- debounced auto-save, gated on a per-workspace `hydrated` flag so it never
saves during load or stomps another workspace's layout on switch
Single config per user (no named/multiple layouts — deliberate v1 scope).
Parent: PLAN-1628.
* fix(insights): save layout only on explicit user changes, not on load per Codex review (round 1)
The auto-save $effect ran once after hydration (loadLayout assigns reactive
state, then flips hydrated=true), firing a PUT /report/layout on mere page
view — which 401s on no-user/legacy-token sessions and bounces the user to
/login. Replace the effect with a scheduleSave() called only from explicit
handlers (toggleCard, selectWindow, toggleCollection, clearCollectionFilter);
hydration never saves. Also capture wsSlug at schedule time and drop the
pending save if the workspace changes mid-debounce, so A's edit can't land
on B.
|
||
|
xarmian
|
5dfc2921b2 |
feat(store): structured status-transition log + backfill (TASK-1637) (#637)
* feat(store): structured status-transition log + backfill (TASK-1637)
Add a status_transitions table capturing every item status change as a
structured, queryable row — written in the same tx as the item update and
never debounced — so the Reports surface (PLAN-1628) can reliably compute
the completed-throughput and cycle-time series.
- migrations/063 + pgmigrations/042: status_transitions table (dual-dialect),
indexed on (workspace_id, created_at) and (item_id, created_at)
- write-path hook in UpdateItemWithPreCheck records from→to on status change
- BackfillStatusTransitions: one-time startup replay parsing the historical
activities.metadata.changes blob (mirrors BackfillWikiLinks), gated on an
empty table; wired into cmd/pad/main.go
- models.StatusTransition + tests (capture, multi-hop, no-op, parser, backfill)
Spike (TASK-1629) found the activity log records status changes only as a
human-readable, debounce-coalesced metadata string — unusable for aggregation.
This is the foundation TASK-1630 (report aggregation) builds on.
* fix(store): record status transitions on item move too per Codex review (round 1)
MoveItemWithPreCheck rewrites fields outside UpdateItemWithPreCheck, so a
status-changing move override (pad item move ... --field status=done) was
not recorded in status_transitions, making the table non-canonical. Insert
the from→to row in the move tx as well, stamped with the target collection.
Adds move-path capture tests (status override + status-preserving move).
* fix(store): make status-transition backfill idempotent per Codex review (round 2)
The empty-table gate isn't atomic, so concurrent replays (a future
multi-replica Postgres deploy; single-instance today) could double-insert
historical rows and overcount reports. Give backfilled rows a deterministic,
activity-derived primary key ("bf_" + activity id) and a dialect-aware
conflict clause (ON CONFLICT DO NOTHING / INSERT OR IGNORE) so a re-run
no-ops instead of duplicating. Count only rows that actually land.
Write-path rows keep using a random newID(), so live data never collides.
* fix(store): accurate from_status under lock + document backfill caveats per Codex review (round 3)
1. from_status was read from the pre-lock `existing` snapshot. When no
precheck ran, a concurrent update (serialized behind the locks we hold)
could make it stale. Capture the status from a fresh in-tx read BEFORE
the UPDATE (reading after would see the new value and drop the hop).
Applied to both UpdateItemWithPreCheck and MoveItemWithPreCheck.
2. Backfill stamps historical rows with the item's current collection_id;
reconstructing the collection at each past status change would require
replaying move history. Documented as a best-effort, historical-only
caveat (exact for the common never-moved case; live write/move paths
stamp the collection at transition time).
* feat(store): track collection done-field + seed create-time transitions per Codex review (round 4)
1. Generalize capture from hard-coded "status" to each collection's done
field (DoneFieldKey: status, or BoardGroupBy field like stage/result for
hiring/interviewing). Add a field_key column recording which field the
row tracks (robust to later BoardGroupBy changes). Applied to update,
move, and backfill paths.
2. Seed a create-time "entered initial status" transition on CreateItem and
in the backfill (Pass 2), so an item created directly in a terminal value
still counts as a completion. Initial value reconstructed from the item's
earliest recorded change, else its current value.
Also: item_id FK is ON DELETE CASCADE so hard-deletes clean up transitions.
Tests cover non-status done-field, create-in-terminal, create-seed, and
cascade-on-delete; full store suite green.
|
||
|
xarmian
|
905876af04 |
feat(backlinks): cross-workspace wiki-links + request-independent ACL (Phase 2b) (#622)
* feat(backlinks): cross-workspace wiki-links + request-independent ACL (Phase 2b) Phase 2b of PLAN-1593 (TASK-1597). Completes the wiki-link reverse index by indexing and surfacing `[[workspace::REF]]` cross-workspace references. Builds on Phase 2a's title work (PR #621). Phase 3 (TASK-1596) owns the UI/MCP/CLI rendering changes. What changed - internal/store/backlinks_visibility.go (new): request-independent ACL helper `Store.ResolveBacklinksVisibility(userID, workspaceID, includeDeletedItems)`. Mirrors the role-determination + collection- merge logic from server.guestResourceFilterCore but doesn't depend on a request context, so cross-ws traversal can compute per-source- workspace ACLs without a `workspaceRole(r)` lookup. The Codex planning-round review caught the prior plan reusing the request- scoped helper as a hidden architectural cost; this is the resolution. - internal/server/server.go: guestResourceFilterCore refactored to delegate to the new store helper. Keeps the request-scoped wrapper signature stable for all existing handler call sites; only the internals move. - internal/links/extract.go: lift the Phase-2a workspace_ref emit gate. WikiLinkKindWorkspaceRef now flows through ExtractWikiLinks alongside ref and title kinds. parseBody recognition was already in place from earlier rounds. - internal/store/wiki_links.go: WikiLinkKindWorkspaceRef branch in replaceWikiLinks stores (target_workspace_id, target_ref) verbatim, resolving the slug→ID via new resolveWorkspaceSlugTx (with per-call cache so repeated `[[ws::X]]` in one body don't re-query). Unknown slugs persist with target_workspace_id=NULL — broken-link semantics, identical to existing ref/title patterns. - internal/store/wiki_links.go: new `Store.GetCrossWorkspaceBacklinks` enumerates accessible workspaces via Store.GetUserWorkspaces (which includes guest-only access — broader than membership query), then per-workspace computes visibility via ResolveBacklinksVisibility and runs the SQL backlinks query with the per-ws (FullCollectionIDs, GrantedItemIDs) predicate inline. Results sorted by updated_at DESC in Go, paginated globally. Per-workspace safety cap (offset+limit) prevents one workspace from dominating the global slice. - internal/store/wiki_links.go: new `Store.CountBacklinks` for same-ws pagination boundary detection. Needed so the handler knows where the cross-ws tier begins for pages 2+. - internal/models/backlink.go: new `SourceWorkspaceSlug string` (omitempty) field. Populated only by cross-ws rows; same-ws rows leave it empty so the existing wire shape is preserved. - internal/server/handlers_backlinks.go: union pagination across same-ws and cross-ws tiers. Same-ws first (matches the renderer's UI mental model — your own workspace's links at the top of the panel). Count-based slice math handles pages 2+ correctly when same-ws is exhausted. Tests - internal/links/extract_test.go: workspace_ref forms emit correctly (bare, display alias, mixed case, invalid-slug fallback to title). - internal/store/wiki_links_xws_test.go (new): six cross-ws scenarios plus a role-matrix test: - end-to-end cross-ws index + query - non-member sees nothing - guest with collection grant sees only that collection - guest with item grant sees only the granted item - unknown workspace slug → broken row, no query results - same-ws rows leave SourceWorkspaceSlug empty - ResolveBacklinksVisibility role matrix (admin/full member/guest with grants/non-member non-grant) Out of scope (Phase 3 / TASK-1596) UI rendering of cross-ws backlinks (workspace badge + workspace- prefixed ref), MCP `pad_item.action: backlinks` cross-ws fields, CLI display tweaks. PLAN-1593 / TASK-1597. * fix(backlinks): admin enumeration + cross-prefix ref fallback + unbounded perWsCap (Codex round 1) Three P2 findings from Codex round 1 against PR #622: Finding 1 — admin users miss cross-ws backlinks. `GetUserWorkspaces` returns only memberships + grant-only guest workspaces, but RequireWorkspaceAccess (middleware_auth.go:481) gives admins implicit access to every workspace. An admin querying for backlinks would silently miss links from workspaces they're not explicitly a member of. Fix: in GetCrossWorkspaceBacklinks, branch on user.Role: - admin → s.ListWorkspaces() (every non-deleted workspace) - non-admin → s.GetUserWorkspaces (memberships + grants) Stale user IDs return empty result rather than erroring. Finding 2 — cross-ws ref matching doesn't handle cross-prefix moves. Same-ws is immune because target_item_id is resolved at parse time and survives renames/moves; cross-ws resolves at query time, so a `[[other-ws::OLD-42]]` row written before the target moved from OLD→NEW collection wouldn't match a query under the NEW ref. Fix: in queryCrossWorkspaceBacklinksForWorkspace, dual ref-match clause: exact `LOWER(wl.target_ref) = LOWER(?)` OR `LOWER(wl.target_ref) LIKE LOWER('%-N')` where N is the item_number from the target ref. Pad prefixes are alphanumeric with no internal `-`, so trailing `-N` uniquely identifies the number suffix — no false positives like "TASK-142" matching "%-42" (LIKE anchors to the trailing literal). Finding 3 — per-workspace cap of 1000 silently broke pagination beyond offset>=1000. The 1000 ceiling was defensive paranoia; the correct math is offset+limit per workspace (worst case all rows come from one workspace and the global slice still needs that many). Fix: drop the 1000 ceiling. perWsCap = offset+limit unconditionally. For runaway offsets the per-workspace transfer cost is proportional; documented as a known characteristic (callers shouldn't be paging past offset=10000 anyway). Regression tests: - TestWikiLinks_CrossWorkspaceAdminSeesAllWorkspaces: admin sees cross-ws backlink without being a workspace member. - TestWikiLinks_CrossWorkspaceRefNumberFallback: move target to new collection, query under new ref, old-ref-stored row still surfaces. PLAN-1593 / TASK-1597. * fix(backlinks): honor OAuth/MCP token workspace allow-list (Codex round 2) Codex round 2 P1: cross-workspace backlinks bypassed the OAuth/MCP token's workspace allow-list (TASK-952). A token consented for workspace A but with the underlying user having access to B would still surface source rows from B via the cross-ws query — leaking data outside the token's consent scope. Fix: thread `allowedWorkspaceSlugs []string` through GetCrossWorkspaceBacklinks. Handler populates it from TokenAllowedWorkspacesFromContext(r.Context()): - nil → no token gate (PAT or pre-TASK-952 token, allow all) - "*" wildcard → allow all - explicit list → strict slug membership Workspace enumeration skips any source workspace whose slug isn't in the allowlist. The same-ws path is unchanged because RequireWorkspaceAccess already gated the target workspace against the allow-list (so we only reach this handler when the target IS in the list). Regression test in wiki_links_xws_test.go covers four shapes: nil, wildcard, target-only (blocks cross-ws), explicit source-workspace (allows cross-ws). PLAN-1593 / TASK-1597. * fix(backlinks): normalize limit at handler boundary (Codex round 3) Codex round 3 P2: the backlinks handler parsed ?limit=N but didn't normalize it before computing the same-ws/cross-ws pagination split. GetBacklinks and GetCrossWorkspaceBacklinks each clamp >300 internally, but the handler's 'remaining := limit - len(sameWs)' used the original (potentially huge) value. With ?limit=301 and more than 50 same-ws backlinks, the first page would mix cross-ws in before same-ws was exhausted, violating the documented tier order. Fix: clamp 'limit' to <=300 at the handler boundary, before any pagination math runs. PLAN-1593 / TASK-1597. * fix(backlinks): normalize same-workspace [[ws::REF]] to ref-kind (Codex round 4) Codex round 4 P2: `[[<current-ws>::TASK-1]]` was being indexed as a workspace_ref row with target_workspace_id = current workspace. But the same-ws GetBacklinks query requires target_item_id (workspace_ref rows leave it NULL), AND GetCrossWorkspaceBacklinks explicitly skips the target workspace — so the link rendered and navigated correctly in the UI but no backlink ever surfaced. The renderer's L307 short-circuits same-workspace fully-qualified form to behave identically to `[[REF]]`; the index must follow. Fix: in replaceWikiLinks, normalize a workspace_ref link to ref-kind when its slug resolves to the current workspace. The promotion canonicalizes the ref (via new links.CanonicalizeRef exported alias) so `[[ws::task-5]]` stores the same canonical shape as `[[TASK-5]]`. Tests: - TestWikiLinks_CrossWorkspaceSameWorkspaceQualifiedNormalized: same-ws fully-qualified `[[ws::REF]]` surfaces in same-ws backlinks and is absent from cross-ws backlinks. PLAN-1593 / TASK-1597. * fix(backlinks): same-ws qualified ref miss doesn't title-fallback (Codex round 5) Codex round 5 P2: my round-4 normalization was too aggressive. It promoted `[[<current-ws>::REF]]` to ref-kind and let the regular ref branch handle it — including the title-fallback path that runs on ref miss. But the renderer's same-ws qualified branch (markdown.ts:472-481) does NOT title-fallback: a ref miss in that path returns the wiki-link verbatim (broken). Only the bare `[[REF]]` path (markdown.ts:513) falls through to title lookup. So my normalization could create ghost backlinks for source bodies like `[[ws::ISO-9001]]` when an item titled "ISO-9001" exists but no ISO collection — the renderer renders broken text, but the index would point at the title-matching item. Fix: handle same-ws qualified refs inline at the top of the loop, BEFORE the switch dispatches. Insert as ref-kind row (resolved or NULL) and `continue` past the switch. Bypasses the title-fallback path entirely, mirroring the renderer's behavior. Regression test in wiki_links_xws_test.go pairs same-ws qualified miss (must NOT title-fallback) with bare ref miss (SHOULD title-fallback) to lock the asymmetry in. PLAN-1593 / TASK-1597. |
||
|
xarmian
|
8e7d4040fd |
feat(backlinks): server-side reverse index for [[...]] (Phase 1) (#620)
* feat(backlinks): server-side reverse index for [[...]] wiki-links (Phase 1)
First phase of PLAN-1593. Today [[REF]] is parsed only at render time
on the client and there's no way to ask "who links to TASK-5?" without
a full-text scan. This change adds a materialized reverse index
(item_wiki_links) that's written every time an item's content changes
and exposes it via REST + CLI.
Phase 1 covers ref-form links only (`[[TASK-5]]` / `[[TASK-5|Display]]`).
Phase 2 (TASK-1595) will extend to titles + cross-workspace; Phase 3
(TASK-1596) adds the web UI panel + MCP action.
What lands here:
* Migrations 061 (SQLite) and 040 (Postgres) create item_wiki_links
with partial indexes on target_item_id, (target_workspace_id, target_ref),
and target_title — the schema accommodates all 5 wiki-link forms
up-front so Phase 2 doesn't ALTER.
* internal/links/extract.go is the canonical parser. It strips fenced
and inline code regions before extracting [[...]] occurrences, so
example refs in docs / code blocks don't pollute the index. Phase 1
emits only WikiLinkKindRef rows; title and workspace_ref kinds parse
successfully but are gated out until Phase 2.
* internal/store/wiki_links.go (replaceWikiLinks + GetBacklinks +
helpers) handles write-time bookkeeping and the read query. Resolution
to target_item_id happens at parse time inside the same transaction
as the items INSERT/UPDATE, so partial state never lands. Broken refs
(target_item_id IS NULL) intentionally persist — they feed a future
broken-links report.
* internal/store/wiki_links_backfill.go + cmd/pad/main.go hook the
idempotent backfill into server startup. Existing items get indexed
on first boot after the migration; subsequent boots are near-no-ops
via an EXISTS short-circuit.
* internal/store/items.go is amended in two places: tryCreateItem
always calls replaceWikiLinks (empty content → no-op DELETE), and
UpdateItemWithPreCheck re-parses whenever input.Content was supplied.
* internal/server/handlers_backlinks.go serves
`GET /api/v1/workspaces/{ws}/items/{itemSlug}/backlinks` with
visibility + guest-grant filtering on the source items.
* internal/cli/client.go adds GetBacklinks; cmd/pad/main.go adds the
`pad item backlinks <ref>` command (registered in groups.go).
Behavior decisions (per PLAN-1593):
- code blocks excluded (fenced + inline)
- self-links filtered at query time (kept in storage)
- repeated mentions stored as separate rows by position
- ordering: source updated_at DESC, position ASC
Tests:
- internal/links/extract_test.go: 26 sub-cases covering ref/title/
workspace-ref discrimination, code-block exclusion (fenced + inline +
unclosed fence), position-is-byte-offset (UTF-8 safety), and edge
inputs.
- internal/store/wiki_links_test.go: 8 integration tests covering the
create/update/delete/self-link/broken-ref/repeated/code-block
scenarios plus backfill idempotence.
All pass. `make check` clean (lint + go test + web build).
Refs: TASK-1594, PLAN-1593, IDEA-1577
* fix(backlinks): visibility-aware pagination + case-insensitive refs per Codex review (round 1)
Two fixes from Codex code review:
P1 — GetBacklinks now takes a visibleCollectionIDs []string argument
that's applied INSIDE the SQL WHERE clause. Previously the handler
fetched LIMIT raw rows and filtered visible ones in Go, so a
restricted user asking for limit=50 could receive an empty page even
when later visible backlinks existed. Pushing visibility into SQL
makes LIMIT/OFFSET count visible rows.
nil → no restriction (owners, editors, root tokens)
[] → see nothing (returns early, no SQL)
[..] → AND s.collection_id IN (?, ?, ...)
Item-level guest grants still apply post-fetch — they're rare enough
that the residual page shrink is acceptable and pushing them into SQL
would balloon the query.
P2 — refPattern now accepts mixed/lowercase refs and parseBody
canonicalizes the prefix to uppercase at the single chokepoint.
Previously the renderer accepted `[[task-5]]` as a real link (its
REF_PATTERN is case-insensitive) but the indexer's ^[A-Z]... pattern
silently dropped it — divergent parsing on the same input. Storage
shape is canonical uppercase so the (workspace, prefix, number)
lookup against collections.prefix (also uppercase) has one shape.
New helper: canonicalizeRef("task-5") → "TASK-5".
Regressions:
internal/links/extract_test.go
+ TestCanonicalizeRef — helper unit tests
+ TestExtractWikiLinks_RefVsTitleFallback updated to assert
mixed/lowercase parses-as-ref-and-uppercases
+ edge-case test renamed from "lowercase ref" to "number-led
not a ref" (lowercase IS a ref now per Codex P2)
internal/store/wiki_links_test.go
+ TestWikiLinks_MixedCaseRefIndexed — `[[task-5]]` produces a
backlink row whose target_ref is "TASK-5"
+ TestWikiLinks_VisibilityAwarePagination — three sub-cases:
nil → all 3, visible-only limit=2 → 2 visible rows (not 1 with
hidden one consuming a slot), empty → 0
All call sites updated (8 in tests + 1 in handler).
`make check` clean (lint + tests + web build).
Refs: TASK-1594, PLAN-1593
* fix(backlinks): SQL-level item-grant filter per Codex review (round 2)
Round 1 fixed pagination for collection-level visibility but Codex
round 2 correctly flagged the same class of bug at the item-grant
layer: `visibleCollectionIDs` returns the UNION (full grants ∪
collections containing granted items), and the handler then
filtered each row's item-level visibility in Go AFTER fetching —
letting hidden rows in a granted-item's collection consume LIMIT
slots.
The refactor moves the precise predicate into SQL. New shape:
type BacklinksVisibility struct {
Unrestricted bool // admin / full-access member
FullCollectionIDs []string // direct collection grants
GrantedItemIDs []string // item-level grants
}
// SQL predicate when Unrestricted=false:
// AND (s.collection_id IN (?...) OR s.id IN (?...))
This matches `guestResourceFilter` (which returns the precise
primitives), so the handler now passes them straight through and
drops the post-fetch filter loop entirely. Pagination is correct
for guests, restricted members, and unrestricted users alike.
New test:
TestWikiLinks_ItemGrantPagination — guest with item-grant on ONE
item in an otherwise-hidden collection sees exactly that one item;
hidden siblings in the same collection do NOT leak in, and limit=2
returns 1 row (not silently shrunken).
Other call sites updated:
- TestWikiLinks_VisibilityAwarePagination → uses
BacklinksVisibility{FullCollectionIDs: ...} and
BacklinksVisibility{} for the no-access case.
- 8 existing tests → BacklinksVisibility{Unrestricted: true}.
- handlers_backlinks.go → no longer calls visibleCollectionIDs;
uses guestResourceFilter exclusively and skips the Go-side filter.
Verification:
- make check clean
- All TestWikiLinks_* pass
Refs: TASK-1594, PLAN-1593
* fix(backlinks): scan EXISTS into bool not int for Postgres parity (Codex round 3)
`SELECT EXISTS(...)` returns boolean on Postgres but integer 0/1 on
SQLite. Scanning into `int` happened to work on SQLite (the modernc.org
driver coerces) but would fail on Postgres — silently disabling the
backfill short-circuit there and meaning upgraded Postgres installs
wouldn't populate backlinks for pre-existing content until each item
got edited.
Fix: scan into bool. Both database/sql drivers in use (modernc.org/
sqlite and lib/pq) coerce their native representation into Go's bool,
so this single shape works on both engines.
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): allow CommonMark 0-3 space indented fences per Codex (round 5)
Round 5 flagged two edge cases in the code-stripping pass:
1. Multi-backtick inline code (``see [[X]]``) — traced through the
parser; my permissive close-on-next-backtick logic already covers
it correctly (range = [opener-start, after-closer-run]). Added
a regression test to lock this in:
TestExtractWikiLinks_CodeBlocksExcluded /
"multi-backtick inline code excludes ref"
2. Indented fenced blocks — CommonMark allows 0-3 leading spaces of
indentation before a fence opener (4+ spaces makes it an indented
code block, a different construct). My fencedCodeRanges only
matched fences at column 0, so ` ```\n[[X]]\n```` ` would
render as code in the UI but leak a false backlink. Fixed both
fencedCodeRanges (opener) and findFenceCloser (closer) to skip
up to 3 leading spaces, with a hard cap at 4 (which would be
indented-code, not a fence). Regression test:
TestExtractWikiLinks_CodeBlocksExcluded /
"indented fenced block (CommonMark 0-3 spaces)"
Not addressed:
- Round-4 escape-body parity finding. extract.go mirrors
renderMarkdown's regex (web/src/lib/utils/markdown.ts:300), which
is the actual render-time link parser; wikiLinksToMarkdown's more
permissive escape grammar is editor-serializer-side and the
renderer can't even consume its escaped output. Indexing what the
user actually sees as a link is the correct invariant.
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): tilde fences + strict closer lines per Codex (round 6)
Two CommonMark conformance gaps in the code-block stripping pass:
1. Tilde-fenced code blocks (~~~) were ignored. marked() treats them
the same as backtick fences, so a [[REF]] inside a tilde block
would render as code in the UI but leak as a false backlink.
Fixed by parameterizing fenceChar across fencedCodeRanges and
findFenceCloser, with separate handling for the backtick-specific
"no backtick in info string" rule (CommonMark §4.5).
2. Closer-line strictness — CommonMark requires the closing fence
line to contain only the fence + optional trailing spaces. The
previous accept-any-fence-prefixed-line check would terminate
a still-open fence prematurely on a line like ```not-closed,
leaking later refs in the still-rendered code block.
Refs reside in 4 new sub-tests under TestExtractWikiLinks_CodeBlocksExcluded:
- tilde fence excludes refs inside
- tilde fence with language tag
- mixed fence types don't pair
- closer-line strictness — backticks plus other text is not a closer
- closer-line strictness — trailing spaces OK
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): inline code closer must match opener length per Codex (round 7)
CommonMark §6.1 requires an inline-code span opened with N backticks
to close on a run of EXACTLY N backticks. The previous "close on next
backtick run of any length" logic would prematurely end the excluded
range on a stray single backtick inside a ``...`` span, leaking any
[[REF]] in the latter half of the code text as a false backlink.
Concrete failure case:
``has ` inside [[X-1]] and more``
→ old: range [0, 7], [[X-1]] indexed (bug)
→ new: range [0, end-of-closer], [[X-1]] excluded (correct)
Fix: track the opener-run length and scan only for matching-length
closer runs. Wrong-length runs in between are code text.
Two new regression sub-cases under TestExtractWikiLinks_CodeBlocksExcluded:
- inline code closer matches opener length — the main case
- single-backtick span unaffected by adjacent multi-backtick run —
asserts the opposite direction (opener=1 doesn't close on ``)
Not addressed:
- Re-flagged round-4/round-7 escape-body parity finding. extract.go
intentionally mirrors renderMarkdown's regex (markdown.ts:300), not
wikiLinksToMarkdown's more permissive escape grammar (markdown.ts:461).
renderMarkdown is the actual link parser at display time; its regex
rejects escaped-`]` bodies, so any link with an escaped `]` in its
body is NOT shown as a clickable link in the UI. Indexing it would
produce phantom backlinks the user can't see. The wikiLinksToMarkdown
permissive grammar is paranoid serialization that the renderer can't
consume — that's a pre-existing inconsistency in the editor pipeline,
not a backlinks bug.
make check clean (lint + tests + web build).
Refs: TASK-1594, PLAN-1593
* fix(backlinks): rune-align snippet end-edge to keep UTF-8 valid (Codex round 8)
The previous snippetAround() trimmed `start` to a rune boundary (so
the leading edge of the snippet was always at a valid codepoint) but
left `end` as a raw +40-byte clamp. When that landed in the middle of
a multi-byte rune — common around emoji or accented text — the
resulting slice was invalid UTF-8 and the JSON encoder would emit
replacement characters in backlink snippets.
Fix: same forward-advance pattern at the end as at the start.
Continuation bytes (10xxxxxx) get skipped until we land on a leading
byte. Going forward keeps the snippet anchored slightly past the
match rather than slightly before it, which is a small UX win
(emoji or accented text right after the link survives intact).
Regression test:
TestWikiLinks_SnippetIsValidUTF8 — pads body with enough 4-byte
emoji on each side that the ±40-byte window cuts through one;
asserts utf8.ValidString on the resulting snippet.
make check clean (lint + tests + web build).
Refs: TASK-1594, PLAN-1593
* fix(backlinks): inline code spans cross newlines, break on blank lines (Codex round 9)
CommonMark §6.1: an inline-code span can cross single newlines but
terminates at a blank line (a line containing no chars or only
whitespace, which ends the enclosing paragraph). My previous scanner
broke at every newline, so multi-line spans like
`pre
[[INSIDE-1]]
post`
would treat the opener as unclosed and leak [[INSIDE-1]] as a false
backlink. Fixed by:
1. The newline branch in the closer scan now peeks ahead via the
new isBlankLineAt() helper. Same-paragraph newlines are
traversed; blank-line breaks terminate the span unmatched.
2. isBlankLineAt() treats any line with only space/tab as blank
(mirroring CommonMark's blank-line definition).
Three new regression sub-cases under TestExtractWikiLinks_CodeBlocksExcluded:
- inline code spans single newline (CommonMark §6.1)
- inline code breaks at blank line (paragraph boundary)
- inline code breaks at whitespace-only blank line
Trade-off: a truly-unclosed inline backtick now consumes from the
opener up to the next blank line instead of just the rest of the
line. False-positive on wiki-links in that span, but the surface
area is small (unclosed backticks are rare in published prose) and
matches the renderer's behavior.
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): accept escaped wiki-link bodies per editor grammar (Codex round 10)
After 3 rounds of disagreement, capitulating on the escape-body parity
finding. My position was technically correct for the CURRENT
renderMarkdown behavior (which uses [^\]]+ and can't parse escaped-
bracket bodies), but the editor's wikiLinksToMarkdown grammar at
markdown.ts:461 explicitly produces such bodies — making the
renderer's regex the inconsistent half of the pipeline, not mine.
Mirroring the editor's grammar in the extractor makes the index
forward-compatible: when the renderer eventually gets fixed, no
change here is needed. The cost is a few "phantom" rows in the
interim (indexed links the renderer doesn't currently display as
clickable), but those are harmless and aligned with author intent.
Changes:
- wikiLinkPattern now uses `\[\[((?:\\.|[^\]\\])+)\]\]` — mirrors
markdown.ts:461 verbatim.
- New splitOnUnescapedPipe() helper — scans for the first `|`
that isn't preceded by `\`. Mirrors splitWikiBody at
markdown.ts:664.
- New unescapeWikiBody() helper — undoes `\]`, `\|`, `\\` escapes
in display text and key. Mirrors unescapeWikiBody at markdown.ts:657.
- parseBody() now uses both helpers — split on unescaped `|`,
unescape both sides.
Regression coverage:
- TestExtractWikiLinks_EscapedBodyChars (5 sub-cases): escaped `]`,
escaped `|`, escaped `\`, non-escape backslash passes through,
Position still points at opening `[[` despite escapes.
- TestSplitOnUnescapedPipe + TestUnescapeWikiBody: direct unit
tests for the helpers (round-trip safety vs the editor's
escape/unescape pair).
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): preserve display text verbatim per Codex round 11 P3
The previous parseBody trimmed the display side of [[X|Display]] but
the WikiLinkRef.Display contract promises verbatim storage and the
renderer at markdown.ts doesn't trim either. Trimming would silently
diverge on padded display text like [[TASK-1| spaces ]] (renderer
keeps the spaces, extractor stripped them).
Fix: drop TrimSpace from the suffix half of the split. Keep trimming
the key/ref side because refPattern is anchored — a leading or
trailing space in the key would force the body to fall through to
the title kind even though the renderer resolves it as a ref.
Regression test:
TestExtractWikiLinks_EscapedBodyChars / "display text preserved
verbatim (no TrimSpace)"
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): distinguish empty display override from no-pipe per Codex round 12
[[REF|]] (explicit empty display) and [[REF]] (no display) are distinct
shapes in the editor: splitWikiBody returns displayOverride="" for the
former, null for the latter, and the renderer uses `displayOverride ??
title` (nullish coalescing, NOT empty-string fallback) so "" is
preserved. The previous extractor collapsed both into display_text=NULL,
violating verbatim-display preservation for the empty-string edge case.
Fix:
- WikiLinkRef gains a HasDisplay bool. parseBody sets HasDisplay=true
iff splitOnUnescapedPipe found a pipe; downstream uses HasDisplay
(not Display!="") to decide whether to persist the override.
- replaceWikiLinks in store: NullString.Valid is keyed off HasDisplay.
display_text='' for explicit empty, NULL for no override.
Regression coverage:
- internal/links/extract_test.go:
"explicit empty display override is distinguished from no pipe"
- internal/store/wiki_links_test.go:
TestWikiLinks_EmptyDisplayDistinct (two-source assert: NOT NULL
for [[REF|]], NULL for [[REF]])
make check clean.
Refs: TASK-1594, PLAN-1593
* fix(backlinks): pointer-typed DisplayText to preserve empty distinction over JSON (Codex round 13)
Round 12 added HasDisplay on the parser side and made the store
preserve display_text='' vs NULL on the DB row, but the wire model
collapsed the distinction at JSON-serialization time:
DisplayText string `json:"display_text,omitempty"`
`omitempty` drops empty strings, so [[REF|]] (empty override) and
[[REF]] (no override) serialized identically on the API and CLI JSON
output. The end-to-end goal of round 12 wasn't reached.
Fix: change DisplayText to *string. nil → no override (field omitted
from JSON via omitempty), pointer to "" → explicit empty override
(field present with empty value). The store's NullString.Valid drives
the assignment, so the SQL round-trip matches the JSON shape.
Knock-on: the CLI's `pad item backlinks` now dereferences the pointer
and prints both populated and empty overrides ("displayed as: ").
Regression coverage:
- TestWikiLinks_EmptyDisplayDistinct extended to assert
withBL.DisplayText is non-nil-pointing-at-"" and noBL.DisplayText
is nil after a GetBacklinks round-trip.
make check clean.
Refs: TASK-1594, PLAN-1593
|
||
|
xarmian
|
0c5ec04fac |
feat(admin): extend user list with aggregations + sort/filter (TASK-1544) (#599)
* feat(admin): extend user list with aggregations + sort/filter (TASK-1544)
GET /admin/users now returns per-user workspace_count, storage_bytes,
last_write_at, and a computed status pill (disabled / no-workspace /
inactive / active, with documented precedence). Adds sort and filter
knobs so the table can scale beyond the existing fixed offset/limit.
Store layer:
- AdminUserSearchParams gains Role, Sort, Order, ActiveWithinDays,
HasWorkspaces, Disabled. Pointer types where tri-state ("no filter"
vs. "filter to false") matters.
- AdminUserListEntry wraps models.User with WorkspaceCount, StorageBytes,
Status — returned in AdminUserSearchResult.Users.
- SearchUsers SQL rewritten: LEFT JOIN against grouped subqueries so
one user owning N workspaces with M attachments each still produces
exactly one row (no aggregation explosion). Both subqueries filter
deleted_at IS NULL to match WorkspaceStorageUsage's existing
definition. Allow-listed sort clause prevents injection.
- computeAdminUserStatus exported for unit tests; precedence locked in
by TestComputeAdminUserStatus.
- TestSearchUsersAggregations covers workspace_count + storage_bytes +
status across a three-user fixture and each new filter/sort knob.
Model + scanner:
- models.User gains LastWriteAt. userColumns + scanUser updated; the
legacy callers (GetUser, ListUsers, etc.) inherit the new field for
free via the shared scanner.
Handler:
- handleAdminListUsers accepts the new params: role, disabled,
has_workspaces, active_within_days, sort, order. Tri-state bools
only fire when the query param is present. Response now embeds
workspace_count / storage_bytes / last_write_at / status.
Part of PLAN-1542. Frontend consumption lands in T1548 (cheap columns)
and T1549 (sort/filter UI).
* fix: address Codex review on TASK-1544
- Tri-state bool parsing in handler now uses strconv.ParseBool — accepts
the canonical truthy/falsy variants ("True"/"TRUE"/"t"/"1" and the
parallel falses), and silently ignores garbage values rather than
treating them as false. Closes the "disabled=TRUE silently means
enabled-only" surprise.
- SearchUsers count query no longer joins the storage aggregation when
HasWorkspaces isn't an active filter. The page query still needs both
joins (the row carries the data), but a typical "give me a count"
call no longer scans every live attachment. The workspace_count join
remains conditional on HasWorkspaces filtering.
Status threshold (>30d vs >=30d): the documented spec and impl both say
">30d" — no change.
|
||
|
xarmian
|
905baaa010 |
feat(oauth): backfill session.Extra into oauth_connections + switch read path (TASK-1522) (#583)
* feat(oauth): backfill session.Extra into oauth_connections + switch read path (TASK-1522) Phase C1 for PLAN-1519. Seeds existing OAuth grant chains into the new connection tables (Phase A) and switches /console/connected-apps to read from them, retiring the session.Extra parse on the read path. Backfill (internal/store/oauth_connections_backfill.go) - Walks oauth_access_tokens + oauth_refresh_tokens to find every distinct request_id chain (including refresh-only chains). - Picks the newest token row per chain — its session.Extra drives the seeded shape, so a chain whose user re-scoped recently reflects the latest decision. - Maps session.Extra shapes to the new tables per IDEA-1517 §2: no key → all_current=1; ["*"] → all_current=1; explicit slugs → all_current=0 + one join row per slug (added_by='user'). - Resolves slugs → workspace IDs; unresolved slugs (deleted / renamed workspace) are counted + logged at WARN, not fatal. - Idempotent on every INSERT (OR IGNORE / ON CONFLICT DO NOTHING) so re-running on every startup is a cheap no-op once stable. - Returns a BackfillOAuthConnectionsResult so the startup log reports chains_seen / connections_created / workspaces_added / unresolved_slugs — operators see fresh work and notice drift. Read-path rewrite (internal/store/connected_apps.go) - ListUserOAuthConnections projects AllowedWorkspaces from GetOAuthConnectionAccess (oauth_connection_workspaces JOIN workspaces) instead of parsing session.Extra strings. - Hydrates Name + MayCreate + AllCurrent + IncludeFuture from oauth_connections so Phase D's mutation UI has them. - Defensive fallback for chains without an oauth_connections row (any leftover the backfill missed): treats as legacy "any workspace, default-on flags" so the connection still renders. Backfill at startup keeps this branch unreachable in production. - Retires parseAllowedWorkspacesFromSession; the new extractAllowedWorkspacesFromSessionExtra helper in oauth_connections_backfill.go is the only consumer of the session.Extra shape on the store side. Model (internal/models/connected_apps.go) - Adds Name / MayCreateWorkspaces / AllCurrentWorkspaces / IncludeFutureWorkspaces. AllowedWorkspaces semantics stay stable (nil = "any"; explicit slugs = chip list) so the existing DTO + frontend continue working unchanged. Phase D exposes the new fields on the wire. Startup wiring (cmd/pad/main.go) - After srv.SetOAuthServer / SetClaimSecret, run the backfill once. Non-fatal on error (partial state is consistent and the next run completes). Quiet at the Debug level on steady-state re-runs; INFO when fresh work landed. Tests - 8 BackfillOAuthConnections cases: empty DB, pre-TASK-952 (no key), wildcard, explicit list, mixed resolvable/unresolved slugs, multi-row chain newest-row-wins, refresh-only chain, idempotent re-run (verified via post-run row count). - TestExtractAllowedWorkspacesFromSessionExtra replaces the retired parseAllowedWorkspacesFromSession test — covers all three IDEA-1517 §2 input shapes + malformed/non-array defensive cases. - TestListUserOAuthConnections_DeduplicatesChain + TestHandleListConnectedApps_DTOShapeAndAuditEnrichment updated to call BackfillOAuthConnections (the production startup hook) before asserting on AllowedWorkspaces — mirrors the real-world flow now that the read path no longer parses session.Extra inline. Parent: PLAN-1519. * fix(oauth): backfill counters reflect actual new rows per Codex review (round 1) PR #583 Codex review round 1 flagged that the backfill counters over-report on steady-state restarts: - wasFreshlyInserted compared updated_at vs created_at — true for every untouched existing row, so every restart counted every pre-existing connection as "created." - slugsAdded++ ran after AddConnectionWorkspace regardless of whether the INSERT OR IGNORE / ON CONFLICT DO NOTHING hit an existing row. Net effect: startup logs "backfill complete" with non-zero counts on every restart instead of the intended quiet "no-op" path — making real fresh work indistinguishable from steady-state. Fix: probe existence BEFORE the insert on both sides. - backfillOneChain reads GetOAuthConnection first; only sets created=true and runs insertOAuthConnectionIfAbsent on a miss. - Per-slug: IsConnectionWorkspaceAllowed pre-check; skip + don't increment when the row already exists. Two cheap PK / indexed lookups per chain. Pre-Phase-C deployments have small chain counts so the added cost is well below the scan already running. Removed the now-unused wasFreshlyInserted helper. Added an assertion in TestBackfillOAuthConnections_Idempotent that both ConnectionsCreated and WorkspacesAdded report 0 on the second run — the regression guard for this exact finding. Parent: PLAN-1519. * fix(oauth): backfill skips slug re-seed on existing rows per Codex review (round 2) PR #583 round 2 caught that the round-1 fix protected the parent oauth_connections row from re-seed but left the join table mutable from stale session.Extra: When a user removes a workspace from their connection's allow-list via Phase D's mutation UI (RemoveConnectionWorkspace), the next server restart would re-run the backfill, find the parent row intact, and re-INSERT the removed slug from the original session.Extra. The user's removal would silently revert every restart. Fix: backfill is a one-shot seed. Once the parent row exists, the new tables are authoritative — legacy session.Extra is frozen reference data, not a reconciliation source. The slug loop only runs when we just inserted a fresh parent row. Added TestBackfillOAuthConnections_DoesNotResurrectRemovedWorkspace as the regression guard: seeds two slugs, removes one, runs backfill again, asserts the removed slug stays gone and the kept slug is untouched. Parent: PLAN-1519. * fix(oauth): atomic per-chain backfill transaction per Codex review (round 3) PR #583 round 3 caught that round 2's "only seed slugs on fresh parent" gate introduced a permanent-partial-state risk: if the process crashes (or AddConnectionWorkspace errors) between inserting the parent row and finishing the slug loop, the next backfill sees created=false, short-circuits the slug seeding, and leaves the connection permanently scoped to a partial allow-list. Fix: per-chain transaction. Parent insert + every slug insert land in one BEGIN/COMMIT pair; any mid-loop failure rolls everything back. The next backfill then sees the chain as un-seeded and retries from scratch — preserving both round 2's "no-resurrection of user-removed slugs" (existence probe inside the tx) and round 3's "no permanent partial seed" (atomic commit). Scope: per-chain (small tx), not whole-backfill. The original no-transaction rationale was about lock-hold duration across thousands of chains; that doesn't apply at chain granularity (one parent + a handful of join rows = sub-millisecond hold). Removed the now-unused insertOAuthConnectionIfAbsent helper; the INSERTs live inline within the transaction. Added TestBackfillOAuthConnections_AtomicOnMidLoopFailure as the regression guard: forces a mid-loop INSERT failure via a duplicate slug in session.Extra (which violates the join table's PK on the second insert), asserts the parent row rolled back, then runs a clean retry and verifies full seed completion. Parent: PLAN-1519. * fix(oauth): surface store errors from backfill + list path per Codex review (round 4) PR #583 round 4 caught two silent-fallthrough paths that could leak partial/incorrect state instead of failing loudly: 1. Backfill slug loop: GetWorkspaceBySlug errors were treated the same as "workspace not found" — both incremented slugsMissed and continued. A real I/O error mid-loop would commit a partial allow-list, and the next backfill's parent-exists short-circuit would make that partial scope permanent. Fix: distinguish (nil, nil) "not found" from (nil, err) "real failure" — return the error so the per-chain transaction rolls back and the next run retries cleanly. 2. ListUserOAuthConnections hydration: GetOAuthConnectionAccess and GetOAuthConnection errors collapsed into the "no oauth_connections row" defensive-fallback branch, returning the legacy "any workspace, default-on flags" shape. On a real store failure that silently broadens a user's scope — e.g. a connection the user explicitly removed a slug from would render as "Any workspace" until the store recovered. Fix: surface store errors from both calls; the defensive fallback path is now exclusively for HasConnection=false, not for error masking. Both findings tighten the failure mode from "silently emit broadened/partial state" to "surface the error so retries happen against accurate data." Existing tests cover the happy paths; the failure paths are exercised by I/O errors against the same store interfaces (no new test added — the change is "return err instead of swallow it" and the assertion of NOT swallowing is the diff itself). Parent: PLAN-1519. |
||
|
xarmian
|
e59d3904c9 |
feat(server): refuse to mark item terminal while it has open children (IDEA-1494) (#571)
* feat(server): refuse to mark item terminal while it has open children (IDEA-1494)
Server-side guard inside handleUpdateItem that rejects a non-terminal →
terminal done-field transition when the item still has at least one
non-terminal child. Returns HTTP 409 with code=open_children plus a
structured details payload listing each blocking child's
{ref, title, status, collection_slug} so MCP-driven agents can
self-recover (ship the listed children, then retry) and the CLI can
render the same list verbatim.
Escape hatch: `--force` on `pad item update` / `pad item bulk-update`
and `force: true` on the MCP pad_item.action: update / bulk-update
inputs both forward into the same ItemUpdate.Force transport field
the handler consumes before any store mutation.
Trigger conditions are tight: the PATCH must change the done-field key
(resolved via TerminalValuesForDoneField against the parent's schema +
settings) AND the new value must be terminal AND the current value
must NOT already be terminal. Terminal → terminal and no-op terminal
transitions bypass the guard; only entering the terminal set is gated.
Per-child evaluation uses the child's own collection schema so
hierarchical workspaces with custom typed collections work without
extra plumbing.
Tests cover: rejection with one open child (with mutation-safety
assertion on the parent), no children, all-terminal children, --force
override, no-op terminal → terminal, terminal → terminal,
non-terminal → non-terminal, custom collection terminal_options
honored, and a parent task (not a plan) — IDEA-1494 optional extra #3.
MCP coverage asserts --force round-trips through both ExecDispatcher
and HTTPHandlerDispatcher and is omitted when force=false.
* fix(server): open-children guard round 2 — visibility, MCP pass-through, TOCTOU (IDEA-1494)
Three Codex round-1 issues, each fixed with the recommended shape:
P1 — visibility leak. The 409 response previously listed every blocking
child by ref/title/status, including children in collections the caller
couldn't see. The INVARIANT still evaluates against ALL children (it's a
data-integrity gate — a restricted user must not be able to close a
parent whose blockers they can't see), but the response payload now
filters to caller-visible children only. Hidden blockers surface as a
new `details.hidden_blocker_count` field plus an alternate human message
when every blocker is hidden ("blocked by N open children you don't
have access to"). Mirrors the visibility helpers (`visibleCollectionIDs`
+ `isItemVisibleToGuest`) used by the per-parent progress endpoint so
the two paths can't drift.
P2 — MCP code/details pass-through. The HTTP classifier was collapsing
409 into the generic `conflict` code and dropping `details`; the stdio
classifier was matching the human "cannot " message against the
validation regex and surfacing `validation_failed`. Both now surface
`open_children` with the structured details intact:
- HTTP: classifyHTTPStatusKind's 409 branch extracts the upstream
code; any non-empty, non-"conflict" code is passed through with
its `details` RawMessage. Generalizes beyond open_children — any
future structured 409 from a handler gets the same treatment.
- Stdio: the CLI writes a `pad-error: {json}\n` marker line on
stderr before the human-readable block (single source of truth for
both views), and classifyExecError detects the marker and lifts
the envelope verbatim. Marker is duplicated as a const between
internal/cli and internal/mcp to avoid pulling the cli package
into the classifier just for one string.
A new ErrOpenChildren error code constant + `Details json.RawMessage`
field on ErrorPayload back the wire shape.
P2 — TOCTOU. The guard previously ran in the handler before the store
transaction began; a concurrent child insert / child status flip could
slip between the children-list read and the parent's UPDATE. Fix:
- New `Store.UpdateItemWithPreCheck(id, input, precheck)` runs the
caller's invariant check inside the same tx, after acquiring the
workspace seq lock AND a new parent-children advisory lock keyed
on the parent ID. UpdateItem is now a thin wrapper passing nil.
- Every UpdateItem unconditionally acquires the parent-children
advisory lock for its own parent (if any) AND for itself-as-parent,
in a fixed order (parent first) so two updaters touching the same
parent always grab that key before the more-specific one — no
AB/BA deadlock.
- New `GetChildItemsTx` reads via the caller's tx; on Postgres the
advisory lock provides the snapshot guarantee (DISTINCT precludes
`FOR UPDATE`), on SQLite the global BEGIN IMMEDIATE write lock
serializes all writers.
- Handler now passes a precheck closure into UpdateItemWithPreCheck
at all three call sites (collab-snapshot path, applier-direct-write
path, main path). The guard's openChildrenGuardError sentinel is
unwrapped after each call so the 409 surfaces cleanly.
Tests:
- TestOpenChildrenGuard_VisibilitySanitizesPayload — restricted
editor sees parent + visible child, hidden child contributes to
hidden_blocker_count, no leak of ref/title/slug.
- TestOpenChildrenGuard_AllBlockersHiddenSurfaceGenericMessage —
open_children=[], hidden_blocker_count>0, message mentions "you
don't have access to."
- TestOpenChildrenGuard_TOCTOURace — 8 iterations of a child-flip
racing a parent-terminal update; asserts the forbidden outcome
(parent=completed AND child=open) never occurs.
- TestClassifyHTTPStatus_OpenChildrenPreservesCodeAndDetails +
inverse generic-409 test.
- TestClassifyExecError_OpenChildrenMarkerLiftsStructuredPayload +
no-marker-falls-through inverse.
* fix(server): open-children guard round 3 — 7 Codex findings closed (IDEA-1494)
P1 — visibility fail-closed. The handler was swallowing
visibleCollectionIDs errors, leaving visIDs==nil which the guard
treats as unrestricted, leaking hidden-child metadata. Now surfaces
the error as 500 BEFORE installing the precheck. Test:
TestOpenChildrenGuard_VisibilityLookupErrorFailsClosed closes the
store DB and asserts no 409+children leak.
P1 — link mutations acquire the advisory lock. SetParentLink,
ClearParentLink, CreateItemLink (when link_type ∈ childLinkTypes via
new isChildLinkType helper), DeleteItemLink (same condition), and
RestoreItem now take `pad:parent-children:<id>` in canonical sorted
order via new AcquireParentChildrenLocks helper. SetParentLink locks
BOTH old and new parents (re-parenting case). Race test
TestOpenChildrenGuard_LinkMutationRace asserts the forbidden
"link-committed-before-parent-flip AND parent flip succeeded" never
occurs by comparing link.created_at to parent.updated_at. Documented
semantics: status-wins + link-after-commit is legal under the
invariant "no open children EXIST AT THE MOMENT of transition" —
the post-condition variant ("no open child may EVER attach to a
terminal parent") is intentionally deferred.
P1 — MoveItem bypass closed. New MoveItemWithPreCheck mirrors
UpdateItemWithPreCheck — acquires workspace seq lock + parent-children
locks, re-reads in tx, runs caller precheck. handleMoveItem builds
the same guard closure using the DESTINATION schema for done-field
resolution (conservative — honors the schema the item moves INTO).
CLI gains `pad item move --force`, client gains MoveItemWithForce
that appends `?force=true` to the move endpoint. MCP catalog +
mapItemMove forward `force` through the route mapper. Tests:
TestOpenChildrenGuard_MoveItem_RejectsTerminalWithOpenChildren and
…_ForceOverrides.
P2 — pre-tx field-read TOCTOU. UpdateItemWithPreCheck and
MoveItemWithPreCheck now re-read the item via new getItemTx INSIDE
the tx (after locks) and pass that fresh snapshot to the precheck
closure; the precheck classifies the transition against the in-tx
view, not the handler-side pre-tx capture. Handler precheck closure
swaps `currentFieldsJS` from the in-tx snapshot. Test:
TestOpenChildrenGuard_PrecheckReadsInTxSnapshot stages a between-load
status mutation and asserts the precheck observes the post-mutation
fields.
P2 — bulk-update carries structured errors. cmd/pad/main.go's
updateFailure struct extended with Code + Details
(json.RawMessage). When client.UpdateItem returns *cli.APIError, the
row preserves the structured envelope. Human-text output also
renders the open-children list inline. Chose JSON-envelope route
over per-row stderr markers because bulk-update already produces a
structured envelope and ExecDispatcher returns stdout verbatim on
exit-0 — no classifier change needed. Test:
TestBulkUpdateStructuredFailuresCarryOpenChildrenDetails confirms
the wire shape the CLI lifts.
P3 — marker hardening. Marker bumped to versioned form
`pad-structured-error/v1:` (was `pad-error:`). cli.StructuredErrorMarker
+ mcp.structuredErrorMarker kept in lockstep with cross-references.
mcp.allowedStructuredErrorCodes whitelists known codes (currently
just open_children); unknown codes fall back to regex classification.
Marker must start the line after whitespace trim (embedded markers
ignored). Last-marker-wins to defeat pre-emption attacks. Tests:
TestClassifyExecError_{UnknownStructuredCode,OldMarkerVersion,
MarkerEmbeddedMidLine,LastMarker}.
P3 — soft-deleted collection schemas honored. New GetCollectionAnyState
mirrors childrenDoneFiltersForParent's inclusion rule; guard uses it
so a child still attached to a soft-deleted collection is evaluated
against ITS schema (custom terminal_options) instead of the default-
status fallback (which would mis-classify and false-block). Test:
TestOpenChildrenGuard_SoftDeletedCollectionSchemaHonored seeds a
custom collection, soft-deletes it while a child remains, and
asserts the terminal status is correctly recognized.
Comprehensive store-mutation audit results recorded in the PR
description (every method touching items.fields / items.collection_id
or item_links).
* fix(server): open-children guard round 4 — multi-parent locks, enum parity, PATCH atomicity (IDEA-1494)
Four Codex round-3 (blast-radius lens) findings, each fixed with the
recommended shape.
P1 — multi-parent lock set. acquireParentChildrenLocksForUpdate and
RestoreItem previously used `LIMIT 1` against item_links, so a child
with BOTH a `parent` link to P1 AND an `implements` link to P2 only
locked one of them. The other parent's open-children precheck could
race against the child's status flip and miss it.
Fix: new listParentChildLockKeys helper runs the same query
GetChildItems' inclusion rule uses (childLinkTypes), returns ALL
distinct parent target_ids, and feeds them into the canonical
multi-lock helper. Both UpdateItemWithPreCheck and RestoreItem now
acquire locks on {self} ∪ {all-parents-via-childLinkTypes}. Test:
TestOpenChildrenGuard_MultiParentChildLocksAll races a child status
flip against terminal-updates on both parents simultaneously.
P2 — lock-order asymmetry. The pre-fix codebase had multiple lock-
acquisition shapes: parent-then-self in acquireParentChildrenLocksForUpdate,
single-key in RestoreItem / CreateItemLink / DeleteItemLink /
ClearParentLink, and a sorted multi-key in SetParentLink. Two
concurrent callers using different ad-hoc orderings could AB/BA
deadlock.
Fix: removed the per-call-site AcquireParentChildrenLock helper
entirely. Every site now goes through AcquireParentChildrenLocks
(the canonical sorted multi-lock helper) — including ones that need
only one ID (the variadic call still sorts a one-element slice).
The helper's doc comment explicitly states the contract: "Ad-hoc
single-key acquisition outside this helper is FORBIDDEN — two call
sites taking distinct keys in different orders WILL deadlock."
Test: TestOpenChildrenGuard_NoDeadlockUnderReverseOrderConcurrency
runs reverse-order re-parents with a 5-second timeout; assertion
fails on hang.
P2 — HTTP/stdio code-surface parity. Round 2's HTTP pass-through
("any non-conflict upstream code") silently widened the ErrorCode
enum beyond stdio's allow-list (`open_children` only). Agents saw
different code surfaces depending on which dispatcher delivered
the response.
Fix: HTTP 409 branch in classifyHTTPStatusKind now consults the
same allowedStructuredErrorCodes whitelist stdio does. Codes
outside the set collapse to ErrConflict (no details), matching
what stdio does for an unknown-code structured marker. Doc on
allowedStructuredErrorCodes updated to make the dual-consumer
contract explicit: "Adding a new structured code is a TWO-WAY
change." Tests:
TestClassifyHTTPStatus_UnknownConflictCodeFallsBackToErrConflict
and TestStructuredErrorCodeParityAcrossTransports.
P3 — PATCH atomicity. A combined PATCH with `parent` + `status=terminal`
on an item with open children used to commit the parent-link change
INLINE (before the guard ran) and then reject the field write.
Caller saw 409 but the parent had already moved.
Fix: parent-link mutation is now DEFERRED — captured into outer-
scope vars during fields validation, executed AFTER
UpdateItemWithPreCheck succeeds. A guard rejection returns before
the link write block, so on rejection the link is untouched.
Documented choice: "reorder, don't tx-wrap" — wrapping SetParentLink
into the same store tx would require threading a *sql.Tx through
the SetParentLink API (which is also called from the
handler_item_links path); reordering is the smaller surgery and
gives the correct outcome on the failure direction. A residual
window remains in the OTHER direction (field write commits, link
write fails) — not made worse by the reorder, and called out
inline for a future tx-wrap pass.
Test: TestOpenChildrenGuard_PatchAtomicRejectionPreservesParentLink
sets up target → oldParent → openChild, sends PATCH {parent=newParent,
status=completed}, asserts 409 AND target.parent_link still points
at oldParent.
* fix(server): open-children guard — emit details.open_children as [] not null on hidden-only rejection (IDEA-1494)
|
||
|
xarmian
|
ec71903be7 |
feat(store): JSONB NOT NULL hardening on items/views + handler shape validation (IDEA-1486+1488) (#566)
* feat(store): NOT NULL hardening on items.fields/tags + views.config (IDEA-1486) Paired ship of IDEA-1486 (sibling-table JSONB NOT NULL hardening) and IDEA-1488 (handler-layer shape validation for ViewUpdate/CollectionUpdate). Generalizes the IDEA-1484 / collections.settings precedent (PR #562) to the remaining nullable JSON columns and closes the shape-validation gap that NOT NULL alone doesn't cover. Schema layer (IDEA-1486 floor): - migrations/056_items_jsonb_not_null.sql: rebuild items with fields TEXT NOT NULL DEFAULT '{}' and tags TEXT NOT NULL DEFAULT '[]', preserving all 7 indexes, recreating the 3 items_fts triggers, and rebuilding the FTS5 index. Foreign-keys-off / on bookends are lifted outside the IDEA-1485 atomic-tx wrapper. - migrations/057_views_config_not_null.sql: rebuild views with config TEXT NOT NULL DEFAULT '{}'. - pgmigrations/035 + 036: SET NOT NULL + SET DEFAULT on the three JSONB columns. Split per-table to mirror the SQLite per-table file granularity. Store layer (IDEA-1486 floor): - items.go UpdateItem and views.go UpdateView normalize "" -> "{}" / "[]" before writing. Same boundary pattern as CreateItem and the IDEA-1484 precedent at collections.go:248. - export.go ImportWorkspace coerces empty-string AND malformed JSON at import time on items.fields, items.tags, and collections.settings. Malformed input is coerce-and-log via slog.Warn (length only, never raw value) so legacy bundles don't fail-stop on one bad row. - remapFieldIDs early-returns "{}" on empty input so the second-pass UPDATE can't write "" verbatim. - Migrated the existing fmt.Printf at export.go:329 to slog.Warn for consistency. Handler layer (IDEA-1488 ceiling): - ViewCreate / ViewUpdate UnmarshalJSON via flexJSONToString with new ErrInvalidConfigType sentinel. - CollectionCreate / CollectionUpdate UnmarshalJSON with new ErrInvalidSettingsType sentinel. - handlers_views.go and handlers_collections.go surface both sentinels as 400 with the domain-level message (mirrors the BUG-1144 precedent at handlers_items.go:641). Tests: - internal/store/items_views_jsonb_test.go: store-coercion + import coercion + log-and-coerce-on-malformed + SQLite schema introspection (7 indexes + 3 FTS triggers + items_fts virtual table survival) + Postgres NOT NULL enforcement + migration re-apply idempotency + item_links round-trip after rebuild. - internal/server/handlers_views_collections_jsonb_test.go: PATCH/POST flexible-shape coverage for views.config and collections.settings, including domain-level 400 message assertions that the response does not leak Go unmarshal internals. Refs: IDEA-1486, IDEA-1488, IDEA-1484 (precedent), IDEA-1485 (substrate). * fix(store,models): codex R1 follow-ups for IDEA-1486 / IDEA-1488 Three concrete defects surfaced by codex R1 against the initial paired ship. All three close holes that defeated parts of the original contract. P1.1: migration 056 missed the playbook invocation_slug unique index. - migrations/056_items_jsonb_not_null.sql: recreate the partial UNIQUE index idx_items_invocation_slug_per_collection from migration 054 verbatim after the other 7 indexes. Without it, the application-layer pre-check in handlers_items.go:checkUniqueFields would be a TOCTOU race with no DB-level guard — the original index that 054 explicitly added as the actual uniqueness backstop would be silently dropped during the items rebuild. - items_views_jsonb_test.go: the schema-introspection test now asserts 8 indexes, not 7. Verified via `grep -rn "ON items(" migrations/` that no other items-touching indexes were missed. P1.2: flexJSONToString didn't validate inner content of JSON-encoded strings. Pre-fix, `{"config": "[]"}` / `{"settings": "not json"}` / `{"fields": "[]"}` / `{"tags": "{}"}` slipped past the shape validators because the `case '"'` branch unmarshalled the envelope and returned the inner string verbatim — bypassing the whole point of IDEA-1488. - models/item.go: after unmarshalling the JSON-encoded string, validate that the trimmed inner content's first byte matches expectedStart ('{' / '[') AND parses as JSON. Empty inner strings still pass through to the store-layer empty-string coercion (IDEA-1486 floor), so legacy "" → default normalization is preserved. - The pre-existing ItemUpdate fields/tags path inherits the same tightening because it routes through this helper — covered by new test file handlers_items_jsonb_inner_shape_test.go. - Parallel handler tests for views.config and collections.settings added to handlers_views_collections_jsonb_test.go. P2: coerceJSONForImport accepted JSON null as well-formed. - store/export.go: json.Unmarshal("null", &m) returns err=nil with m staying nil; the prior code returned the raw "null" string verbatim, which lands as JSONB null on Postgres (satisfies NOT NULL since SQL NULL ≠ JSONB null) or text "null" on SQLite. The non-nil check on the unmarshalled value routes JSON null to the existing log-and-coerce path with the rest of the malformed shapes. - items_views_jsonb_test.go: extended import test with an item carrying fields=null / tags=null; expects both coerced to "{}" / "[]" and the structured slog.Warn emitted. Verified: make test (SQLite) and the full ./... suite against the existing port-5445 Postgres container both pass cleanly. Refs: IDEA-1486, IDEA-1488, codex R1 review. * fix(store,server): codex R2 follow-ups for IDEA-1486 / IDEA-1488 Two defects surfaced by codex R2. P1 is a real ship-breaker; P2 closes a parity gap that R1 missed. P1: migration backfill normalized only SQL NULL, not malformed/wrong- shape JSON. The four new migrations originally wrote `WHERE x IS NULL`. Rows with fields = '' / 'null' / '[]' / 'not json' all survived the filter, then violated the post-migration NOT NULL+shape contract. Concrete ship- breaker on SQLite: 056 recreates the partial UNIQUE index on json_extract(fields, '$.invocation_slug') from migration 054, and json_extract errors on rows whose fields fails json_valid — a single bad row breaks CREATE INDEX mid-migration. Toggle-verified: with the NULL-only WHERE, the new SQLite test fails at exactly that CREATE INDEX with "SQL logic error: malformed JSON (1)". Widened the backfill clauses: - migrations/056: UPDATE items WHERE fields IS NULL OR json_valid(fields)=0 OR json_type(fields)!='object' (same trio for tags with 'array'). - migrations/057: same trio for views.config. - pgmigrations/035: WHERE fields IS NULL OR jsonb_typeof(fields)!='object'. JSONB rejects invalid JSON on write so the json_valid leg isn't needed on Postgres; only the shape check matters. - pgmigrations/036: same shape check on views.config. Regression tests in internal/store/items_views_jsonb_test.go: - TestItemsViewsJSONB_SQLiteBackfillRepairsMalformedShapes: applies migrations through 053 (skipping 054 which would itself error on malformed rows), seeds every observable shape pathology — SQL NULL, empty string, JSON null literal, wrong-shape JSON, non-JSON garbage — then applies 055/056/057. Asserts every malformed row is repaired AND the partial UNIQUE index actually fires on duplicate invocation_slug post-rebuild (proving the CREATE INDEX path executed end-to-end). - TestItemsViewsJSONB_PostgresBackfillRepairsMalformedShapes: parallel Postgres coverage; seeds JSONB null / array / primitive via direct ::jsonb cast and asserts the widened WHERE clause repairs each. P2: handleCreateItem didn't unwrap ErrInvalidFieldsType/ErrInvalidTagsType. R1's flexJSONToString tightening propagated the sentinels through every UnmarshalJSON path, but handleCreateItem (POST /items) still returned 'invalid JSON: <wrapped>' from decodeJSON. PATCH and the view/collection POST/PATCH handlers already unwrapped — POST was the outlier. - internal/server/handlers_items.go: mirror the PATCH-side errors.Is handling at the POST path. Brief, three-line diff. - handlers_items_jsonb_inner_shape_test.go: new TestCreateItem_ JSONEncodedStringInnerShapeValidated covers POST with fields=`[]`, fields=42, tags=`{}`, tags={"x":1}, plus a valid positive control. Asserts no "invalid JSON:" wrapper and presence of the sentinel message verbatim. Backfill-pattern audit (codex R2's grep prompt): only 055 / pg-034 (collections.settings, already shipped) exhibits the same NULL-only WHERE gap. Per the brief: NOT touched — retroactive repair belongs to a separate IDEA. Other NULL-only backfills (043/pg-023's oauth_providers, 044/pg-024's expires_at) handle their respective shapes correctly or aren't JSON columns. Verified: make test (SQLite) clean. Full ./... suite against the existing port-5445 Postgres container clean (one unrelated flake in internal/collab passed on rerun). Refs: IDEA-1486, IDEA-1488, codex R2 review. |
||
|
xarmian
|
438cb6180a |
fix(mcp): flexible JSON shapes on item create + clearer field surface (BUG-1431, BUG-1432) (#547)
* fix(mcp): flexible JSON shapes on item create + clearer field surface (BUG-1431, BUG-1432)
BUG-1432 root cause (real): models.ItemCreate.Tags is a Go string, so
the default unmarshaler rejected the natural JSON-array shape every
agent sends (`tags: ["foo","bar"]` → "cannot unmarshal array into Go
struct field ItemCreate.tags of type string", HTTP 400). On Postgres
the alternative — passing `tags: "foo,bar"` per the catalog's old
"Comma-separated tags" description — landed as a non-JSON value in
the JSONB column and surfaced as a generic HTTP 500. SQLite's TEXT
column silently accepted the corrupt value, which is why local repros
didn't show it.
Codex's independent investigation called out the asymmetry: ItemUpdate
already had a flexible UnmarshalJSON for `fields`/`tags` per BUG-1144,
but ItemCreate didn't. This PR mirrors that flexibility on the create
path and aligns the MCP surface description with reality.
BUG-1431 root cause (real, not the misdiagnosis the agent reported):
the dispatcher's `parseFieldKVP` only accepted the CLI-style array-of-
"key=value" shape, rejecting the JSON-native `field: {key: value}` map
shape with "expected array or string, got map[string]interface {}".
Agents naturally try the map shape and got a non-actionable error;
that drove the BUG-1409 agent to mis-blame status placement. Empirical
repro confirmed that `status` actually works in both top-level AND
inside-fields positions today (Tests 1, 4 in the investigation); the
real surface problem was the missing map shape on `field`.
Changes:
- internal/models/item.go: add UnmarshalJSON to ItemCreate mirroring
ItemUpdate's BUG-1144 pattern. Accepts `fields` as object or
JSON-encoded string; `tags` as array or JSON-encoded string; either
field absent / null leaves Go zero value. Wrong shapes surface
ErrInvalidFieldsType / ErrInvalidTagsType (existing sentinels) so
agents see clean domain errors instead of "Go struct field" leaks.
- internal/mcp/dispatch_http.go: parseFieldKVP now accepts
map[string]any in addition to the existing array/string shapes. Map
shape preserves non-string values verbatim (e.g. number from a typed
flag), matching the array path's existing pass-through for non-string
entries.
- internal/mcp/catalog_item.go: update `tags` description from
"Comma-separated tags" (wrong on both SQLite and Postgres) to
"Tags as a JSON array of strings, e.g. [\"v1\",\"frontend\"]". Update
`field` description to clarify it's the escape hatch for
SCHEMA-DECLARED custom fields, name the dedicated top-level params
agents should reach for instead (status/priority/category/parent/
role/assign/tags), and note the new map-shape acceptance. Tool-level
prose updated to match.
Tests:
- TestItemCreateUnmarshalFlexFields (mirror of
TestItemUpdateUnmarshalFlexFields): 9 cases covering array/string/
null/absent/wrong-shape tags + object/string/array fields, plus a
smoke test that other fields decode normally alongside the new
flex paths.
- TestParseFieldKVP_Variants: extended with 3 new map-shape cases
(basic map, empty-key-skipped, non-string-value preserved).
End-to-end verification: 5 input shapes via curl against the live
handler. Pre-fix `tags: ["foo","bar"]` returned HTTP 400; post-fix
returns HTTP 201 with `tags="[\"foo\",\"bar\"]"` in the column.
`tags: {x:1}` (wrong shape) now returns a clean
domain-level 400 instead of leaked Go internals. Existing back-compat
paths (JSON-encoded string forms) preserved.
Related: PR #546 (BUG-1430 rate limit) addressed the original 500
cascade that drove the agent's specific misdiagnoses in BUG-1409.
* fix(mcp): forward tags array on update + drop unsupported map-shape doc per Codex review (round 1)
Codex round 1 caught two issues:
[P1] dispatch_http_advanced.go's PATCH builder filtered on `string`
only when forwarding `tags`, so a schema-conforming
`pad_item.update tags: ["a","b"]` was silently dropped. Now forwards
verbatim like mapItemCreate does — the handler's ItemUpdate
flex-unmarshaler (BUG-1144) normalizes any shape downstream.
Regression test added.
[P2] The `field` description claimed `{key: value}` map shape was
accepted, but the schema Type stays `array<string>` so schema-following
clients won't send the map shape. parseFieldKVP's map-shape handling
(added in the previous commit) stays as defensive parsing for clients
that ignore the schema, but the description no longer promises a shape
the published schema doesn't advertise. Tool-level prose updated to
match.
* fix(mcp): revert speculative parseFieldKVP map-shape support per Codex review (round 2)
Codex round 2 [P2] pointed out the map-shape parseFieldKVP support
added in the first commit is dead code in practice:
1. The advertised schema for `field` is `array<string>` — no
schema-conforming client sends a map.
2. `BuildCLIArgs` rejects map-shaped repeatable flags before they
reach the HTTP dispatcher.
3. Even if a map did reach the dispatcher, `hasFieldChanges`
doesn't recognize map shapes as field changes — `pad_item.update
field: {effort: "l"}` would skip the merge and PATCH without
`fields`.
Either completing the support (fix hasFieldChanges + BuildCLIArgs +
ItemUpdate Unmarshal) OR reverting was the right call. Reverting
keeps the surface consistent with the schema and removes the
unreachable code; future agents who want to override fields can use
the documented `["key=value"]` array shape.
BUG-1431's functional fix lands as the catalog description tightening
(the empirical repro confirmed `status` placement already works in
both forms; the agent's misdiagnosis was rooted in unclear docs, not
broken code). BUG-1432's flexible JSON unmarshal on ItemCreate stays
— that's the real fix verified by the live-handler repro.
* fix(mcp): preserve empty-string tags no-op + table-driven test per Codex review (round 3)
Codex round 3 [P2] caught a regression introduced in round 1's fix: by
switching the tags forwarding guard from \`v.(string) && v != ""\` to
\`v != nil\` to support array shapes, the empty-string filter for tags
on update was lost. \`pad_item.update tags: ""\` would now forward an
empty string to ItemUpdate, which treats it as an explicit
empty-string write — corrupting the JSON/JSONB tags column (500 on
Postgres).
Fix: type-switch on tags. Empty string skips (matches pre-fix
behaviour); arrays (including empty array \`[]\`, the legitimate
"clear tags" case) and non-empty strings forward.
Tests: the single-shape array test is replaced with a table-driven
TestDispatchItemUpdate_TagsForwarding covering array, empty array,
empty string (no-op), and comma-separated back-compat. Each case
asserts the tags key's presence/absence and shape in the PATCH body.
|
||
|
xarmian
|
c38b3bf5cd |
feat(playbooks): add invocation_slug + arguments schema fields (TASK-1378) (#517)
* feat(playbooks): add invocation_slug + arguments schema fields (TASK-1378)
Foundational change for PLAN-1377 — playbooks become first-class invokable
procedures. Two new optional fields land on the Playbooks collection
schema:
- `invocation_slug` (text, kebab-case, unique-per-workspace among non-null
values): enables `/pad <slug>` direct invocation. Nullable so
trigger-only playbooks (e.g. on-release checklists) don't need one.
- `arguments` (json, array of {name, type, required, default, description}):
declares the playbook's argument contract; mirrors the body's
`## Arguments` section in queryable form.
Plumbing pieces:
- `models.FieldDef` grows two general-purpose options — `Pattern` for
regex validation and `UniqueScope` for collection-level uniqueness.
Both are opt-in; existing schemas are unaffected.
- `items.ValidateFields` learns the `json` field type (accepts any
JSON-decodable value) and applies `Pattern` to string-typed values.
- `handlers_items.checkUniqueFields` queries `Store.ListItems` to enforce
`UniqueScope == "workspace_collection"` on create + update.
- Two migrations (SQLite 054, Postgres 033) JSON-patch the playbooks
schema on existing workspaces so the new fields show up without a
workspace re-init.
- TypeScript `FieldDef` mirrors the Go side.
Parent: PLAN-1377.
* fix(playbooks): address Codex review round 1 findings (TASK-1378)
P1 — EditCollectionModal now round-trips opaque pattern/unique_scope
metadata. EditableField carries the new keys; the load + save paths
preserve them so re-saving the playbooks collection from the UI doesn't
strip server-side validation rules the modal doesn't yet expose
dedicated controls for. fieldFromDef mirrors the change for templates.
P2 — checkUniqueFields' pre-write ListItems check is now backed by a
partial unique index (idx_items_invocation_slug_per_collection,
SQLite + Postgres) scoped to non-empty, non-deleted rows. The pre-check
still gives users a friendly error message in the common case; the
index closes the TOCTOU race between two concurrent writers. The
create-conflict error message is now generic enough to cover both the
slug constraint and the new invocation_slug index.
P2 — `json` field type now rejects raw strings, numbers, and bools. Only
objects, arrays, and null are accepted, so a generic web text input
can't silently corrupt a structured field by emitting "[]" instead of
an actual array. FieldEditor.svelte routes `json` fields to a
read-only summary in both readonly and edit modes; dedicated editors
(like TASK-1384's playbook editor that owns `arguments`) own the
structured form.
P3 — invocation_slug regex now requires a minimum of two characters
(`^[a-z0-9][a-z0-9-]*[a-z0-9]$`) in the Go const, the SQLite migration,
the Postgres migration, and the validate tests. Single-letter slugs
would shadow plausible NL tokens (e.g. `/pad a ...`) and the doc
comment already claimed the two-char floor; this aligns code with
intent.
Parent: PLAN-1377.
* fix(playbooks): address Codex review round 2 findings (TASK-1378)
P2.1 — checkUniqueFields no longer passes IncludeArchived=true. The
application-layer pre-check now matches the partial unique index's
`deleted_at IS NULL` predicate so a soft-deleted playbook releases its
slug back to the pool and reclaiming it succeeds instead of 409'ing.
P2.2 — handleUpdateItem now maps UNIQUE constraint / duplicate key
errors from UpdateItem to HTTP 409, mirroring the create path. A true
concurrent-update race that slips past checkUniqueFields and trips the
partial unique index used to surface as a misleading 500.
(Not addressed in this round: Codex's third finding — concern about the
partial unique index applying to "every collection" — is, on close
reading, not what the index does. `ON items(collection_id, json_extract(...))`
scopes uniqueness to the (collection_id, slug) pair, so two items in
different collections with the same `invocation_slug` value coexist
fine. The migration-failure risk is theoretical: `invocation_slug` is
a brand-new field key, so no pre-existing items can have it set, and
no migration-time duplicates can exist. If a future custom collection
adopts the same field name, opting into per-collection uniqueness is
exactly the intended semantic of FieldDef.UniqueScope.)
Parent: PLAN-1377.
* fix(playbooks): map restore-path UNIQUE violations to 409 (TASK-1378)
Codex round 3: restoring an archived playbook can hit the partial
unique index on invocation_slug if a replacement item already claimed
the slug. Map UNIQUE constraint / duplicate key errors from RestoreItem
to HTTP 409 with a targeted message, matching the create + update paths.
Parent: PLAN-1377.
* fix(playbooks): map collab-snapshot UNIQUE violations to 409 (TASK-1378)
Codex round 4: the collab-snapshot PATCH branch under
`s.collab.UnderItemLock` ran its own UpdateItem call and fell through
to writeInternalError on any non-stale-snapshot error. A concurrent
edit racing the invocation_slug partial unique index would surface as
500 instead of 409. Mirror the main UpdateItem error mapping.
Codex's other round-4 finding — the partial unique index applying to
"every collection" — is not addressed because the index IS already
collection-scoped: `ON items(collection_id, json_extract(fields,
'$.invocation_slug'))`. Two items in different collections with the
same slug coexist; only same-collection duplicates conflict. Migration
duplicates are impossible because `invocation_slug` is a brand-new
field key with no pre-existing items setting it. A custom collection
that later adopts the same field name opts into per-collection
uniqueness, matching the FieldDef.UniqueScope="workspace_collection"
semantic.
Parent: PLAN-1377.
|
||
|
xarmian
|
7456b5aed6 |
feat(store): add workspace-scoped monotonic seq column to items (TASK-1352) (#492)
* feat(store): add workspace-scoped monotonic seq column to items (TASK-1352) Adds an `items.seq` column that bumps on every mutation (create/update/soft-delete/restore) as the cursor mechanic for the local-first read model's delta sync (PLAN-1343, DOC-1342 design decision #1). Each mutation stamps `MAX(seq) + 1 WHERE workspace_id = ?` inside the same transaction that performs the write, with a Postgres advisory lock keyed on the workspace serializing concurrent seq-bumping mutations. SQLite's single-writer rule covers the same guarantee there. Migration backfills existing rows with sequential per-workspace seqs in (updated_at, id) order so every workspace has a non-zero MAX(seq) floor immediately. Adds an idx_items_workspace_seq index supporting both the `/items-index` cursor read and the future `/items-changes` range scan. The Seq field is now populated through every items SELECT helper (GetItem, GetItemIncludeDeleted, ListItems, ListItemsIndex, listItemsFTS, SearchItems, ItemsModifiedSince, GetChildItems, ListStarredItems, ResolveItemIncludeDeleted, GetItemBySlugIncludeDeleted) and the workspace import path stamps it via the same MAX+1 subquery so imported rows don't all collapse to seq=0. Parent: PLAN-1343. Foundation for TASK-1353 (wire seq into /items-index cursor) and TASK-1354 (/items-changes delta endpoint). * fix(store): bump items.seq on role reorder, MoveItem, and field migrations per Codex review (round 1) Codex round 1 flagged that UpdateRoleSortOrder was rewriting items.role_sort_order without bumping the new workspace-scoped seq column — delta-sync clients would miss role-board reorders until a full refresh. The same gap applied to MoveItem (collection change) and MigrateItemFieldValues (bulk select-option rename), which are also user-visible mutations the cursor must surface. Each path now: - acquires the workspace seq advisory lock (no-op on SQLite) - stamps seq = MAX(seq)+1 inside the same transaction The bulk rename gives all rows affected by a single statement the same seq value (MAX+1 at statement start). That preserves the "no overlap, no gap" cursor contract — a client at cursor < MAX sees them all in one batch, at cursor >= MAX sees none. |
||
|
xarmian
|
bdcb62e902 |
fix(api): accept nested object/array for PATCH items fields/tags (BUG-1144) (#485)
The PATCH /api/v1/workspaces/{ws}/items/{ref} endpoint previously
demanded `fields` and `tags` arrive as JSON-encoded strings, because
models.ItemUpdate declares them as *string to mirror the storage shape.
Sending the natural nested-object shape any reasonable HTTP client
would produce returned HTTP 400 with a leaked Go unmarshal error
naming the internal struct field — confusing for anyone integrating
against Pad over HTTP (webhook reactors, custom dashboards, non-CLI
agents, third-party MCP bridges).
This is the symmetric input-side counterpart to BUG-991, which was
fixed at the MCP boundary in PR #364 with dual-emit normalization
rather than the full Plan-sized models.Item migration.
Fix: add a custom ItemUpdate.UnmarshalJSON that accepts either shape
on the wire and normalizes to the canonical string internally. The
struct field type stays *string, so the validation/storage/web/CLI
pipeline is untouched. All in-process Go callers construct ItemUpdate
literals (15 grepped call sites) and never hit UnmarshalJSON, so the
change is invisible to them.
Wrong shapes (e.g. `{"fields":42}`, `{"tags":{"x":1}}`) now return a
domain-level 400 — `"fields" must be a JSON object or a JSON-encoded
string` — surfaced via sentinel errors (ErrInvalidFieldsType /
ErrInvalidTagsType) that the handler unwraps from decodeJSON's
"invalid JSON: %w" wrapper.
Coverage:
- models/item_test.go: 10 sub-tests covering object, array, string,
null, absent, and wrong-type cases for both fields and tags.
- server/handlers_items_test.go: 6 PATCH integration sub-tests
asserting back-compat, the BUG-1144 repro now returns 200, and
that error responses no longer leak Go struct field names.
Smoke-tested against the live server with the exact repro curl from
BUG-1144 (HTTP 200), plus malformed (HTTP 400 with clean message)
and stringified-string back-compat (HTTP 200).
|
||
|
xarmian
|
18087463ce |
feat(collab): op-log cursor protocol — force-refresh + watermark advance (TASK-1319) (#472)
* feat(collab): op-log cursor protocol — force-refresh + watermark advance (TASK-1319)
Closes both holes left by TASK-1309:
1. Long-disconnected tab + external-write race. A reconnecting client
announces its highest applied item_yjs_updates.id via `?since=<id>`.
If that id is below MIN(id) for the item, rows it expected to
replay have been pruned and the server sends a `force_refresh`
control frame and closes the conn. Client recreates the Y.Doc
and lazy-seeds from items.content. Without this, Tab A's stale
state would silently overwrite an external CLI/MCP write on the
next 5s flush.
2. Browser-only-edited items never GC'd. Browser collab-snapshot
PATCHes now carry an op_log_cursor body field. The store advances
items.content_flushed_op_log_id only when the cursor matches the
current MAX(op-log.id) — proving the markdown captures every
persisted op. SQL CASE clause re-evaluates MAX at COMMIT time so
a peer op landing between client-side cursor capture and the
UPDATE leaves the watermark untouched (no over-advancement).
Combined cursor mechanism:
- Server attaches op_log_cursor JSON control frames after replay,
after every successful AppendYjsUpdate (originator), and to every
peer's binary fan-out (so all peers stay in lockstep without a
round trip).
- Client persists per-tab in sessionStorage (NOT localStorage —
avoids cross-tab cursor leakage that would force-refresh stable
sessions).
- Server's MIN(id) check + force_refresh fires only when a non-zero
`since` is below MIN; `since=0` is treated as a fresh client.
New store methods: MinOpLogID, MaxOpLogID. New ItemUpdate field:
OpLogCursor *int64. New control message types: op_log_cursor,
force_refresh. New OpEvent.OpLogID for cursor piggyback. Existing
collab tests updated to drain TextMessage cursor frames.
Tests cover: initial cursor frame after replay (populated + empty
op-log), force_refresh fires when since<MIN, delta replay when
since>=MIN, cursor broadcast to originator + peers on append, and
watermark advancement gated on cursor==MAX.
Parent: PLAN-1248. Builds on TASK-1309.
* fix(collab): skip stale-Ydoc flush on force_refresh teardown per Codex review (round 1)
A force_refresh tear-down means the local Y.Doc cursor is below the
server's MIN(item_yjs_updates.id) — its derived markdown is stale.
Without this guard the collab $effect cleanup runs flushCollabNow
on the way out and silently PATCHes that stale markdown back to
items.content, overwriting the canonical content the fresh provider
is supposed to lazy-seed from. Per Codex round 1 [P1] of TASK-1319.
* fix(collab): force_refresh on empty op-log + cancel pending flush per Codex review (round 2)
Two P1 fixes:
1. Manager.Join now force_refreshes when since>0 and the op-log is
empty (hasMin==false), not just when since<MIN. After
PruneAndApply wipes the entire op-log, MIN is undefined; the
original predicate would have admitted the stale tab and let its
on-open Y.encodeStateAsUpdate write resurrect the pre-prune
document.
2. The +page.svelte onForceRefresh handler now also clears
collabFlushTimer. Without this a 5s timer that armed before the
force_refresh frame arrived can still fire AFTER the cleanup
ran, PATCHing stale Y.Doc-derived markdown to items.content.
New test: TestRoomManagerForceRefreshOnEmptyOpLogWithSince covers
the empty-op-log branch.
Per Codex round 2 [P1] of TASK-1319.
* fix(collab): include forceRefreshNonce in Editor key so it remounts on force_refresh per Codex review (round 3)
The collab $effect cleanup runs on forceRefreshNonce bump, but the
<Editor> {#key} was `${item.id}:true` — itemID doesn't change, so
the keyed Editor wasn't unmounting. The Tiptap Collaboration
extension only binds in onMount, so the editor stayed wired to the
stale (destroyed) Y.Doc while a fresh provider+doc were set up
in parallel. Edits would either be unsynced or eventually flush
stale markdown again.
Adding forceRefreshNonce to the key forces the Editor to remount
in lockstep with the doc swap. Per Codex round 3 [P1] of TASK-1319.
* fix(collab): refetch item.content before lazy-seed on force_refresh per Codex review (round 4)
After force_refresh the collab $effect rebuilds the Y.Doc and the
lazy-seed (TASK-1261) seeds it from item.content. But item.content
was the cached page-state copy — possibly stale relative to the
server (the WS force_refresh can beat the SSE/visibility refresh
that would otherwise update it). Lazy-seeding stale content into
a fresh op-log re-introduces exactly the staleness force_refresh
was supposed to clear: the next 5s flush PATCHes that stale view
back to canonical items.content.
onForceRefresh now does an api.items.get() before bumping the
nonce so the rebuild's lazy seed reads server-fresh content. A
failed fetch falls through to the bump anyway (an editor on
possibly-stale content is still better than a broken editor).
Per Codex round 4 [P1] of TASK-1319.
* fix(collab): suppress cursor during replay + move force_refresh check before getOrCreate per Codex review (round 5)
Two more findings:
1. [P1] writeLoop sends op_log_cursor frames for live ops broadcast
during the replay window. A client disconnecting after one of
those cursors lands but BEFORE the rest of replay completes
would persist a cursor pointing past unreplayed rows. On
reconnect with since=that-cursor, server replays nothing — the
client's Y.Doc would be missing causally-required ops.
Fix: per-roomConn replayDone atomic.Bool. writeLoop suppresses
cursor frames while it's false. runConn flips it after the
post-replay initial cursor is on the wire. Live binary frames
continue to flow during replay (Yjs CRDT commutativity); only
the cursor metadata is gated.
2. [P2] Force-refresh path leaked an empty room. getOrCreate
inserted into m.rooms before the force_refresh bail-out left
an orphan entry that PruneSweep would later treat as 'active'
and skip indefinitely.
Fix: schema-rebuild + force_refresh checks now run BEFORE
getOrCreate. Both are store-only mutations and the per-item
lock is held throughout, so concurrency is unchanged.
New test: TestRoomManagerCursorSuppressedDuringReplay regression-
guards the cursor-suppression behaviour.
Per Codex round 5 [P1+P2] of TASK-1319.
* fix(collab): tighten initial cursor + sync-destroy provider on force_refresh per Codex review (round 6)
Two more P1 fixes:
1. runConn's empty-replay fallback used MaxOpLogID() to anchor
the initial cursor. A live op landing between replayTo
returning and the cursor write would be reflected in MAX
but its binary frame might not have flowed through this
conn's writeLoop yet — the cursor would advertise an id
the client hasn't received. Initial cursor is now strictly
max(highestReplayed, since); MaxOpLogID is removed from
the opLogStore interface.
2. Provider.handleControlMessage's force_refresh branch now
calls this.destroy() SYNCHRONOUSLY before invoking the
onForceRefresh callback. Previously the consumer's recovery
path (async items.get refetch) would race the provider's
own onClose-triggered reconnect, which would re-open with
since=0 and push Y.encodeStateAsUpdate of the stale Y.Doc
— recreating the corruption force_refresh was meant to
prevent. destroy() sets destroyed=true so scheduleReconnect
short-circuits.
Per Codex round 6 [P1] of TASK-1319.
* fix(collab): block flush scheduling during force_refresh recovery per Codex review (round 7)
Previously, after onForceRefresh fires:
1. Provider is destroyed synchronously.
2. Async items.get refetch is in flight.
3. forceRefreshNonce bumps after refetch resolves.
4. $effect cleanup runs, then rebuild.
But during steps 2-3 the editor component is still mounted with
the stale Y.Doc, and a local edit fires handleContentUpdate which
calls scheduleCollabFlush. clearTimeout earlier in onForceRefresh
only canceled the timer at THAT moment; a new edit during the
refetch window arms a fresh timer that fires before cleanup. That
PATCHes stale Y.Doc-derived markdown back to canonical content,
recreating the corruption force_refresh was meant to prevent.
Fix: forceRefreshInFlight flag set in onForceRefresh, blocks
scheduleCollabFlush, resets after the fresh provider is wired
(end of $effect run). Per Codex round 7 [P1].
* fix(collab): gate runCollabFlush itself on force_refresh in-flight per Codex review (round 8)
scheduleCollabFlush blocked the 5s timer path, but direct callers
of flushCollabNow / runCollabFlush (beforeunload handler,
rich-to-raw toggle) bypassed the guard. A page reload or raw
toggle DURING the force_refresh recovery window still PATCHed
stale Y.Doc-derived markdown to canonical items.content.
Pulling the guard into runCollabFlush covers every caller in one
spot and returns 'deduped' so the result-shape contract holds.
Per Codex round 8 [P1] of TASK-1319.
* fix(collab): distinct 'skipped' result for force_refresh path; raw-toggle aborts per Codex review (round 9)
runCollabFlush returning 'deduped' on the force_refresh-blocked
path was indistinguishable from a legitimate same-content dedupe.
The rich→raw toggle treats 'deduped' as 'server already has this
markdown' and seeds rawSeedMarkdown from it — letting the user's
next raw edit overwrite canonical items.content with content
derived from the stale Y.Doc.
Add a distinct 'skipped' result for the force_refresh path. Raw
toggle aborts on it (with a 'try again in a moment' toast); other
callers fall through unchanged because no other call site
behaviorally depends on 'deduped' vs 'skipped'.
Per Codex round 9 [P1] of TASK-1319.
* fix(collab): server-side gate + post-await client guard against stale collab-snapshot per Codex review (round 10)
A force_refresh frame can arrive WHILE a collab-snapshot PATCH is
already mid-flight to the server. The client-side
forceRefreshInFlight check at PATCH-start can't catch this race;
the request lands at the server with stale Y.Doc-derived markdown.
Two-pronged fix:
1. Server: handler now checks op_log_cursor against MIN(op-log.id)
for collab-snapshot PATCHes and returns 409 Conflict when
cursor < MIN. Such cursors prove the flushing tab's Y.Doc was
built on rows that have been pruned (PruneAndApply, schema
rebuild, dormant GC). The markdown is, by construction, stale.
2. Client: post-await check on forceRefreshInFlight returns
'skipped' instead of 'flushed' so saveStatus / lastFlushedContent
don't seed from a known-stale base even if the server happened
to accept the PATCH (e.g. MIN advanced after handler validation).
New tests: TestCollabSnapshotRejectsCursorBelowMin (gate fires),
TestCollabSnapshotAcceptsCursorAtOrAboveMin (negative path).
Also de-leak an unused slice in the round-5 cursor-suppression test
so staticcheck stays clean.
Per Codex round 10 [P1] of TASK-1319.
* fix(collab): reject collab-snapshot when cursor>0 and op-log empty per Codex review (round 11)
The HTTP-layer gate I added in round 10 mirrored only PART of the
WS-upgrade force_refresh predicate. Round 5 had already taught us
that 'op-log entirely pruned' is a separate stale path from
'cursor below MIN' (PruneAndApply, schema rebuild, dormant GC all
leave hasMin=false), and the WS check now uses
`since > 0 && (!hasMin || since < minID)`. The HTTP gate had
only the second clause.
Mirror the WS predicate at the handler so a stale collab-snapshot
PATCH against an empty op-log gets a 409 too. New regression:
TestCollabSnapshotRejectsCursorOnEmptyOpLog.
Per Codex round 11 [P1] of TASK-1319.
* fix(collab): reject collab-snapshot cursor=0 on non-empty op-log per Codex review (round 12)
Round-11 gate accepted cursor=0 unconditionally. But a stateful tab
whose previous session disconnected BEFORE receiving the
post-replay cursor frame (network blip during the writeMu burst
between replay binaries and the cursor) ends up with sessionStorage
cursor=0 + a non-empty Y.Doc populated by prior replay binaries.
On reconnect with since=0 the server treats it as fresh, replays
nothing if the op-log was meanwhile pruned, and the client's
on-open Y.encodeStateAsUpdate resurrects pre-prune ops. The next
flush carries cursor=0 + stale-derived markdown.
The gate now refuses any incompatible cursor:
- cursor>0 + empty op-log (prior rule)
- cursor<MIN + non-empty op-log (prior rule, now naturally
catches cursor=0 too because 0 < any positive MIN)
The WS replay path is unchanged — full replay from since=0 is
the recovery for clients that genuinely lost their cursor; the
corruption manifested through the flush PATCH which we now gate.
New test: TestCollabSnapshotRejectsCursorZeroOnNonEmptyOpLog.
Per Codex round 12 [P1] of TASK-1319.
* fix(collab): close cursor=0 client/server gaps + lock validation+write atomically per Codex review (round 13)
Four P1 issues addressed:
1. Client always sends op_log_cursor (including 0) so the server
gate sees the field. Previously cursor=0 was omitted, which
silently bypassed the server's stale-snapshot rejection.
2. Provider construction now resets sessionStorage cursor to 0
when the Y.Doc is empty. The Y.Doc isn't persisted across
page reload, so a stored cursor=N + fresh empty Y.Doc would
announce since=N to the server and miss rows 1..N from
replay (server only replays id > N).
3. onOpen skips Y.encodeStateAsUpdate when lastOpLogID === 0.
A populated Y.Doc + cursor=0 is the network-blip-during-cursor-
write failure mode; pushing that state can resurrect ops the
server has pruned. Server replay + lazy-seed handle recovery
without our push.
4. Server gate now runs INSIDE the per-item collab setup lock
(new RoomManager.UnderItemLock helper) so a concurrent prune
(PruneAndApply, schema rebuild, dormant GC) cannot land
between the MIN check and the items.content write. Without
this, a tight race let stale snapshots overwrite canonical
content the prune just installed.
Per Codex round 13 [P1] of TASK-1319.
* fix(collab): gate handleDocUpdate on cursorAnchored to close stale-Ydoc edit path per Codex review (round 14)
Round 13 fix skipped on-open send for lastOpLogID===0, but local
edits via handleDocUpdate still propagated. A populated Y.Doc +
no-cursor-yet client could type, the edit would land in the
op-log with id N, server would send originator cursor=N, and
the next 5s flush would carry an 'anchored' cursor that passed
the server's MIN check — overwriting items.content with stale-
Y.Doc-derived markdown.
Add a cursorAnchored boolean. Set on first op_log_cursor frame
receipt (including cursor=0 against an empty op-log — that's a
legitimate 'server has nothing' signal). handleDocUpdate refuses
to send before this. Local edits buffer in the editor; once the
cursor arrives (or force_refresh rebuilds the provider), the
existing reconnect/edit paths catch them up.
Per Codex round 14 [P1] of TASK-1319.
* fix(collab): buffer + flush pre-anchor local updates per Codex review (round 15)
Round 14 silently dropped local Yjs updates fired before the
first op_log_cursor frame anchored the session. Yjs updates are
incremental: a dropped keystroke leaves later ops referencing
structs no peer can resolve, breaking convergence.
Buffer pre-anchor updates in a Uint8Array[] (capped at 1000 to
prevent unbounded growth in pathological 'anchor never arrives'
scenarios — overflow triggers force_refresh-style recovery).
On the first cursor frame, flush the buffer in order so the
server gets every causally-required struct before any post-
anchor updates land.
Per Codex round 15 [P1] of TASK-1319.
* fix(collab): destroy provider before force_refresh on pre-anchor buffer overflow per Codex review (round 16)
Round 15 overflow path called onForceRefresh but didn't destroy
the provider synchronously. A late op_log_cursor arriving before
the page-level rebuild (the recovery callback is async — refetches
items.content) would flip cursorAnchored=true, the partially-
populated buffer would flush, but the DROPPED prefix (the
overflowed entries) would leave server-side ops causally
incomplete — exactly the bug the buffer was supposed to prevent.
destroy() sets destroyed=true, removes message listener,
short-circuits scheduleReconnect, closes the socket. Late cursor
frames can no longer anchor a doomed provider.
Per Codex round 16 [P2] of TASK-1319.
* fix(collab): refuse rebuild on refetch fail + broaden on-open gate to cursorAnchored per Codex review (round 17)
Two findings:
[P1] force_refresh recovery bumped forceRefreshNonce in finally
even when the item.content refetch failed. The rebuild then
lazy-seeded from the cached (possibly-stale) item.content, and
the next flush would PATCH that stale view back to the server.
Move the bump into .then() so a failed refetch surfaces a
'please reload' toast and leaves the editor effectively
read-only (forceRefreshInFlight stays true, blocking flushes).
[P2] Send-on-open gate was lastOpLogID > 0, which silently
dropped local edits made during a brief offline window after a
legitimate 'cursor=0' anchor (empty op-log session). Switch to
cursorAnchored — the boolean specifically distinguishes
'unanchored' (stale Y.Doc + no server confirmation) from
'anchored at cursor=0' (legitimate empty op-log).
Per Codex round 17 [P1+P2] of TASK-1319.
* fix(collab): force_refresh on cursor=0 against non-empty Y.Doc per Codex review (round 18)
cursor=0 means the server's op-log is currently empty. A
non-empty Y.Doc at first-cursor receipt implies the ops came
from an earlier connection within this provider's life that
never reached its post-replay cursor frame, followed by a
server-side prune (PruneAndApply, schema rebuild, dormant GC)
during our disconnect. Anchoring at cursor=0 in that state
would mark a stale Y.Doc as authoritative; the next on-open
state push or flush would resurrect pre-prune state and
overwrite canonical items.content.
Detect the configuration via Y.encodeStateVector length and
invoke the same force_refresh-style recovery the explicit
server frame triggers: destroy provider, clear sessionStorage,
fire onForceRefresh so the page rebuilds from items.content.
Per Codex round 18 [P1] of TASK-1319.
* fix(collab): gate cursor=0 force_refresh on remoteSyncApplied per Codex review (round 19)
Round 18 force_refreshed the provider whenever cursor=0 arrived
against a non-empty Y.Doc. But local pre-anchor edits (user typed
before the initial cursor=0 of a legitimate empty-op-log session
arrived) ALSO populate Y.Doc — yet those edits live in
preAnchorUpdates and were supposed to flush on anchor. The
predicate spuriously triggered force_refresh, dropping the
buffered local edits.
Track remoteSyncApplied (set when readSyncMessage applies
anything to Y.Doc — replay binary or live peer op). Only force_
refresh on cursor=0 when remoteSyncApplied is true: that's the
true 'remote replay landed but server now reports empty op-log
=> mid-session prune' signature.
Per Codex round 19 [P1] of TASK-1319.
* fix(collab): repair brace mis-merge in wsProvider cursor=0 guard
The round-19 patch overlapped the round-18 inner block, producing
an extra brace + over-indented body. Collapsing into a single
clean block restores parseability without changing semantics
beyond what round 19 already documented.
* fix(collab): gate syncStep2 reply on cursorAnchored per Codex review (round 20)
readSyncMessage writes an inline syncStep2 reply when it receives
a peer's syncStep1. That reply embeds our current Y.Doc state.
If a peer's syncStep1 arrives before our first op_log_cursor
(pre-anchor window), the reply path bypasses handleDocUpdate's
cursorAnchored gate and lets potentially-stale Y.Doc state reach
the server before the cursor=0 + remoteSyncApplied force_refresh
recovery has a chance to fire.
Suppress the reply while unanchored. Peer state propagation
still works: the buffered preAnchorUpdates flush on anchor, and
the lazy-seed rebuild after a force_refresh seeds canonical
content from items.content.
Per Codex round 20 [P1] of TASK-1319.
* fix(collab): fold mid-replay live op ids into post-replay cursor + remoteSyncApplied only on apply per Codex review (round 21)
Two more findings:
[P1 server] writeLoop suppresses cursor frames during replay to
prevent the client persisting a cursor past unreplayed rows.
But binary frames for those live ops still go through
(commutativity), so the client APPLIES them to its Y.Doc. The
post-replay initial cursor only covered max(highestReplayed,
since), leaving the cursor below the highest applied op. On
empty-replay sessions this trips the client's
'cursor=0 + remoteSyncApplied' force_refresh path and discards
buffered pre-anchor edits.
Track maxLiveOpLogIDDuringReplay on the roomConn (atomic
compare-and-swap) and fold it into the post-replay cursor.
[P1 client] remoteSyncApplied was set on every MESSAGE_SYNC,
including syncStep1 (which only carries a state vector — it
doesn't apply state). A peer's syncStep1 arriving pre-anchor
would falsely flag remote-sync-applied and trip the cursor=0
force_refresh on legitimate empty-op-log sessions. Set the
flag only after readSyncMessage returns, and only for
syncStep2 / update subtypes.
Per Codex round 21 [P1] of TASK-1319.
* fix(collab): widen writeMu critical section + drop omitempty on op_log_id per Codex review (round 22)
Two more P1s:
[P1 server] writeLoop's mid-replay record-max happened OUTSIDE
writeMu, so runConn's post-replay read could race the record:
runConn loads → writeLoop's atomic store of higher value →
runConn sends cursor below the live id. Move the entire
per-event sequence (binary write + replayDone observation +
record-or-send) inside writeMu, and have runConn acquire
writeMu around its read+cursor-write+replayDone-flip. The lock
serializes the two paths cleanly: writeLoop events that ran
first have already recorded; events that arrive after replayDone
flips emit their own cursor frames.
[P1 protocol] OpLogID had `omitempty` JSON tag — a legitimate
cursor=0 (empty op-log session) serialized as
`{"type":"op_log_cursor"}` with no op_log_id field. The
client's strict-type check then rejected it as malformed,
leaving the session unanchored and local edits buffered
forever. Drop omitempty so 0 is wire-visible. Other control
types (applier_request/ack) carry an extra op_log_id:0 in
their JSON, which their client dispatches ignore.
Per Codex round 22 [P1] of TASK-1319.
* fix(collab): route originator cursor through writeLoop FIFO per Codex review (round 23)
readLoop sent the originator's op_log_cursor directly via
sendOpLogCursor right after AppendYjsUpdate, bypassing the bus/
writeLoop ordering. With a peer op already queued in rc.bus, the
sequence on the wire could be:
1. originator cursor=N (newer local op)
2. peer binary (older op)
3. peer cursor=M < N (rejected by client's max-take logic)
Client persists cursor=N. If the client then disconnects before
applying the peer binary, reconnect with since=N replays nothing
(server has nothing > N) and the older peer op is lost forever
to this client's Y.Doc.
Fix: writeLoop now processes self events too — skipping the
binary echo (the originator already has Y.Doc state) but routing
the cursor frame through the same FIFO bus channel as peer ops.
The originator's cursor=N now arrives strictly AFTER all
older-id peer events on the same channel.
Per Codex round 23 [P1] of TASK-1319.
|
||
|
xarmian
|
191b887e23 |
feat(versions): VersionSource attribution + collab coexistence (TASK-1267) (#465)
The collab 5s-flush PATCH (TASK-1260) sends
`?source=collab-snapshot` with a body of just `{ content }`. Without
a handler-side stamp, `Store.UpdateItem`'s default coerced empty
input.Source to "web" on the version row and the per-(actor, source)
throttle suppressed every collab-driven snapshot following the user's
last manual web edit — version-diff effectively went silent during
co-edit sessions.
Adds:
- ItemUpdate.VersionSource: overrides per-version-row Source
attribution WITHOUT mutating items.source. The latter feeds
WorkspaceHasAgentActivity's `source IN ('cli', 'mcp')` filter,
so a CLI/MCP-created item the user opens in the editor would
otherwise silently flip out of the agent-activity tally on every
auto-flush.
- Store.UpdateItem prefers VersionSource over Source for version
row creation; falls back to Source then "web" if neither set.
- handlers_items.go stamps `input.VersionSource = "collab-snapshot"`
for `?source=collab-snapshot` PATCHes (when not already set).
Tests:
- internal/store/items_collab_versions_test.go: store-level
reverse-patch reconstruction over a CLI→web→collab-snapshot
edit sequence; verifies IsDiff=true on at least one row.
- internal/server/handlers_items_collab_versions_test.go: full
HTTP-level test of the route; asserts a collab-snapshot version
row is created AND that items.source stays "cli".
Four rounds of Codex review.
|
||
|
xarmian
|
04514817ae |
feat(store): add Yjs op-log table + store methods (TASK-1252) (#450)
* feat(store): add Yjs op-log table + store methods (TASK-1252)
Persistence groundwork for the dumb-relay WebSocket server in PLAN-1248.
The item_yjs_updates table records every Yjs binary update (browser
edits, future designated-applier conversions of CLI/API content
changes) so reconnecting peers can replay updates since their last
known cursor and cold rooms can rebuild their in-memory Y.Doc.
Schema (mirrored across SQLite + Postgres):
- id monotonic — INTEGER PRIMARY KEY AUTOINCREMENT (SQLite)
/ BIGSERIAL (Postgres). Never reused, even after
deletes; serves as the cursor every reconnecting
client compares against.
- item_id FK with ON DELETE CASCADE so item deletion reclaims
op-log space automatically.
- update_data raw Yjs binary update — BLOB / BYTEA. Opaque to the
server.
- schema_version stamped per row. Mismatch on connect drives
TASK-1268's snapshot-and-rebuild flow.
- created_at ISO8601 UTC TEXT, matching pad's cross-dialect
timestamp convention (see migrations/047_attachments).
Drives PruneYjsUpdatesBefore.
Store API (internal/store/yjs_updates.go):
- AppendYjsUpdate — validates non-empty itemID/data/schemaVersion,
inserts and returns the new monotonic id (RETURNING on Postgres,
LastInsertId on SQLite). Empty-zero-byte updates are rejected at
the Go layer rather than relying on NOT NULL — they're a no-op
that would only pollute the log.
- LoadYjsUpdatesSince — strict id > sinceID filter, ordered by id
ascending. sinceID=0 returns everything (cold-room rebuild path).
Tolerates either RFC3339 or "YYYY-MM-DD HH:MM:SS" timestamp formats
on read so any future operator-written / CURRENT_TIMESTAMP-style row
doesn't blow up the load path.
- PruneYjsUpdatesBefore — created_at < cutoff, scoped to itemID.
Returns rows-affected count. Used by the eventual GC sweeper
(out of scope for this task).
Tests cover: append + monotonic ids, load-since-cursor filtering,
input validation, prune scoped to itemID, and ON DELETE CASCADE on
parent item removal. Pass on SQLite locally; Postgres mirror migration
+ store methods are dialect-agnostic.
Parent: PLAN-1248. First task of Phase 1 — Backend foundation.
* docs(store): document AppendYjsUpdate per-item serialization contract per Codex review (round 1)
P1: Postgres BIGSERIAL ids are allocation-ordered, not commit-order.
Concurrent appends to the same item could in theory produce a cursor
gap — a slower transaction can hold a smaller id while a faster one
commits a larger id first, and a reader that advances past the visible
larger id would later miss the smaller id when it commits.
The dumb-relay room manager (TASK-1255) is the sole writer per item by
design — there's exactly one goroutine appending per Y.Doc — so the
hazard does not manifest in practice. The fix is at the API contract
level: the doc comment now spells out the serialization requirement,
why the room manager satisfies it, and the multi-replica re-enforcement
note for the future Redis-fanout IDEA. We do not take an internal
advisory lock because that would be paid by every append even though
the caller already holds the per-room mutex.
No code change — contract is at the doc comment.
|
||
|
xarmian
|
f9d3244660 |
feat(connected-apps): user-facing OAuth connection management page (TASK-954) (#390)
* feat(connected-apps): user-facing OAuth connection management page (TASK-954)
Adds /console/connected-apps where a logged-in user can see every
OAuth grant chain they've authorized via the MCP consent flow
(Claude Desktop, Cursor, …) and revoke any of them. Joins to the
DCR client metadata for the display name + logo, and to the MCP
audit log (TASK-960) for the "last used" + "30-day calls" columns.
Pieces:
- internal/store/connected_apps.go — ListUserOAuthConnections walks
oauth_access_tokens + oauth_refresh_tokens, dedups by request_id,
hydrates client metadata, parses session_data for the workspace
allow-list, classifies granted_scopes into a coarse capability
tier. RevokeUserOAuthConnection verifies ownership (ErrConnection
NotFound for stranger's chains — anti-enumeration; same shape as
for unknown chains) then calls the existing RevokeRefreshTokenFamily
+ RevokeAccessTokenFamily so the next /mcp call gets 401.
- internal/models/connected_apps.go — OAuthConnection + CapabilityTier
models.
- internal/server/handlers_connected_apps.go — REST endpoints:
GET /api/v1/connected-apps (list) + DELETE /api/v1/connected-apps/{id}
(revoke, idempotent, 204). Wrapped in requireCloudMode group.
List enriches with MCPConnectionStatsForUser (audit aggregates) —
soft-fails on the audit lookup so a broken audit table degrades
to "no last-used data" instead of a broken page. Revoke records
an "oauth_connection_revoked" entry in audit_trail via the
existing CreateActivity path.
- web/src/routes/console/connected-apps/+page.svelte — list with
per-app card (logo, name, capability badge, workspace chips with
+N expander, connected/last-used relative times, 30-day count),
Details expander showing scope_string + workspace list + redirect
URIs, Revoke button → confirm modal → optimistic refresh, friendly
empty state linking to /connect.
- web/src/routes/console/+layout.svelte — Connected Apps nav link
(cloud-mode-gated, between Settings and Billing).
- web/src/lib/api/client.ts + types/index.ts — typed client +
ConnectedApp interface.
Tests cover:
- Store: chain dedup across rotation siblings, subject filtering
(Bob can't see Alice's), inactive chains excluded, ownership
check on revoke, idempotent re-revoke, capability tier mapping,
session-data allowed_workspaces parsing (both []string and JSON
[]interface{} round-trips).
- Handler: cloud-mode gate (404 outside), owner-only filtering,
DTO field shape + audit enrichment populating last_used_at +
calls_30d, revoke ownership 404 (not 403 — anti-enumeration),
idempotent 204, audit_trail row written.
`make check` clean (lint + go test ./... + svelte-kit build).
Parent: PLAN-943.
* fix(connected-apps): point empty-state link at getpad.dev (Codex review round 1)
Codex caught: the empty-state link to /connect 404s because /connect is a
pad-web (marketing site) route, not a docapp route. From inside the
authenticated console at app.getpad.dev, the right target is the
absolute https://getpad.dev/connect URL — same pattern the +error.svelte
page uses for its "Back to getpad.dev" + "/docs" links.
* fix(console nav): exclude /console/connected-apps from Workspaces active match (Codex round 2)
Codex caught: the Workspaces nav predicate `isActive('/console') && !isActive('/console/settings') && ...` was missing the new /console/connected-apps prefix, so both Workspaces AND Connected Apps lit up when viewing the connected-apps page.
Same shape as the existing exclusions for settings / billing / admin.
|
||
|
xarmian
|
d8b1d98e08 |
feat(mcp): persistent audit log for /mcp tool calls (TASK-960) (#389)
* feat(mcp): persistent audit log for /mcp tool calls (TASK-960)
Adds a 90-day-retention audit log of every MCP request. Drives the
"last used" + "30-day calls" columns the connected-apps page (TASK-954)
will read, and gives ops + on-call a forensics surface via a new
admin /console/admin/mcp-audit page.
Schema deviation from the spec, documented in migration 049:
the original task body called for `token_id REFERENCES oauth_tokens(id)`
but pad has no `oauth_tokens` table — instead an OAuth grant chain is
identified by `request_id` (preserved across refresh-token rotations,
see migration 048), and PAT-authenticated MCP requests have no OAuth
identity at all. The audit row therefore carries `(token_kind,
token_ref)` — `oauth` + request_id for OAuth, or `pat` + api_tokens.id
for PATs. The connected-apps page in TASK-954 will filter on
token_kind='oauth' to surface third-party connections only.
Pieces:
- internal/store/migrations/049_mcp_audit.sql + pgmigrations/028 — table.
- internal/models/mcp_audit.go — typed entry + 30-day stats DTO.
- internal/store/mcp_audit.go — insert / list-by-user / list-by-connection
/ list-all / per-connection-stats aggregator / 90-day retention sweeper.
- internal/server/middleware_mcp_audit.go — async writer + sweeper +
middleware that wraps /mcp behind MCPBearerAuth. Hot path is
non-blocking enqueue with drop-on-overflow + atomic drop counter.
- internal/server/middleware_mcp_auth.go — both PAT + OAuth branches now
stash WithMCPTokenIdentity so the audit row attributes correctly.
- internal/server/handlers_mcp_audit.go — read endpoints:
GET /api/v1/connected-apps/{id}/audit (owner-scoped) +
GET /api/v1/admin/mcp-audit (admin-only).
- web/src/routes/console/admin/mcp-audit/+page.svelte + tab in admin layout.
- Tests cover required-field validation, round-trip, pagination,
owner-only filtering, last-used + 30-day aggregates, retention sweep,
body-sniff parser, canonical-JSON arg hashing, buffer-full drop path,
status-to-result classification, admin gate, DTO field shape.
`make check` clean (lint + go test ./... + svelte-kit build).
Parent: PLAN-943.
* fix(mcp-audit): emit denied row on rate-limit reject per Codex review (round 1)
PR #389 round 1 caught: MCPAuditLog is mounted INSIDE MCPBearerAuth, so
when bearer auth's per-token rate-limit fires (429) it returns before
next.ServeHTTP — and the wrapping audit middleware never sees the
response. classifyMCPResult mapped 401/403/429 with no path that could
actually reach it.
Fix: emitMCPAuditDenied helper called directly from the rate-limit
deny branches of both PAT + OAuth paths. Resolved user + token
identity are already in scope at that point, so the audit row gets
attributed correctly. Pre-auth rejections (no/invalid bearer) stay
un-audited because there's no user to attribute them to and the
audit_trail table covers those auth-event signals already.
Threading: handleMCPPATAuth + handleMCPOAuthAuth now take the entry
timestamp so the denied row carries real latency.
Test: TestMCPAudit_RateLimited_RecordsDeniedRow drives a real PAT
through the rate limiter, drains to 429, and asserts the audit row
lands with status="denied" + error_kind="rate_limited" + the right
tool_name from the request body.
|
||
|
xarmian
|
2a00775481 |
feat(oauth): schema + storage layer (TASK-1023, sub-PR A of TASK-951) (#370)
* feat(oauth): schema + storage layer for OAuth 2.1 server (TASK-1023, sub-PR A of TASK-951)
First of 5 sub-PRs landing the OAuth 2.1 authorization server in
PLAN-943. This one is foundation only — no HTTP exposure, no fosite
import, no public surface change.
Schema (5 tables, parallel SQLite + Postgres migrations):
- oauth_clients — RFC 7591 Dynamic Client Registration; public clients only for v1
- oauth_authorization_codes — short-lived codes for the auth-code grant
- oauth_access_tokens — opaque HMAC; subject denormalized for fast user-bound queries
- oauth_refresh_tokens — same shape; access_token_signature link + request_id chain
- oauth_pkce_requests — PKCE session keyed by auth-code signature
Storage layer (internal/store/oauth.go):
- 12 public methods covering fosite's ClientManager + CoreStorage +
PKCERequestStorage + TokenRevocationStorage interface shapes,
using pad-internal types so the package stays fosite-free.
- Three sentinel errors (ErrOAuthNotFound, ErrOAuthInvalidatedCode,
ErrOAuthInactiveToken) that sub-PR B's adapter maps to the
matching fosite errors.
- request_id IS the chain identifier (fosite preserves it across
rotations — handler/oauth2/flow_refresh.go:86), so family
revocation is a single indexed UPDATE rather than a separate
chain_id column.
14 tests covering: client CRUD + idempotent delete + empty-slice
normalization, auth-code create/get/invalidate (including the
"return payload alongside ErrInvalidatedCode" contract fosite
relies on for family revocation), access-token CRUD + delete,
refresh CRUD + RotateRefreshToken (single-row flip), refresh-token
family revocation (entire chain via request_id, leaves other
chains untouched), access-token family revocation, PKCE CRUD, and
required-field validation.
Both backends share test bodies via testStore(t); set
PAD_TEST_POSTGRES_URL=... to run the same suite against Postgres.
Out of scope for this PR (subsequent sub-PRs):
- fosite import + adapter (sub-PR B / TASK-1024)
- DCR + authorize + token endpoints (sub-PR C / TASK-1025)
- revoke + introspect endpoints (sub-PR D / TASK-1026)
- MCPBearerAuth OAuth integration (sub-PR E / TASK-1027)
* fix(oauth): always insert active=true; drop broken zero-value Active override per Codex review (round 1)
Codex round 1 caught a P1 in insertOAuthRequestRow:
active := defaultActive
if req.Active != defaultActive {
active = req.Active // <- zero-value collides
}
When defaultActive=true and req.Active=false (the zero value), this
branch fires and the row is stored with active=FALSE — silently
producing immediately-revoked tokens. Any sub-PR B adapter that
built an OAuthRequest without explicitly setting Active=true would
ship broken.
Fix: hardcode active=TRUE on insert. Drop the defaultActive
parameter (it's always true for the three flagged tables; PKCE
has no active column). Pre-seeding inactive isn't a supported flow
— fosite never does it, and tests that need a revoked row do
Create + Invalidate / Rotate / RevokeFamily as a two-step.
Regression test TestOAuth_Insert_AlwaysActive constructs an
OAuthRequest with zero-value Active and asserts the row is
readable as active for all three table types (codes, access,
refresh). Without the fix the test fails on the first GetAccessToken
call with ErrOAuthInactiveToken.
* fix(oauth): RotateRefreshToken revokes both refresh + access families per Codex review (round 2)
Codex round 2 caught: my RotateRefreshToken only marked the named
refresh row inactive, but fosite's reference MemoryStore.RotateRefreshToken
(storage/memory.go:497-504) revokes BOTH the refresh family AND the
access family for the grant's request_id. Without this, every access
token issued before a refresh remained active until TTL — defeating
the rotation's invalidation contract.
Fix: RotateRefreshToken now delegates to RevokeRefreshTokenFamily +
RevokeAccessTokenFamily (both already existed). The signatureToRotate
parameter becomes vestigial — fosite passes it but the family revoke
catches every chain member regardless of which row triggered the
rotation. The new pair fosite immediately issues via
CreateAccessTokenSession + CreateRefreshTokenSession inherits the
same request_id (flow_refresh.go:86) and lands active=TRUE per the
round-1 hardcode, so the net post-rotation state is "all old rows
in this grant inactive, the new pair active."
Test rewrite: TestOAuth_RotateRefreshToken_FlipsActiveOnSingleRow
asserted the OPPOSITE behavior (only one row touched) — that was
the original bug. Replaced with TestOAuth_RotateRefreshToken_RevokesEntireGrant
which seeds a refresh + access pair in the same chain, plus a
distinct unrelated grant, then asserts after rotation:
- old refresh + old access both inactive
- unrelated grant untouched (request_id-scoped)
* fix(oauth): DeleteOAuthClient cascades dependent rows in a tx per Codex review (round 3)
Round 3 finding: DeleteOAuthClient errored with FK constraint
violation for any client that had ever issued a grant. The
migrations declare client_id FKs without ON DELETE CASCADE — by
design, so a stray DELETE FROM oauth_clients elsewhere fails
loudly rather than silently nuking grants — but that meant the
"officially supported" delete path was unusable.
Fix: DeleteOAuthClient now runs five sequential DELETEs inside a
single transaction:
1. oauth_pkce_requests
2. oauth_refresh_tokens
3. oauth_access_tokens
4. oauth_authorization_codes
5. oauth_clients
Order matters (children before parent) because the FKs aren't
cascading. The tx makes it atomic — if any step fails, nothing's
deleted, so we never leave a half-deleted client. Idempotent
because every WHERE matches nothing on a non-existent client.
Test TestOAuth_DeleteOAuthClient_CascadesDependentRows seeds a row
in each of the four dependent tables, deletes the client, and
asserts ErrOAuthNotFound on every dependent row + the client itself.
Without the fix this fails on the first DELETE FROM oauth_clients
with an FK constraint violation.
* fix(oauth): SELECT FOR UPDATE row lock in DeleteOAuthClient on Postgres per Codex review (round 4)
Codex round 4 caught a Postgres race in DeleteOAuthClient: the
five-DELETE cascade is atomic, but between the child-row deletes
and the parent delete, a concurrent fosite handler can insert a
fresh grant/token referencing the same client_id. The parent
DELETE then fails with an FK violation and the whole tx rolls
back — the cascade is correct, but unreliable under concurrent
OAuth issuance.
Fix: take SELECT id FROM oauth_clients WHERE id = ? FOR UPDATE
as the very first statement in the tx (Postgres only). The
exclusive row-level lock blocks any concurrent statement that
tries to read the client row — which fosite does on FK resolution
during grant/token inserts — until our tx commits.
Skipped on SQLite because:
(a) BEGIN IMMEDIATE serializes the entire write workload globally
(DSN configures _txlock=immediate per store.go), so the race
doesn't exist.
(b) FOR UPDATE syntax isn't reliably accepted across SQLite
drivers.
ErrNoRows on the lock query is treated as "client doesn't exist
yet" — the subsequent DELETEs match nothing and the call remains
idempotent. Tests still pass on the SQLite path; the Postgres
path's race fix will be exercised by CI's PAD_TEST_POSTGRES_URL
runs and any future concurrency test we add.
|
||
|
xarmian
|
0f05012169 |
fix(mcp): six surgical fixes for BUG-987 (bugs 6, 8, 11, 12, 13, 14) (#361)
* fix(mcp): six surgical fixes for BUG-987 (bugs 6, 8, 11, 12, 13, 14)
Hotfix follow-up to v0.1.0-rc.3's Claude Desktop dogfood. Six
surgical fixes; bigger items (5, 7, 9, 10) deferred to separate
tasks.
- Bug 6: `pad project next --format json` was emitting the entire
dashboard, indistinguishable from `pad project dashboard --format
json`. Now slices to suggested_next only. cmd/pad/main.go.
- Bug 8: standup blockers carried empty `ref` strings, blocking
agent linkback to the actually-blocked items. dashboard's
attention[].item_ref is canonical; the standup composer in
internal/mcp/dispatch_http_slice4.go just wasn't propagating it.
Same fix applied to suggested_next entries.
- Bug 11: cobra's auto-emitted "Usage: pad item block ..." help
block leaked into MCP error envelopes via classifyExecError. The
Usage text references OLD CLI verb names (pre-v0.2 catalog) that
agents using the new surface have no business seeing, and bloats
every error response. New stripCobraUsageBlock helper truncates
stderr at the first line-anchored "Usage:" marker before
classification + envelope construction.
- Bug 12: BuildCLIArgs validation errors (missing required arg, type
mismatch) came out of env.Dispatch as bare-text NewToolResultErrorf
results, breaking the structured envelope contract. New helper
validationFailedFromBuildErr wraps them as ErrValidationFailed
envelopes with the field name extracted via regex from the
underlying message.
- Bug 13: every Task / Idea / Plan with a `priority` field got a
phantom `convention: { enforcement: "<priority>" }` surfaced on
its response, because ExtractItemConventionMetadata's legacy
fallback treated `priority` as the Convention enforcement tier
unconditionally. Restructured to track hasConventionShape
separately from hasMetadata — only Convention-specific markers
(structured convention field, trigger, scope, surfaces, commands,
direct enforcement) flip the shape flag. category alone is
insufficient (Ideas / Bugs / Roadmap items legitimately use it).
Final guard returns nil when only category was matched.
- Bug 14: GetRoleBreakdown's unassigned row was emitted with empty
role_name + role_slug, presenting as a "phantom" entry in the
dashboard. Now explicitly labelled "Unassigned" / "unassigned"
while keeping role_id null so it's still distinguishable from a
real role.
Tests:
- internal/mcp/bug987_test.go (new) — stripCobraUsageBlock + classify
+ validation envelope wrapping + env.Dispatch integration.
- internal/models/item_test.go — three cases covering non-Convention
items (Task, Idea, Plan with priority) returning nil metadata, and
one preservation test for legacy Conventions with priority field.
- internal/store/agent_roles_test.go (new) — confirms unassigned row
carries explicit "Unassigned" / "unassigned" labels.
Live verified: pad_project action=next returns just the suggestions
array; pad_item action=create with no fields returns validation_failed
with field=collection; pad_item action=link with self-target returns
without Usage-block leakage.
Deferred to separate items (per BUG-987 triage):
- Bug 5: text vs JSON returns across note, decide, star, unstar,
delete, bulk-update — needs CLI-side handler updates per command.
- Bug 7: suggested_next algorithm — needs to consider in-progress
items, not just open ones; behavior change needs design.
- Bug 9: fields/tags double-stringified — potentially breaking for
web UI/CLI consumers.
- Bug 10: decision_log/notes embedded in fields blob duplicating
top-level arrays — might require data migration.
Parent: BUG-987.
* fix(mcp): HTTP transport equivalence + ordering for BUG-987 per Codex review (round 1)
Two findings from Codex review of PR #361:
1. project.next on HTTP transport still returned the full dashboard.
The route table mapped "project next" directly to /dashboard, so
the CLI fix (slice to suggested_next) didn't reach OAuth-authed
agents going through HTTPHandlerDispatcher. Catalog actions must
produce equivalent shapes on stdio and HTTP — that's the contract
that lets agents be transport-agnostic.
Fix: new dispatchProjectNext method on HTTPHandlerDispatcher that
fetches the dashboard via the existing fetchDashboardJSON helper,
slices to suggested_next[], re-encodes, and runs through
packageJSONResult so it gets the same {items: [...]} wrap as
other list responses.
Also retires the broken route-table entry — replaced with a
comment pointing at the new method so future contributors don't
re-add a passthrough.
Test: TestDispatch_ProjectNext_SlicesToSuggestedNext + the empty-
array case. Asserts dashboard-only top-level fields (summary,
active_items) don't leak into the response — that's the whole
point of project.next being distinct from project.dashboard.
2. ExtractItemConventionMetadata's priority→enforcement legacy
fallback ran BEFORE surfaces/scope/commands had a chance to flip
hasConventionShape, so a Convention with only `{scope, priority}`
would silently drop enforcement.
Fix: move the priority fallback to AFTER all marker checks. Direct
`enforcement` still resolves first; the legacy priority fallback
runs at the bottom once shape detection is complete.
Tests: two new cases covering scope-only and commands-only legacy
Conventions — both must resolve enforcement via the priority
fallback.
Parent: BUG-987.
|
||
|
xarmian
|
a0336e0248 |
feat(attachments): bundle attachments + manifest in workspace export (TASK-884) (#305)
* feat(attachments): bundle attachments + manifest in workspace export (TASK-884)
GET /workspaces/{ws}/export?format=tar streams a gzip'd tar bundle:
pad-export.json # the existing WorkspaceExport JSON
attachments/manifest.json # uuid → {filename, mime, size, hash, ...}
attachments/<uuid>.<ext> # original blobs only — no thumbnails
Default (no ?format) keeps returning JSON so existing automation
hitting the endpoint without a query param continues to work
unchanged. The CLI's pad workspace export now opts into the bundle
by default; pass --json for the legacy items-only output.
Implementation:
- store.WorkspaceAttachmentsForExport returns originals only
(parent_id IS NULL); thumbnails are re-derived on import via the
existing pipeline so shipping them would double the bundle size.
- handleExportWorkspaceBundle streams chunks straight into the
response writer rather than buffering — a workspace with multi-
GB of attachments would otherwise pin that much memory.
- AttachmentManifest is versioned (separate from WorkspaceExport
version) so the bundle layout can evolve independently.
- bundleAttachmentPath is exported (lowercase package fn) so the
import path in TASK-885 can resolve manifest entries to tar
entries without duplicating the filename logic.
- CLI gates against writing binary tar.gz to a TTY and appends the
conventional extension when -o is passed without one.
Tests:
- TestExportBundle_RoundTrip: two uploads → bundle contains
pad-export.json + manifest + 2 blobs whose bytes match the
uploads + manifest decodes cleanly + WorkspaceExport decodes.
- TestExportBundle_HidesThumbnails: synthetic thumbnail row, the
manifest excludes it.
- TestExportBundle_LegacyJSONStillWorks: no ?format param returns
application/json with a decodable WorkspaceExport (backward
compat regression guard).
Parent: PLAN-866. TASK-885 (import path + UUID remap) consumes the
manifest produced here.
* fix(attachments): stream export bundle + revert default to JSON per Codex (round 1)
Two findings from Codex on PR #305:
1. CLI buffered the entire response in memory via RawGet → io.ReadAll,
defeating the server-side streaming design and risking OOM on a
multi-GB bundle. Added Client.RawStream which copies the response
body straight into an io.Writer; export now opens the target file
and streams directly into it.
2. Default tar.gz output broke `pad export → pad import` round trip
because the import handler still only accepts JSON. Reverted the
CLI default to JSON; bundle is now opt-in via --bundle. The flag
docstring notes that TASK-885 will flip the default once import
handles bundles.
* fix(attachments): surface tar/gzip close errors and truncation per Codex (round 2)
Codex round 2 finding: deferred tw.Close() / gzw.Close() ignored
errors. If a backend returned fewer bytes than size_bytes claimed,
io.Copy returned nil, the tar writer's "missed N bytes" trip fired
at Close, and the handler still completed a 200 OK with a corrupt
bundle that gunzip would later refuse to decompress — silently from
the operator's perspective.
Two changes:
1. The deferred close now logs both tw.Close() and gzw.Close()
errors with structured context, so a corruption-on-finalize
trip shows up in the operator log.
2. streamAttachmentToTar checks the bytes-copied count against
a.SizeBytes after io.Copy and returns a per-attachment error
when they disagree. The error is logged with attachment_id +
storage_key so an operator can correlate the corruption with
the row to investigate.
Regression test: TestExportBundle_TruncatedBlobLogsError forces a
size_bytes/blob desync via direct UPDATE and asserts the resulting
bundle bytes don't decode cleanly. (HTTP status stays 200 because
headers are already on the wire by the time we detect the desync;
that's an inherent limitation of mid-stream errors, but the new
logs + close-error surfacing make the failure observable.)
* fix(attachments): X-Bundle-Status trailer for export-stream success per Codex (round 3)
Codex P1 round 3: even with the per-blob truncation log + tar/gzip
close-error logs, mid-stream failures looked successful to clients.
The CLI's RawStream finished without a transport error, the file
landed on disk, and "Exported workspace" printed regardless of
whether the bundle was actually complete.
Two complementary signals now mark a clean stream:
1. HTTP trailer X-Bundle-Status. The handler declares the trailer
in the initial Trailer header and sets it to "ok" only after
tw.Close() and gzw.Close() both return without error. CLI checks
the trailer after streaming and discards the file + returns
error if it's absent or non-"ok".
2. The handler skips the deferred clean close on the error path,
leaving the gzip footer unwritten. A client that ignores the
trailer (curl, third-party tooling) still sees a corrupt gzip
stream that gunzip refuses to decompress.
CLI: pad workspace export --bundle now removes any partial output
file on failure rather than leaving a corrupt one behind.
Client.RawStream signature changed to return (bytes, *http.Response,
error) so callers can inspect resp.Trailer; the only caller is the
export command.
Tests: TestExportBundle_TruncatedBlobAbortsStream now asserts both
signals (trailer absent + gzip/tar can't fully decode), and
TestExportBundle_SuccessTrailer pins the happy-path trailer.
|
||
|
xarmian
|
d3a543db6f |
feat(attachments): admin per-user storage quota override UI (TASK-883) (#304)
* feat(attachments): admin per-user storage quota override UI (TASK-883)
Surfaces the storage_bytes plan_overrides key in the admin user-detail
page so operators can lift or tighten an individual user's quota
without poking at JSON via the API directly.
Frontend (console/admin/+page.svelte):
- Dedicated "Storage quota override" input below the existing
overrides grid. Storage is byte-counted, not row-counted, so a
number input forcing the admin to type 536870912 for 512MB
would be hostile. Accepts:
• "10 GB" / "500MB" / "1.5 GB" (IEC shorthand)
• "1024" (raw bytes)
• "-1" (unlimited)
• "" (clear → falls back to plan default)
- Live parse preview ("= 10.0 GB (10,737,418,240 bytes)") so the
admin can verify the unit was understood.
- "Reset to plan default" button clears the field; save commits
the absence as a removed override key.
- Pre-fills with the current effective override formatted in the
largest exact unit so a previously-set "10 GB" doesn't reload as
"10737418240".
Backend:
- ActionPlanOverridesChanged audit constant.
- handleAdminUpdateUser now logs an audit event with old/new
override JSONs whenever plan_overrides is patched. Lets operators
correlate a mysteriously-allowed upload with the override that
enabled it.
Tests:
- TestAdminUpdateUser_StorageOverrideRoundTrip: PATCH with
storage_bytes:1073741824 → GET shows the new override → audit
feed contains plan_overrides_changed event → clearing the
override removes it.
- TestAdminUpdateUser_NonAdminForbidden: member-role user cannot
PATCH another user's plan_overrides (regression guard for the
audit-log path).
Parent: PLAN-866. The Settings → Storage page (TASK-882) reflects
the new effective limit immediately after save because both call
the same WorkspaceStorageInfo helper.
* fix(admin): parse plan_overrides JSON on read, clear via empty string per Codex (round 1)
Two related bugs in the admin user-detail page that Codex caught
in PR #304 round 1:
1. The save path sent JSON null when every override field was
blank, but the Go handler uses a *string and JSON null decodes
to a nil pointer — the handler's existing nil-vs-non-nil branch
then skips the update, meaning "Reset to plan default" reported
success without actually clearing the override. Fixed by
sending "" (empty string) which routes through
SetUserPlanOverrides("") and clears the column.
2. The form-populate path treated u.plan_overrides as an object
while the API actually returns the raw column value as a JSON
string. So `'storage_bytes' in ov` was checking string indices
on a literal '{"storage_bytes":1073741824}' string, returning
false, and any user with stored overrides loaded a blank form.
This was a pre-existing bug in the workspaces / api_tokens /
etc. fields too — fixed for all of them by parsing the JSON in
parsePlanOverrides() before reading keys, with a defensive
"future-proof" branch in case the API ever switches to a
decoded object.
TS type for AdminUser.plan_overrides updated to `string | null`
to match the actual API contract.
Backend regression test added (TestAdminUpdateUser_OmittedOverrides
Preserved) that pins the other half of the contract: PATCH with
plan_overrides absent must NOT clear the column. The test was
straightforward to add because the existing test infrastructure
(bootstrapFirstUser, doRequestWithCookie) already covers the
admin auth path.
|
||
|
xarmian
|
6461aafd16 |
feat(store): attachments table + Attachment model (TASK-869) (#286)
Adds the schema groundwork for inline images and file uploads — see DOC-865 (Attachments — architecture & migration design). - migrations/047_attachments.sql — SQLite migration. Table + 4 indexes (workspace, item, hash, parent). Partial indexes on workspace/item/parent match the items table convention. The hash index is full (not partial) so dedupe can resurrect a soft-deleted blob if the same bytes are re-uploaded without writing a duplicate. - pgmigrations/026_attachments.sql — Postgres mirror with BIGINT for size_bytes; same partial-index pattern. - internal/models/attachment.go — Go model with all columns. Uses pointer types for nullable columns (item_id, width, height, parent_id, variant, deleted_at) so JSON omitempty works correctly. No call sites yet — purely schema groundwork. Verified the migration runs cleanly on a fresh install and on the live dev DB. Parent: PLAN-866. |
||
|
xarmian
|
7cda0d7896 |
feat: rebrand to Perpetual Software + new tagline (IDEA-832) (#273)
Migrates from xarmian/pad to PerpetualSoftware/pad across the entire
repo and updates the product subtitle to "Collaborate with your AI
agents".
Go module rename
- go.mod: github.com/xarmian/pad → github.com/PerpetualSoftware/pad
- All Go imports updated across cmd/pad, internal/{cli,server,store,
models,collections,items,events,metrics,webhooks} (~130 files)
- Test fixtures with the literal repo slug ("xarmian/pad" in JSON
shapes, SSH/HTTPS git URL strings, workspace_context fixtures)
also updated, including the secondary repo entry
(xarmian/pad-web → PerpetualSoftware/pad-web — pad-web was also
moved to the org per branch context)
Docs / config
- README badges, install instructions, brew tap, Docker image, source
build path, sponsor link (sponsor link kept as personal @xarmian)
- Subtitle: "Project management for developers and AI agents." →
"Collaborate with your AI agents." (README, manifests, web layout
meta, .goreleaser homebrew description)
- CONTRIBUTING.md, SECURITY.md, skills/INSTALL.md
- .goreleaser.yaml: homebrew_casks owner, GHCR image, release github
owner, cosign cert-identity regex, comments
- .github/workflows/release.yml: tap/release comments
- deploy/k8s/deployment.yaml: container image
- docs/deployment.md: clone URL
- web/static/{site.webmanifest,manifest.json}: description
- web/src/routes/+layout.svelte: meta description + og:description
Brew tap path is PerpetualSoftware/tap/pad (CamelCase, matches
GitHub user case). GHCR image is ghcr.io/perpetualsoftware/pad
(lowercased per GHCR's URL normalization). CODEOWNERS @xarmian and
FUNDING.yml github: xarmian intentionally retained — those are the
personal maintainer / sponsor account, separate from the org repo.
Verification: go build ./..., go test ./... (all pkgs pass), web
build, and make install all clean (TASK-844, TASK-845).
|
||
|
xarmian
|
afe721d202 |
feat(cli): add Cloud mode to pad init, drop Docker option (TASK-837, TASK-838) (#272)
Merging despite Go (PostgreSQL) red — those failures (TestListItems_FTS_HyphenatedSearchTerm/task-five + TestAdminBillingStats_SidecarSidecarError_DegradesToLocalOnly TempDir cleanup race) are pre-existing on main and tracked in BUG-842. Codex reviewed in 3 rounds (round 1 clean → round 2 found a real semantic bug → fix → round 3 clean). Tests, vet, and lint all green; remaining check failures are documented pre-existing. |