mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-22 18:43:45 +00:00
6433cc51ea1dd68b81f7b287c30385ad4400528f
244 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xarmian
|
de1beb47a9 |
feat(cli): pad library get + list --full + server-side category filter (TASK-1562) (#613)
CLI layer for PLAN-1560 (`pad_library` MCP tool + matching CLI surface).
Wires the HTTP work landed in TASK-1561 through to the `pad library`
subcommands.
## `pad library list` changes
- `--category` is now a server-side filter (the old client-side
display-only skip-loop is dead and removed).
- New `--full` flag. Default JSON output for playbooks now returns the
`summary` field (first non-heading paragraph, ~240 char cap) instead
of the full `content`; `--full` opts back into full bodies for
callers that want to pipe everything.
- Table output gains a summary hint line under each playbook and a
`/pad <slug>` chip when an invocation slug is declared, so the
library becomes self-documenting as a discovery surface.
- `--type` now validates explicitly instead of silently producing an
empty list for unknown values.
## NEW `pad library get <title>`
Calls `GET /api/v1/library/entry?title=X` and renders either a
conventions card (title, category, trigger, surfaces, enforcement,
commands, body) or a playbooks card (title, category, trigger, scope,
invocation slug, argument count, body). Conventions-first precedence
matches `pad library activate`.
JSON output returns the full envelope.
404 errors return a clean `not found in library: "<title>"` message
with exit code 1.
## CLI client
- `GetConventionLibrary(category)` — pass category as a server-side
query param.
- `GetPlaybookLibrary(category, summary)` — same plus the summary
toggle; `summary=true` strips Content and returns Summary instead.
- NEW `GetLibraryEntry(title)` returning `*LibraryEntryResponse`.
- `LibraryPlaybook` gained an omitempty `Summary` field so a single
type round-trips both the legacy and summary shapes.
## Drive-by
Switched `/library/entry` 400/404 from a flat `{error: "..."}` body to
the canonical `writeError(code, message)` envelope used by the rest of
the API. The CLI's `parseError` now hands back a typed `APIError` that
`pad library get` pattern-matches on `Code=="not_found"` for the clean
404 message. Updated `TestLibraryEntry_MissingTitle` and `_NotFound`
to assert the new envelope.
## Verification
go build / go vet / go test ./... all green. golangci-lint clean on
cmd/pad/..., internal/cli/..., internal/server/.... End-to-end smoke
tests via the installed binary confirmed: list summary mode, list
--full, list --category filter, get convention card, get playbook
envelope, get 404 exit-1, --type validation.
Parent: PLAN-1560. Unblocks TASK-1563 (MCP catalog wiring).
|
||
|
xarmian
|
2df6edeaab |
feat(server): library endpoints gain ?category, ?summary, /library/entry (TASK-1561) (#612)
Extends the convention + playbook library HTTP layer to support the shape
the upcoming `pad_library` MCP tool and the updated `pad library` CLI need:
- `GET /api/v1/convention-library?category=X` — server-side filter,
case-sensitive exact match. Unknown categories return an empty slice,
not 404.
- `GET /api/v1/playbook-library?category=X&summary=true` — same filter
plus a new summary mode that strips Content and injects Summary
(first non-heading paragraph, ~240 char cap). Web UI and existing
consumers omit the flag and see the legacy full-body shape. Summary
mode deep-copies category slices so a request never mutates the
package-level library data — TestPlaybookLibrary_SummaryDoesNotMutate
Global pins this.
- `GET /api/v1/library/entry?title=X` — NEW. Returns one matched entry
in a `{type, convention|playbook}` envelope. Conventions-first
precedence mirrors the dispatcher's `library activate` so a title
resolves to the same kind in both surfaces. 400 on missing title,
404 on no match.
Hoisted `playbookSummary` to `collections.PlaybookSummary` so the
bootstrap handler and the new library endpoints share one algorithm.
Bootstrap continues to call it for every playbook entry it returns.
Adds 12 handler tests + the existing bootstrap-summary test stays
green after the move. Lint clean on touched packages; `make check`
gate is blocked by a pre-existing gofmt issue in
internal/store/workspace_members.go captured as BUG-1565.
Parent: PLAN-1560. Unblocks TASK-1562 (CLI) and TASK-1563 (MCP catalog).
|
||
|
xarmian
|
8a85eca713 |
feat(web): admin user table — cheap aggregation columns (TASK-1548) (#603)
* feat(web): add cheap aggregation columns to admin user table (TASK-1548)
Surfaces the per-user aggregations T1544 added to GET /admin/users:
- Workspaces (numeric, after Role)
- Storage (used bytes, formatted via the existing formatStorageBytes
helper — same units the storage-override field accepts)
- Last Write (relative time, color-coded by writeRecency: green <7d,
yellow <30d, red ≥30d, gray italic when never)
- Status pill (replaces the standalone "disabled" badge; renders for
disabled / no-workspace / inactive; suppressed for "active" to keep
the table calm)
Implementation:
- AdminUser interface in admin.svelte.ts gains last_write_at,
workspace_count, storage_bytes, status — matching the server-side
JSON shape from T1544.
- writeRecency() helper in +page.svelte buckets the timestamp into a
CSS class. Visual half of the API's status pill; same age windows.
- .num-cell utility: right-aligned, tabular-nums so digits line up
across rows (workspace_count and storage cells share it).
- edit-row colspan updated to 9 (cloud_mode) / 8 (self-hosted) to span
the new columns.
Frontend purely additive — no API changes, no Go changes. T1549 wires
up pagination + sort + filter; T1550-T1555 build the modal.
Part of PLAN-1542.
* fix: address Codex review on TASK-1548
1. handleAdminGetUser response now includes last_write_at, storage_bytes,
and status — matching handleAdminListUsers' shape. Without these,
the row-merge after PATCH (role change, disable, plan edit) kept
stale values; e.g. disabling an active user wouldn't show the new
"disabled" status pill until the full list reloaded.
- Store.UserStorageUsage: new helper that sums attachments across
all workspaces owned by the user. Mirrors WorkspaceStorageUsage's
definition.
- Store.ComputeAdminUserStatusValue: exported wrapper around the
existing private helper so handler code can compute the pill
value without round-tripping through SearchUsers.
2. writeRecency threshold: 30d is now "stale" (inclusive) rather than
"cold". Matches server-side computeAdminUserStatus which only flips
to "inactive" on > 30 days. Eliminates a 1-day boundary mismatch
between the recency color and the status pill.
3. A11y: write-recency cell now carries aria-label with the bucket
name ("Last write: 12d ago (stale)"), so screen-reader users get
the same meaning the color conveys.
|
||
|
xarmian
|
48323e229e |
feat(admin): GET /admin/users/{id}/metrics windowed engagement metrics (TASK-1547) (#602)
Final backend task for PLAN-1542. Returns three engagement signals that
power the metric tiles on the admin user modal's Overview tab (T1553):
- days_since_write: derived from users.last_write_at (T1543). nil when
the user has never had a write recorded.
- writes_7d: COUNT of write-class activities (created/updated/archived/
restored/moved/commented) authored in the last 7 days.
- collections_touched_30d: COUNT(DISTINCT collection_id) of items the
user has authored writes for in the last 30 days. Goes through
activities.user_id (not items.last_modified_by, which is an attribution
string — see T1543's architecture note).
api_requests_7d is intentionally NOT included; no per-request log exists.
Filed as a follow-up (IDEA-1556) that will add this as an additive,
non-breaking field once the request-log table lands.
Implementation:
- Store.GetUserMetrics in users.go runs three small queries: scalar
SELECT for last_write_at, one COUNT(*) over activities, and a
JOIN(activities, items) for the DISTINCT collection count. All
three are index-backed (idx_activities_user from migration 022).
- No caching layer in this PR. The queries are cheap, and a per-user
short cache fits more naturally at the handler boundary if needed —
premature here.
- Handler handleAdminGetUserMetrics wired at GET /admin/users/{userID}/metrics.
requireAdmin gate; 404 on missing user.
Tests: TestGetUserMetrics seeds a workspace with two collections, six
activities (five inside 7d, one ancient outside both windows), verifies
all three metrics. TestGetUserMetricsEmptyUser covers the no-activity
case (nil days_since_write, zero counts, no error).
|
||
|
xarmian
|
a04e5217c6 |
feat(admin): GET /admin/users/{id}/activity paginated feed (TASK-1546) (#601)
* feat(admin): GET /admin/users/{id}/activity paginated feed (TASK-1546)
New endpoint returns activities originated by the user — item writes,
comments, account-level actions the user took themselves — in reverse-
chronological order with offset pagination.
Scope decision: feed shows activities where activities.user_id = userID
(events the user authored). Admin actions targeting this user as a
subject (role_changed where target_user_id is in metadata) are NOT
included; that "received" sub-feed needs a JSON predicate and is filed
as a follow-up. T1554 (modal Activity tab) consumes the current shape.
Implementation:
- Store.ListUserActivity mirrors the existing ListWorkspaceActivity /
ListDocumentActivity helpers in activities.go. Same column projection,
same LEFT JOIN users u for actor name. Hard cap at limit=50.
- Handler asks for limit+1 rows so it can flag "next_offset" without a
separate COUNT query — trims the extra before responding. Returns
next_offset=null when the page is the last.
- Route wired at GET /api/v1/admin/users/{userID}/activity. 404 on
missing user; requireAdmin gate.
- Pagination is offset-based (matching the sibling endpoints) rather
than cursor-based as the task body suggested. For per-user feeds the
dataset is bounded and between-page drift is acceptable for an admin
tool. Cursor can be added later if needed; the response shape is
forward-compatible (next_offset → next_cursor would just rename).
Test: TestListUserActivity covers cross-user isolation, action filter,
offset pagination across pages without overlap, and the 50-row hard cap.
Part of PLAN-1542.
* fix: address Codex review on TASK-1546
Lift the store-side cap on ListUserActivity from 50 to 100. The handler
caps the public per-page at 50 and asks the store for limit+1 (51) to
flag "more available" without a separate COUNT. Previous store-side
cap of 50 silently truncated that probe, so next_offset would be null
even when row 51 existed — clients iterating at page-size=50 would stop
one page short of the actual end.
The HTTP layer remains the source of truth for the per-page maximum;
the inner cap is now just protection against pathological internal
callers. Regression test seeds 51 activities and verifies the store
returns all 51 when asked.
|
||
|
xarmian
|
eff0824238 |
feat(admin): GET /admin/users/{id}/detail per-workspace breakdown (TASK-1545) (#600)
* feat(admin): GET /admin/users/{id}/detail per-workspace breakdown (TASK-1545)
New endpoint that returns the user vitals plus a per-workspace breakdown
enriched with the aggregations the admin user modal's Workspaces tab
needs: collections_count (excluding system collections — playbooks,
conventions, anything else is_system=1), items_open (status NOT IN a
hardcoded terminal set), items_total, members_count, storage_bytes
(matches WorkspaceStorageUsage's definition), and last_activity_at
(MAX items.updated_at across non-deleted items).
Implementation:
- Store.GetUserWorkspacesDetailed in workspace_members.go uses correlated
subqueries rather than a wide JOIN+GROUP BY — a single user belongs to
at most tens of workspaces in practice, so the readability wins over
micro-optimizing. Caps at 50 rows (frontend caps at 20 in T1552).
- AdminUserWorkspaceDetail embeds the existing AdminUserWorkspace to keep
the JSON shape backward-compatible with /workspaces consumers.
- adminOpenItemTerminalStatuses is a hardcoded list (done/completed/
rejected/archived/implemented/cancelled). A schema-aware terminal_options
check is a separate follow-up — flagged in the struct doc.
- Handler handleAdminGetUserDetail wired at GET /admin/users/{userID}/detail.
Returns 404 on missing user; defensive []AdminUserWorkspaceDetail{}
serialization so JSON consumers see [] rather than null.
Test: TestGetUserWorkspacesDetailed seeds a workspace with two user-facing
collections, one system collection, three items (two open, one terminal),
two members, and one attachment; verifies all six aggregations.
Part of PLAN-1542. T1552 (modal Workspaces tab) consumes this.
* fix: address Codex review on TASK-1545
Three real issues in the items_open count clause, all in the same SQL
fragment:
- Use s.dialect.JSONExtractText("i.fields", "status") instead of the
SQLite-only JSON_EXTRACT(i.fields, '$.status'). The endpoint would
have failed on Postgres deployments.
- Wrap the extracted value in LOWER(COALESCE(..., '')) so items with
NULL/missing status fields still register as "open" (NULL NOT IN
(...) is not TRUE in SQL, which would have undercounted), and so
case-variant statuses match. Matches the interpretation used in
search.go and items.go.
- Source the terminal list from models.DefaultTerminalStatuses rather
than a private list — previous local list was missing 'resolved',
'wontfix', 'fixed', 'disabled', 'deprecated' (overcounting open
items in workspaces that use those statuses).
adminOpenItemsCountClause is now a Store method (was a package-level
fn) because it needs the dialect.
|
||
|
xarmian
|
0c5ec04fac |
feat(admin): extend user list with aggregations + sort/filter (TASK-1544) (#599)
* feat(admin): extend user list with aggregations + sort/filter (TASK-1544)
GET /admin/users now returns per-user workspace_count, storage_bytes,
last_write_at, and a computed status pill (disabled / no-workspace /
inactive / active, with documented precedence). Adds sort and filter
knobs so the table can scale beyond the existing fixed offset/limit.
Store layer:
- AdminUserSearchParams gains Role, Sort, Order, ActiveWithinDays,
HasWorkspaces, Disabled. Pointer types where tri-state ("no filter"
vs. "filter to false") matters.
- AdminUserListEntry wraps models.User with WorkspaceCount, StorageBytes,
Status — returned in AdminUserSearchResult.Users.
- SearchUsers SQL rewritten: LEFT JOIN against grouped subqueries so
one user owning N workspaces with M attachments each still produces
exactly one row (no aggregation explosion). Both subqueries filter
deleted_at IS NULL to match WorkspaceStorageUsage's existing
definition. Allow-listed sort clause prevents injection.
- computeAdminUserStatus exported for unit tests; precedence locked in
by TestComputeAdminUserStatus.
- TestSearchUsersAggregations covers workspace_count + storage_bytes +
status across a three-user fixture and each new filter/sort knob.
Model + scanner:
- models.User gains LastWriteAt. userColumns + scanUser updated; the
legacy callers (GetUser, ListUsers, etc.) inherit the new field for
free via the shared scanner.
Handler:
- handleAdminListUsers accepts the new params: role, disabled,
has_workspaces, active_within_days, sort, order. Tri-state bools
only fire when the query param is present. Response now embeds
workspace_count / storage_bytes / last_write_at / status.
Part of PLAN-1542. Frontend consumption lands in T1548 (cheap columns)
and T1549 (sort/filter UI).
* fix: address Codex review on TASK-1544
- Tri-state bool parsing in handler now uses strconv.ParseBool — accepts
the canonical truthy/falsy variants ("True"/"TRUE"/"t"/"1" and the
parallel falses), and silently ignores garbage values rather than
treating them as false. Closes the "disabled=TRUE silently means
enabled-only" surprise.
- SearchUsers count query no longer joins the storage aggregation when
HasWorkspaces isn't an active filter. The page query still needs both
joins (the row carries the data), but a typical "give me a count"
call no longer scans every live attachment. The workspace_count join
remains conditional on HasWorkspaces filtering.
Status threshold (>30d vs >=30d): the documented spec and impl both say
">30d" — no change.
|
||
|
xarmian
|
0a09c1dca7 |
feat(store): add users.last_write_at column + write-path hook (TASK-1543) (#598)
* feat(store): add users.last_write_at column + write-path hook (TASK-1543)
Engagement metrics need a "last write" signal distinct from last_active_at
(which is bumped on any authenticated request, so it includes reads). Adds:
- Migration 060: users.last_write_at + index. Backfills from activities
table (the canonical record of who-did-what) using the action set
that handlers_items.go / handlers_comments.go actually emit:
created/updated/archived/restored/moved/commented.
- Store.TouchUserWrite(ctx, userID): mirrors TouchUserActivity. Same
5-minute throttle to avoid write-amplification, silent no-op on
empty userID so callers don't have to guard.
- Hook in logActivityWithMetaReturningID (handlers_documents.go) — every
item-write action funnels through this single helper, so one TouchUserWrite
call covers item create/update/archive/restore/move and comment authoring.
- Explicit hook in handlers_attachments.go after CreateAttachment, since
uploads don't go through logActivity.
Test: TestTouchUserWrite covers empty-userID no-op, first-write set,
in-throttle suppression, and out-of-throttle advance.
Architecture note: items.last_modified_by / comments.created_by are
attribution strings ("user"/"agent"/"cli"), not user IDs. The user
identity lives in activities.user_id, populated at the handler layer
where currentUser is in scope. That's why the hook lives in handlers,
not the store layer — and why the backfill reads from activities.
Part of PLAN-1542 (admin user management enhancements).
* fix: address Codex review on TASK-1543
- Add Postgres migration 039 (pgmigrations counterpart to 060). Same
ALTER + index + activities-backfill, using TEXT to match the existing
last_active_at / disabled_at column types in pgmigrations/020-021.
Without this, Postgres deployments silently no-op TouchUserWrite
because the column doesn't exist (and the call's UPDATE error is
swallowed by design).
- Hook TouchUserWrite in handleCreateCommentReply. The reply handler
doesn't go through logActivity (no "commented" activity emitted for
replies — verified by grep), so the activity-helper hook misses it.
Explicit call after a successful CreateComment.
|
||
|
xarmian
|
e27f805ffc |
feat(web): unify dashboard onboarding banners around needs_onboarding signal (TASK-1530) (#594)
IDEA-1516 Phase 3. The pre-IDEA-1516 design split workspace onboarding
guidance across two banners — OnboardingIdeaBanner (gated on the
retired IDEA-1 / BACK-1 / FEAT-1 seed-item pattern from PLAN-1496) and
OnboardingChecklist (gated on a totalItems === 0 heuristic that
predates the canonical needs_onboarding flag from TASK-1504). Both
fired competing CTAs on the same screen; neither read the canonical
signal.
Backend (internal/server/handlers_dashboard.go):
- Add `NeedsOnboarding bool json:"needs_onboarding"` to
DashboardResponse, populated via the existing
Store.WorkspaceHasUserCreatedItems EXISTS query (same predicate
AgentBootstrap.NeedsOnboarding uses). Web reads it from the
dashboard fetch the page already does — no second round-trip
against the heavier bootstrap endpoint.
Frontend:
- Add `needs_onboarding: boolean` to TS DashboardResponse type
- Delete OnboardingIdeaBanner.svelte entirely (signal retired,
no remaining consumers); the back-end onboarding_seed field
stays for now per spec — separate cleanup
- Delete OnboardingChecklist.svelte; replace with
OnboardingNudgeBanner.svelte — single message + "Connect agent →"
CTA that opens the workspace's already-mounted
ConnectWorkspaceModal. Dismissible, preserves the existing
`pad-onboarding-dismissed-{wsSlug}` localStorage key so users who
dismissed the old checklist don't get re-prompted
- Workspace +page.svelte: collapse the two banner blocks into one
gated on `needsOnboarding && !onboardingDismissed`; reshow button
follows the same signal. Drop the now-orphaned `.connect-card`
CSS — its function is subsumed by the banner's CTA. Drop the
unused OnboardingIdeaBanner / OnboardingChecklist imports and the
`onboardingSeed` derived state
Smart-suppression deferred to a follow-up. The existing
api.workspaces.claimCode endpoint returns suppression info but
generates a real claim code as a side effect in the not-suppressed
case — calling it on every workspace page-load with needs_onboarding=true
is awkward. The CTA still opens the modal, which renders its own
suppression state correctly; users get the right experience with one
extra click on the rare suppressed case. A dedicated read-only
GET /workspaces/{ws}/connect-status endpoint is a separate piece
of work.
|
||
|
xarmian
|
8041b46e36 |
fix(sse): surface write errors and link keepalive to IdleTimeout (BUG-1532) (#590)
Two SSE-handler tidy-ups flagged during the BUG-1531 investigation. 1. writeSSEEvent now returns the underlying fmt.Fprintf error. Previously every event/keepalive write swallowed any error from the response writer — when the client TCP went away the handler kept looping, pulling events off the bus channel, and discarding them while waiting for the ctx.Done() cancellation to propagate. Now any write failure exits the handler immediately so the bus subscription is released and we stop fanning broadcast traffic into a dead socket. Marshal errors stay local (don't tear down a healthy stream over one un-marshalable payload). All five callsites + the keepalive Fprintf are updated to log at DEBUG (broken-pipe on client disconnect is normal traffic, not an error worth WARN-level noise) and return. 2. The 30s keepalive interval and the 120s IdleTimeout now live in named constants (sseKeepaliveInterval, httpIdleTimeout) with an init() guard that panics if `3 × keepalive >= IdleTimeout`. Used to be magic numbers in two files; bumping one without the other in lockstep silently created a window where idle SSE streams would get TCP-reset by the http.Server's idle deadline. The init guard fires at process start so a misconfigured constant is visible immediately, not three months from now when someone notices intermittent reconnect storms. Three new tests pin the contracts: - TestWriteSSEEvent_SurfacesWriteErrors - TestWriteSSEEvent_MarshalErrorIsLocal - TestSSEKeepaliveIdleTimeoutInvariant Closes BUG-1532. Full ./internal/server suite passes (~78s). |
||
|
xarmian
|
b801867053 |
chore(web): npm audit fix — svelte 5.55.8, devalue 5.8.1, mermaid 11.15.0 (#589)
* chore(web): npm audit fix — patch-bump svelte, devalue, mermaid
Resolves three advisories surfaced by the Web CI job (and visible on
recent main commits before this):
- svelte: 5.55.5 → 5.55.8 (moderate × 4: SSR XSS via spread,
hydratable Promise XSS, DOM clobbering, ReDoS in
<svelte:element>)
- devalue: 5.6.x → 5.8.1 (high: DoS via sparse array deser)
- mermaid: 11.14 → 11.15.0 (moderate × 4: Gantt infinite-loop DoS,
classDef CSS/HTML injection, config CSS injection)
All three are in-range patch bumps — package.json untouched, only the
lockfile changes. No major bumps, no API churn.
Tiptap deliberately untouched: @tiptap/core, @tiptap/extension-
collaboration, @tiptap/y-tiptap stay at 3.22.5 — the CLAUDE.md
lockstep rule (coordinated bumps + Y.Doc schema-version bump) only
applies when those three move together, and nothing here does.
Verified:
- npm audit: 0 vulnerabilities
- npm run check: 0 errors (existing 6 warnings unchanged)
- make build + make install: clean, server boots and serves the new
bundle.
* test(oauth): TestConsent_ApproveWithSpecificWorkspaces locates connection by shape
The test asserts the persisted oauth_connections row from a specific-
workspaces consent (allowed_workspaces=[alpha,beta]) has
AllCurrentWorkspaces=false and the right slug list. It indexed via
conns[0], which only worked when the approve flow was the most-recent
connection — but the test also calls runAuthCodeFlow above the
assertion to mint a bearer for the introspect call, and that helper
posts allowed_workspaces=["*"] → wildcard connection. With
ListUserOAuthConnections ordered ConnectedAt DESC, conns[0] is now the
wildcard bearer row, not the alpha+beta row this test is verifying.
We can't move the bearer-mint below the assertion (the introspect call
needs the bearer first), and the auth-code response doesn't surface the
request_id we'd need to look up the right connection directly. The
specific-workspaces flow is the only one in this test with
AllCurrentWorkspaces=false, so locate by that shape — surgical fix that
matches the test's actual intent.
Verified with `go test -count=3 -run TestConsent_ApproveWithSpecificWorkspaces`
(deflakes across orderings) and the full ./internal/server suite.
|
||
|
xarmian
|
bfce069f28 |
fix(web,build): search palette hang on numeric query + graceful SSE shutdown (BUG-1531) (#588)
* fix(web,build): search palette hang on numeric query + graceful SSE shutdown (BUG-1531) CommandPalette's reactive `$effect` subscribed to every workspace's `localSearch.epoch` + `localIndex.bootstrapStateFor`. Bare-digit queries short-circuit to `exactItemNumberLookup` (synchronous, very fast) and stacked re-fires of the effect inside one microtask tick whenever an SSE delta arrived — Svelte tripped `effect_update_depth_exceeded` and the palette froze. Treat bare-digit queries the same as `body:` queries (skip the subscription reads) and wrap `doSearch()` in `untrack()` so its internal reactive reads can't smuggle hidden dependencies into the effect. The SSE churn that fanned the loop was rooted in `make install` using `killall -9` — SIGKILL drops every open SSE stream mid-chunk so every browser tab logs `ERR_INCOMPLETE_CHUNKED_ENCODING` and reconnects. Switch to SIGTERM + 5s wait + SIGKILL fallback so the server's existing graceful-shutdown path (cmd/pad/main.go:811-857) actually runs and the http.Server writes a final 0-chunk on each open stream. Follow-up tidy-ups (unchecked write errors in writeSSEEvent, link the 30s keepalive to the 120s IdleTimeout in code) tracked in BUG-1532. * fix(web): track workspace slug in palette $effect per Codex review (round 1) After wrapping doSearch() in untrack(), the workspaceStore.current?.slug read that doSearch performs at line 209 no longer registered as a tracked dep of the search effect. The non-body / non-bare-digit branch still reads the slug via localIndex.bootstrapStateFor(...), so workspace switches re-fire the effect for that branch — but body: and bare-digit queries skip that block entirely. Without an explicit slug subscription they wouldn't re-dispatch on workspace switch; an in-flight server response would land stale, get discarded by isSameDispatch(), and loading could stick true. Hoist `void workspaceStore.current?.slug` into the unconditional void block so all four query shapes re-fire on workspace switch. Refs BUG-1531. * chore: gofmt handlers_claim_code_test.go Drive-by formatting fix to unblock CI on this PR. The file landed slightly unaligned in #586 (TASK-1525) — gofmt straightens the struct tag column on claimCodeResponse. |
||
|
xarmian
|
fc6afd01be |
feat(connect): unified Connect-to-agent modal + claim-code endpoint (TASK-1525) (#586)
* feat(connect): unified Connect-to-agent modal + claim-code endpoint (TASK-1525)
Phase E of PLAN-1519. Repurposes the avatar-menu "Connect a project…"
modal as a one-stop hub where users can connect ANY agent surface
(claim-code → existing OAuth grant, fresh MCP OAuth, or local CLI)
to the current workspace.
Backend
- GET /api/v1/workspaces/{slug}/claim-code — generates a stateless
6-digit HMAC claim code (re-uses the verifier's secret + bucket
math) for the calling member, OR reports `suppressed: true` when
smart-suppression detects the workspace is already covered by one
of the user's active OAuth connections (wildcard OR explicit
allow-list rows). Returns `expires_at` at the current bucket
boundary so the UI can drive a countdown.
- store.IsWorkspaceCoveredForUser — single indexed query against
oauth_connections + oauth_*_tokens; filters by ACTIVE tokens so a
dangling revoked connection row doesn't suppress fresh modals.
- Tests cover 412 (disabled), 404 (non-member), 200 + matching code,
wildcard suppression, explicit-allow-list suppression, and the
revoked-connection-doesn't-suppress invariant.
Frontend
- ConnectWorkspaceModal rewritten as a tabbed unified modal:
- Agent (claim code) — fetches on activate, live countdown,
auto-refetches at bucket roll-over, renders smart-suppression
panel that links to /console/connected-apps, and renders the
locked prompt block
`Authorize the pad workspace '<slug>' with claim code <code>.`
per IDEA-1517 §4.
- MCP setup — subsumed from the now-deleted ConnectMCPModal: URL
block + client-card grid linking to per-client docs.
- CLI — existing install + `pad init` flow, unchanged.
- Default tab: Agent when mcpPublicUrl is set; CLI when not. MCP tab
hidden entirely on self-host without a public MCP URL.
- ConnectBanner simplified: single modal state, generic "Connect an
AI agent to this workspace" copy, no MCP/CLI dual-modal branching.
- ConnectMCPModal.svelte deleted (fully subsumed).
- API client gets `workspaces.claimCode(slug)` + `ClaimCodeResponse`
TypeScript type.
- TopBar + workspace home callsites pass `mcpPublicUrl` from
authStore so the unified modal can pick the right default tab.
Verification
- go build ./... clean
- go test ./... — all packages green (server + store)
- cd web && npm run build — clean
Parent: PLAN-1519. Phases A-D already shipped (oauth_connections
schema, MCP claim action, /authorize redesign, connections-page
mutation UI); this lands Phase E. Phase F (TASK-1526) will wire
post-create auto-open from IDEA-1516's new-workspace modal; Phase G
(TASK-1527) is cross-agent paste validation of the locked prompt
string.
* fix(connect): require membership at claim-code generation; guard modal against stale-response races per Codex review (round 1)
1. Guest-grant generation gap. RequireWorkspaceAccess admits item-grant
guests who aren't workspace members; claim-code REDEMPTION requires
full membership. Generating without the same check handed guests a
valid-looking code + prompt that the claim endpoint always 404s.
Add an explicit GetWorkspaceMember check after getWorkspace and
return 403 not_a_member to fail closed on the same response shape
the redemption path would have used.
2. Stale-response race in the modal. ConnectWorkspaceModal stays
mounted across workspace switches (TopBar reuses the same
instance), so an older claimCode fetch can resolve AFTER a newer
one and stomp claimState with suppression or a code for the wrong
workspace. Add a monotonic seq + captured-slug guard mirroring the
refreshHasAgentActivity pattern already in ConnectBanner.
Test additions:
- TestHandleWorkspaceClaimCode_GrantOnlyGuest_403 asserts a non-member
authenticated caller never gets a 200 + code from the generation
endpoint.
|
||
|
xarmian
|
26aa800b3b |
feat(oauth): connected-apps mutation endpoints + edit UI (TASK-1524) (#585)
* feat(oauth): connected-apps mutation endpoints + edit UI (TASK-1524)
Phase D for PLAN-1519. Adds four mutation endpoints under
/api/v1/connected-apps/{id}/... and extends the console page with
an inline Edit panel per connection card.
Backend (internal/server/handlers_connected_apps.go + server.go)
- PATCH .../name — rename, trim + cap at 120 chars
- PATCH .../flags — atomic set of may_create / all_current /
include_future (rejects toggling
all_current=false when the join table
is empty — empty-allow-list invariant
from IDEA-1517 §3 Acceptance)
- POST .../workspaces — add workspace; membership-checked, 404
uniform when the user isn't a member
(no enumeration leak)
- DELETE .../workspaces/{slug} — remove; idempotent for missing
slugs; rejects last-workspace removal
when all_current=false (same orphan
guard as the flags handler)
All four route through requireConnectionOwner which returns the
same 404 envelope for non-owned connections as the existing
Revoke endpoint. Each handler echoes the updated DTO so the page
can re-render in place; respondWithConnection handles both
active-token chains (via ListUserOAuthConnections) and connections
without token rows (direct fetch of oauth_connections + the access
projection).
DTO (connectedAppDTO) gains name + the three scope flags. Model
already carried the fields (TASK-1522).
Frontend (web/...)
- ConnectedApp TS type extended; api.connectedApps gains
rename/updateFlags/addWorkspace/removeWorkspace methods.
- Connections page: Edit button per card opens an inline panel
with: connection name input (debounced save), three scope-flag
toggles (auto-save), allow-list chips with X-to-remove plus a
workspace picker that lists memberships not already in the list.
- Last-workspace removal disabled at the UI level (chip-remove
disabled when list length <= 1); API enforces the same invariant
if a tampered call slips through.
- Workspaces fetched lazily on first Edit open (cached for the
page lifetime).
Tests
- 7 handler tests cover happy paths + edge cases:
- rename trims/caps, non-owner 404
- flags happy path + empty-allow-list block
- add workspace happy path + non-member 404
- remove workspace happy path + last-blocked + idempotent missing
- loginTestUserAs helper to seed a second user for the non-owner
case (the existing loginTestUser hardcodes a single email).
Parent: PLAN-1519.
* fix(oauth): wildcard→specific toggle now works after pre-stage per Codex review (round 1)
PR #585 round 1 caught that switching all_current_workspaces from
true to false was effectively impossible: the flags handler's
empty-allow-list guard called GetOAuthConnectionAccess, which
intentionally short-circuits on wildcard and reports zero slugs.
Even with join rows present, the guard always tripped. The UI
compounded the issue by hiding the allow-list editor while in
wildcard mode, so users had no way to pre-stage workspaces.
Backend fix: new Store.ConnectionWorkspaceCount(requestID) returns
the raw join-row count regardless of the parent's wildcard flag.
The flags handler now uses this; the remove-workspace handler's
"orphan guard" also routes through the new count (plus an
IsConnectionWorkspaceAllowed probe so a no-op removal of a slug
that isn't even in the list doesn't trip the guard).
Frontend fix: the allow-list editor renders unconditionally inside
the Edit panel. While wildcard is on, an "Inert while wildcard is
on" badge clarifies that staged workspaces don't take effect until
the user flips the flag. Chip-remove disabled only when actively
in specific mode AND about to drop to zero — wildcard-mode removes
are always allowed.
Added TestHandleUpdateConnectedAppFlags_WildcardToSpecific_AfterPrestage
as the regression guard: seeds a wildcard connection, adds one
workspace via the API, then asserts the flag flip succeeds and
the resulting DTO carries the pre-staged slug.
Parent: PLAN-1519.
* fix(oauth): staged-while-wildcard rows now visible per Codex review (round 2)
PR #585 round 2 caught that the round-1 fix (always-rendered
allow-list editor + Backend ConnectionWorkspaceCount) was
incomplete: pre-staging a workspace while wildcard=true succeeded
on the server but the response DTO still suppressed the staged
slugs (ListUserOAuthConnections + respondWithConnection both set
AllowedWorkspaces=nil when AllCurrentWorkspaces=true). The UI
rendered "No workspaces staged" even after a successful add, so
users had no way to see or remove a mistaken staged row.
Backend fix: new Store.ListConnectionWorkspaceSlugs returns the
join table's slugs regardless of the wildcard flag.
ListUserOAuthConnections + respondWithConnection both route
through it now. The hot-path GetOAuthConnectionAccess still
short-circuits on wildcard (correct for the introspection path —
when wildcard is on, slugs are irrelevant for gating); the read-
for-display path needs to surface them.
Frontend fix: isAnyWorkspace now reads the all_current_workspaces
flag directly (drives the "Any workspace" badge), independent of
the slug list. The slug list drives the edit panel chips. Legacy
fallback for missing-flag wire shapes preserved.
Added TestHandleAddConnectedAppWorkspace_VisibleUnderWildcard as
the regression guard: seeds a wildcard connection, adds a
workspace, asserts the slug appears in DTO.AllowedWorkspaces
while AllCurrentWorkspaces stays true.
Parent: PLAN-1519.
|
||
|
xarmian
|
93b9590bf9 |
feat(oauth): consent screen rewrite + new-tables write path (TASK-1523) (#584)
Phase C2 for PLAN-1519. Rewrites the /oauth/authorize consent flow
per IDEA-1517 §2a: collects a display name, three scope flags
(may_create_workspaces, all_current_workspaces, include_future_workspaces),
and the per-workspace allow-list — then writes oauth_connections +
oauth_connection_workspaces instead of session.Extra.
Backend (internal/server/handlers_oauth.go)
- parseConsentPayload returns a structured consentDecision carrying
name, three flags, and resolved workspace IDs. Two-radio +
checkbox UI maps to flags per §2a's table; backward-compat shim
still accepts the legacy allowed_workspaces=["*"] / explicit-list
shape so pre-TASK-1523 clients and fixtures keep working.
- handleOAuthAuthorizeDecide INSERTs oauth_connections (parent) +
oauth_connection_workspaces (children) BEFORE NewAuthorizeResponse.
Ordering matters: an INSERT failure prevents code minting, so
the failure mode is "user retries" rather than "token issued
with no connection-level state" (which the dual-read gate would
silently broaden to no allow-list). A successful INSERT followed
by NewAuthorizeResponse failure leaves an orphan oauth_connections
row — harmless, no token references it. Per-slug INSERT failures
roll back via DeleteOAuthConnection (FK CASCADE clears the join).
- session.SetAllowedWorkspaces call retired; the dual-read
introspection gate from Phase A still consults legacy session.Extra
for pre-TASK-1523 tokens during the soak period.
- /oauth/authorize accepts ?suggested_name= URL param, trimmed +
120-char-capped, threaded through to the template for prefill.
Frontend (consentTmpl)
- Per §2a mockup: identity header, name input (with prefill),
data-permissions block (capability_tier radios — unchanged from
pre-TASK-1523), workspace-access radio ("All my workspaces"
default vs "Only specific workspaces"), conditional picker that
slides in for the specific mode, may_create_workspaces checkbox
(default-on), footer disclosure, and Authorize/Deny buttons.
- Picker shows a search input when the user has >10 workspaces;
per-row role label.
- Mobile layout (max-width: 480px) stacks the actions vertically
with Authorize above Deny.
- "Pick at least one workspace" helper text appears when the
user is in specific mode with zero workspaces selected.
- Empty-workspace case still renders the "create or join one
first" empty state; server validates regardless of JS.
Tests
- 2 new tests: TestConsent_ApproveWithNewShape (end-to-end with
connection_name + workspace_access + may_create_workspaces +
allowed_workspaces — asserts the values land on the
oauth_connections row and join table); TestConsent_SuggestedNamePrefill
(?suggested_name= URL param prefills the form input).
- Existing approve-flow tests updated: the introspection response
no longer carries allowed_workspaces (we stopped writing
session.Extra), so assertions now read the connection row +
join table via ListUserOAuthConnections — same guarantee, new
shape. TestOAuth_Authorize_RendersConsentWhenLoggedIn seeds a
workspace so the consent form renders its allow-list section
(the new template hides it when the user has zero memberships).
Parent: PLAN-1519.
|
||
|
xarmian
|
905baaa010 |
feat(oauth): backfill session.Extra into oauth_connections + switch read path (TASK-1522) (#583)
* feat(oauth): backfill session.Extra into oauth_connections + switch read path (TASK-1522) Phase C1 for PLAN-1519. Seeds existing OAuth grant chains into the new connection tables (Phase A) and switches /console/connected-apps to read from them, retiring the session.Extra parse on the read path. Backfill (internal/store/oauth_connections_backfill.go) - Walks oauth_access_tokens + oauth_refresh_tokens to find every distinct request_id chain (including refresh-only chains). - Picks the newest token row per chain — its session.Extra drives the seeded shape, so a chain whose user re-scoped recently reflects the latest decision. - Maps session.Extra shapes to the new tables per IDEA-1517 §2: no key → all_current=1; ["*"] → all_current=1; explicit slugs → all_current=0 + one join row per slug (added_by='user'). - Resolves slugs → workspace IDs; unresolved slugs (deleted / renamed workspace) are counted + logged at WARN, not fatal. - Idempotent on every INSERT (OR IGNORE / ON CONFLICT DO NOTHING) so re-running on every startup is a cheap no-op once stable. - Returns a BackfillOAuthConnectionsResult so the startup log reports chains_seen / connections_created / workspaces_added / unresolved_slugs — operators see fresh work and notice drift. Read-path rewrite (internal/store/connected_apps.go) - ListUserOAuthConnections projects AllowedWorkspaces from GetOAuthConnectionAccess (oauth_connection_workspaces JOIN workspaces) instead of parsing session.Extra strings. - Hydrates Name + MayCreate + AllCurrent + IncludeFuture from oauth_connections so Phase D's mutation UI has them. - Defensive fallback for chains without an oauth_connections row (any leftover the backfill missed): treats as legacy "any workspace, default-on flags" so the connection still renders. Backfill at startup keeps this branch unreachable in production. - Retires parseAllowedWorkspacesFromSession; the new extractAllowedWorkspacesFromSessionExtra helper in oauth_connections_backfill.go is the only consumer of the session.Extra shape on the store side. Model (internal/models/connected_apps.go) - Adds Name / MayCreateWorkspaces / AllCurrentWorkspaces / IncludeFutureWorkspaces. AllowedWorkspaces semantics stay stable (nil = "any"; explicit slugs = chip list) so the existing DTO + frontend continue working unchanged. Phase D exposes the new fields on the wire. Startup wiring (cmd/pad/main.go) - After srv.SetOAuthServer / SetClaimSecret, run the backfill once. Non-fatal on error (partial state is consistent and the next run completes). Quiet at the Debug level on steady-state re-runs; INFO when fresh work landed. Tests - 8 BackfillOAuthConnections cases: empty DB, pre-TASK-952 (no key), wildcard, explicit list, mixed resolvable/unresolved slugs, multi-row chain newest-row-wins, refresh-only chain, idempotent re-run (verified via post-run row count). - TestExtractAllowedWorkspacesFromSessionExtra replaces the retired parseAllowedWorkspacesFromSession test — covers all three IDEA-1517 §2 input shapes + malformed/non-array defensive cases. - TestListUserOAuthConnections_DeduplicatesChain + TestHandleListConnectedApps_DTOShapeAndAuditEnrichment updated to call BackfillOAuthConnections (the production startup hook) before asserting on AllowedWorkspaces — mirrors the real-world flow now that the read path no longer parses session.Extra inline. Parent: PLAN-1519. * fix(oauth): backfill counters reflect actual new rows per Codex review (round 1) PR #583 Codex review round 1 flagged that the backfill counters over-report on steady-state restarts: - wasFreshlyInserted compared updated_at vs created_at — true for every untouched existing row, so every restart counted every pre-existing connection as "created." - slugsAdded++ ran after AddConnectionWorkspace regardless of whether the INSERT OR IGNORE / ON CONFLICT DO NOTHING hit an existing row. Net effect: startup logs "backfill complete" with non-zero counts on every restart instead of the intended quiet "no-op" path — making real fresh work indistinguishable from steady-state. Fix: probe existence BEFORE the insert on both sides. - backfillOneChain reads GetOAuthConnection first; only sets created=true and runs insertOAuthConnectionIfAbsent on a miss. - Per-slug: IsConnectionWorkspaceAllowed pre-check; skip + don't increment when the row already exists. Two cheap PK / indexed lookups per chain. Pre-Phase-C deployments have small chain counts so the added cost is well below the scan already running. Removed the now-unused wasFreshlyInserted helper. Added an assertion in TestBackfillOAuthConnections_Idempotent that both ConnectionsCreated and WorkspacesAdded report 0 on the second run — the regression guard for this exact finding. Parent: PLAN-1519. * fix(oauth): backfill skips slug re-seed on existing rows per Codex review (round 2) PR #583 round 2 caught that the round-1 fix protected the parent oauth_connections row from re-seed but left the join table mutable from stale session.Extra: When a user removes a workspace from their connection's allow-list via Phase D's mutation UI (RemoveConnectionWorkspace), the next server restart would re-run the backfill, find the parent row intact, and re-INSERT the removed slug from the original session.Extra. The user's removal would silently revert every restart. Fix: backfill is a one-shot seed. Once the parent row exists, the new tables are authoritative — legacy session.Extra is frozen reference data, not a reconciliation source. The slug loop only runs when we just inserted a fresh parent row. Added TestBackfillOAuthConnections_DoesNotResurrectRemovedWorkspace as the regression guard: seeds two slugs, removes one, runs backfill again, asserts the removed slug stays gone and the kept slug is untouched. Parent: PLAN-1519. * fix(oauth): atomic per-chain backfill transaction per Codex review (round 3) PR #583 round 3 caught that round 2's "only seed slugs on fresh parent" gate introduced a permanent-partial-state risk: if the process crashes (or AddConnectionWorkspace errors) between inserting the parent row and finishing the slug loop, the next backfill sees created=false, short-circuits the slug seeding, and leaves the connection permanently scoped to a partial allow-list. Fix: per-chain transaction. Parent insert + every slug insert land in one BEGIN/COMMIT pair; any mid-loop failure rolls everything back. The next backfill then sees the chain as un-seeded and retries from scratch — preserving both round 2's "no-resurrection of user-removed slugs" (existence probe inside the tx) and round 3's "no permanent partial seed" (atomic commit). Scope: per-chain (small tx), not whole-backfill. The original no-transaction rationale was about lock-hold duration across thousands of chains; that doesn't apply at chain granularity (one parent + a handful of join rows = sub-millisecond hold). Removed the now-unused insertOAuthConnectionIfAbsent helper; the INSERTs live inline within the transaction. Added TestBackfillOAuthConnections_AtomicOnMidLoopFailure as the regression guard: forces a mid-loop INSERT failure via a duplicate slug in session.Extra (which violates the join table's PK on the second insert), asserts the parent row rolled back, then runs a clean retry and verifies full seed completion. Parent: PLAN-1519. * fix(oauth): surface store errors from backfill + list path per Codex review (round 4) PR #583 round 4 caught two silent-fallthrough paths that could leak partial/incorrect state instead of failing loudly: 1. Backfill slug loop: GetWorkspaceBySlug errors were treated the same as "workspace not found" — both incremented slugsMissed and continued. A real I/O error mid-loop would commit a partial allow-list, and the next backfill's parent-exists short-circuit would make that partial scope permanent. Fix: distinguish (nil, nil) "not found" from (nil, err) "real failure" — return the error so the per-chain transaction rolls back and the next run retries cleanly. 2. ListUserOAuthConnections hydration: GetOAuthConnectionAccess and GetOAuthConnection errors collapsed into the "no oauth_connections row" defensive-fallback branch, returning the legacy "any workspace, default-on flags" shape. On a real store failure that silently broadens a user's scope — e.g. a connection the user explicitly removed a slug from would render as "Any workspace" until the store recovered. Fix: surface store errors from both calls; the defensive fallback path is now exclusively for HasConnection=false, not for error masking. Both findings tighten the failure mode from "silently emit broadened/partial state" to "surface the error so retries happen against accurate data." Existing tests cover the happy paths; the failure paths are exercised by I/O errors against the same store interfaces (no new test added — the change is "return err instead of swallow it" and the assertion of NOT swallowing is the diff itself). Parent: PLAN-1519. |
||
|
xarmian
|
aec67e202e |
feat(mcp): workspace.create + workspace.claim actions + claim-code mechanics (TASK-1521) (#582)
Phase B for PLAN-1519. Adds two MCP actions so agents can bring a workspace into an OAuth connection without re-auth — the agent-first onboarding story IDEA-1517 §1 set out to fix. pad_workspace.action: create - New action on the shared catalog (stdio + cloud both pick it up). - POSTs to /api/v1/workspaces; handler auto-adds the new workspace to the calling OAuth connection's allow-list (added_by='agent-create') when the grant carries may_create_workspaces=true. Phase A wired the oauth_connection_workspaces table this writes to. PAT / CLI-session callers fall through silently (no request_id → no side effect). - Backed by a new non-interactive `pad workspace create <name>` Cobra command for the stdio MCP shell-out path. `pad workspace init` remains the guided human flow. pad_workspace.action: claim - New POST /api/v1/oauth/claim endpoint redeems a 6-digit stateless HMAC code minted from (user_id, workspace_id, 5-min time bucket) with a sliding 5–10 minute lifetime. Constant-time compare. Code format derived per IDEA-1517 §4. - Verifies workspace membership before code (privilege-escalation guard); uniform 404 envelope so the endpoint can't be used to probe existence vs. membership. - Side effect inserts a row in oauth_connection_workspaces with added_by='claim'. Idempotent — re-claiming returns 200 with already_added=true. - 412 connection_not_persisted when the OAuth grant predates Phase C (no oauth_connections row); 412 claim_disabled when the deployment hasn't wired SetClaimSecret. - `pad workspace claim <code> --workspace <slug>` Cobra command backs the stdio MCP shell-out. MCP server instructions - Appended IDEA-1517 §5 paragraph teaching agents the claim flow as a peer top-level section. Same string lands universally on every MCP handshake response (stdio + cloud both read instructions.md). Tests - 10 claim-code unit tests: determinism, zero-pad, length-prefix collision guard, current/previous bucket accept, aged-out reject, wrong-everything rejects, short-secret fails closed. - 7 handler tests: 412 when secret disabled, 400/404/401 vocabulary, PAT caller note path, 412 connection_not_persisted, idempotent insert. - 5 MCP-catalog tests: actions registered, schema params advertised, description mentions both actions, route mappers produce correct HTTP shape, routeTable carries the entries. - Bumped the existing read-only catalog bijection + fixture-input fixtures so the new actions resolve cleanly. Parent: PLAN-1519. |
||
|
xarmian
|
3f6bcc0ff2 |
feat(oauth): per-connection state tables + dual-read introspection gate (TASK-1520) (#581)
* feat(oauth): per-connection state tables + dual-read introspection gate (TASK-1520) Phase A foundation for PLAN-1519 / IDEA-1517's per-OAuth-connection state overhaul. Promotes the consent-time workspace allow-list out of session.Extra (per-token, re-minted on every refresh-token rotation) into dedicated tables keyed by request_id (the grant chain identifier preserved across rotations). Schema (SQLite migration 059 + Postgres migration 038): - oauth_connections: one row per grant chain with name + three scope flags (may_create_workspaces, all_current_workspaces, include_future_workspaces). - oauth_connection_workspaces: mutable allow-list join table; PK on (request_id, workspace_id); FK ON DELETE CASCADE; added_by audit column. Store (internal/store/oauth_connections.go): Create/Get/Rename/SetScopeFlags/ Add+Remove+IsAllowed/Delete CRUD. GetOAuthConnectionAccess is the hot-path projection — one PK lookup + one indexed join when the wildcard flag is off, nothing else when it's on. Dual-read gate (internal/server/middleware_mcp_auth.go): OR-merges the legacy session.Extra allow-list with the new-table projection. A workspace is allowed iff either source allows it; either source's wildcard makes the gate unrestricted. New tables stay empty until Phase C writes the consent screen, so the dual-read is a no-op until then — and existing OAuth grants keep working unchanged through the Extra path. I/O errors on the new path fall back to the Extra path so a transient outage of the new tables can't regress existing connections. Tests: - 10 store tests cover CRUD, FK cascade, wildcard short-circuit, sorted slug projection, idempotent add/remove, ErrOAuthConnectionNotFound on missing rows. - 11-case table-driven test on mergeAllowedWorkspaces directly verifies PLAN-1519's acceptance criterion: "token with allow-list in session.Extra still passes; token with empty session.Extra but row in oauth_connection_workspaces also passes." Plus wildcard precedence, union dedup, and fail-closed-on-empty-scope. - BenchmarkMergeAllowedWorkspaces measures policy-function overhead on the hot path (the store-side lookup is the other half of the dual-read cost). Parent: PLAN-1519. * fix(oauth): fail-closed on connection lookup error per Codex review (round 1) PR #581 Codex review round 1 caught two issues: 1. middleware_mcp_auth.go: GetOAuthConnectionAccess errors fell through to "no connection" + nil allow-list = unrestricted. Post-Phase-C (when the new tables are authoritative and session.Extra is empty), a DB read error on a scoped token would silently grant access to every workspace the user belongs to. Now fails closed with a 401 matching the IntrospectToken storage-error policy, increments MCPAuthzDenialsTotal{connection_lookup_error} for ops visibility. 2. oauth_connections.go: CreateOAuthConnection's docstring claimed "scope flags default ON if not supplied," but the method writes the three Go bools verbatim — and Go zero-values for bool are false, not true. The schema-level DEFAULT TRUE is unreachable through this path. Docstring updated to clarify that defaults live at the form-rendering layer; the store is a faithful pass-through. Parent: PLAN-1519. * test(oauth): add store-side bench for GetOAuthConnectionAccess per Codex review (round 2) PR #581 Codex review round 2 flagged the docstring references to bench_oauth_connections_test.go pointing at a file that didn't exist — only the in-memory mergeAllowedWorkspaces bench was wired. Add the store-side bench so the documented file is real and PLAN-1519 Phase A's "Hot-path benchmark: dual-read overhead measured and documented" acceptance bullet is satisfied end-to-end. Three shapes covered: Wildcard (PK lookup, join short-circuited), Explicit (PK + indexed scan + small workspaces join), and NoRow (the dominant Phase-A path until Phase C wires the write path). Local numbers (Ryzen 3 5300U, SQLite WAL): 12µs/12µs/37µs respectively — all comfortably sub-millisecond. Parent: PLAN-1519. |
||
|
xarmian
|
2a1a00e385 |
test,docs: blank+onboard+needs_onboarding integration test + CLAUDE.md update (TASK-1507,1508) (#580)
PLAN-1496's final consolidation pair, shipping together because both are small cleanup passes that close the plan out. TASK-1507 (tests): Most of the test coverage required by this task was already added incrementally in the prior PRs that built each surface: - Blank template (4 focused tests, PR #575): TestSeedFromBlankTemplate, TestBlankTemplateShape, TestBlankTemplateExcludesSoftwareCollections, TestBlankTemplateUsesMinimalVocabularies, TestBlankTemplateAppearsInPicker - Onboard auto-seed (PR #576): TestSeedFromTemplateAlwaysIncludesOnboardPlaybook (walks all six templates), TestSeedWithEmptyTemplateNameSkipsOnboard (locks the empty-templateName escape-hatch invariant), TestOnboardPlaybook_Contract (invocation_slug, trigger, mode-enum, ADAPT-DON'T-CURATE rule in the body) - needs_onboarding (PR #578): TestBootstrapNeedsOnboardingFlag (lifecycle: fresh → user item → flag flips), TestBootstrapNeedsOnboardingIgnoresTemplateSeeds (template seeds don't count) - Retired-pattern updates (PR #577): TestSoftwareTemplatesShipNoSeedItems inverse invariant; TestDashboardOnboardingSeed_NilForAllTemplates collapsed from three IDEA-1/BACK-1/FEAT-1 tests. This commit adds ONE integration smoke test that ties the three subsystems together at the bootstrap layer: - TestBootstrapBlankWorkspaceOnboardReady creates a blank-template workspace, fetches bootstrap, and asserts: needs_onboarding=true (nudge fires) AND the onboard playbook is in bootstrap.playbooks AND its status is "active" AND its trigger is "manual" (in the blank template's seeded vocabulary). If any one of the three pieces regresses silently, the integration breaks and this test catches it before /pad onboard stops dispatching on day one. TASK-1508 (docs): - CLAUDE.md "Data Model / Templates" section: added Blank under a new "Custom" category bullet pointing at the new Onboarding section; called out the PLAN-1496 retirement of the IDEA-1 / BACK-1 / FEAT-1 first-person seed pattern; updated design history reference to include PLAN-1496. - CLAUDE.md "API" section: added the /api/v1/workspaces/{ws}/agent/bootstrap endpoint with a note about the needs_onboarding flag (was previously documented only inline in the Playbooks section). - CLAUDE.md: new top-level "Onboarding" section between Playbooks and Testing. Covers: auto-seeded everywhere, surface-agnostic body, adaptation posture (library entries are starting points), the three TASK-1510/1511/1512 mutation primitives, the needs_onboarding bootstrap flag + skill nudge, the four retired surfaces (pad onboard cobra, OnboardingPrimaryRef, *OnboardingItems generators, standalone skill workflow section), and a code map. Verification: - go test ./...: clean (full suite passes including the new integration test) - make lint: 0 issues Parent: PLAN-1496. |
||
|
xarmian
|
96a32aa39d |
feat(bootstrap,skill): add needs_onboarding flag + retire legacy Onboarding workflow (TASK-1504,1505) (#578)
PLAN-1496's bootstrap-signal + skill-cleanup pair, shipped together
because TASK-1505's nudge rendering depends on TASK-1504's bootstrap
field.
TASK-1504 (bootstrap: needs_onboarding):
- internal/store/items.go: new WorkspaceHasUserCreatedItems(workspaceID)
store method. Backed by SELECT EXISTS with the predicate
`source != 'template'` — defined as the inverse of template seeding
rather than enumerating user-side source values, so new attribution
surfaces (mcp, api, future) count automatically.
- internal/server/handlers_bootstrap.go: AgentBootstrap struct gets
the NeedsOnboarding bool field (always emitted — not omitempty,
since the agent reads it on every /pad invocation). BuildAgentBootstrap
computes it via the new store method. On query error the flag falls
back to false (safe default: don't nag).
- Visibility filtering deliberately omitted — needs_onboarding is a
workspace-level state signal, not a per-user view. Two members
reading bootstrap concurrently should see the same answer.
- Two focused tests:
- TestBootstrapNeedsOnboardingFlag walks the lifecycle (fresh
workspace → create user item → flag flips).
- TestBootstrapNeedsOnboardingIgnoresTemplateSeeds locks the
template-seeds-don't-count invariant on the startup template,
which ships seeded conventions, playbooks, and the onboard
playbook itself.
TASK-1505 (skill update):
- skills/pad/SKILL.md:
- Context Loading section: new bullet documenting needs_onboarding
with the exact nudge wording the agent should render when true,
plus the "don't nag past first user item" + "respect prior
decline" rules.
- Onboarding workflow section: deleted (~30 lines). Replaced with
a one-paragraph pointer at the /pad onboard playbook. The skill
is the dispatcher; the playbook body is the script.
- Routing entry under "set up my workspace": simplified from the
bloated PR #577 round-3/4 text into a clean two-bullet form
(canonical phrasing + legacy IDEA-1 phrasing both → /pad onboard).
- internal/mcp/prompts_data.go: the pad_onboard MCP prompt body
was duplicating the same step-by-step script the SKILL.md section
carried. Replaced with the same dispatch-to-playbook pointer.
internal/mcp/prompts_test.go: TestPromptsLockstep_CoreCommands
fragments updated to assert the new dispatch fragments
(`pad playbook list`, `pad playbook show onboard`).
Parent: PLAN-1496.
|
||
|
xarmian
|
0930743304 |
feat: retire IDEA-1 seed pattern + 'pad onboard' cobra + surface blank in init (TASK-1501/1502/1503) (#577)
* feat: retire IDEA-1 seed pattern + 'pad onboard' cobra + surface blank in init (TASK-1501,1502,1503)
PLAN-1496's legacy-onboarding teardown:
TASK-1501 (remove seed items + update banner):
- internal/collections/templates_onboarding.go (and the _product/_scrum
siblings) deleted — these generated the IDEA-1/PLAN-2/TASK-3/DOC-4 +
BACK-1/SPRINT-2/BUG-3/DOC-4 + FEAT-1/FB-2/ROAD-3/DOC-4 first-person
seeds. The /pad onboard playbook (TASK-1499 / TASK-1500) is the
replacement.
- startup/scrum/product templates: SeedItems lines removed.
- post-init banner in printOnboardingHints: now points at "/pad onboard"
in one line, then web UI link, then dashboard hint. The "use pad to
get IDEA-1 / BACK-1 / FEAT-1" branch is gone.
TASK-1502 (retire cobra + OnboardingPrimaryRef plumbing):
- OnboardingPrimaryRef struct field on WorkspaceTemplate removed. The
dashboard's banner auto-discovers seeds via item_number=1 +
source="template" + created_by="system", so the field was redundant
even before retirement.
- onboardingPrimaryRef() helper in cmd/pad/main.go removed.
- 'pad onboard' Cobra subcommand removed (~160 lines). It scanned the
project directory for build/test/CI markers and seeded library
conventions — useful behavior but CLI-only, unreachable from
MCP-only agents. The /pad onboard PLAYBOOK now covers it.
- internal/cli/detect.go and workspace_context_detect.go stay; still
used by the web-side workspace-context save path.
TASK-1503 (Blank in interactive picker):
- The picker already surfaces Blank because templates_picker.go iterates
GroupTemplatesByCategory, and the IDEA-1479 Blank template entry lives
in CategoryCustom. Verified the output renders correctly with the
TASK-1498 description + icon update.
- 'pad workspace init --help' Long now mentions Blank explicitly +
points users at /pad onboard. Helps discoverability without restructuring
the picker.
Test changes (delete or rewrite tests that exercised the retired pattern):
- internal/collections/templates_test.go: six tests deleted (StartupOnboardingItemsOrderAndShape,
ScrumOnboardingItemsOrderAndShape, ProductOnboardingItemsOrderAndShape,
Startup/ScrumProduct/TemplatesDeclareOnboardingPrimaryRef). New
TestSoftwareTemplatesShipNoSeedItems replaces them with the inverse
invariant: software templates ship zero seed items.
- internal/server/handlers_dashboard_test.go: three IDEA-1/BACK-1/FEAT-1
expectation tests collapsed into TestDashboardOnboardingSeed_NilForAllTemplates,
which asserts the auto-discovery finds no seed because seeds no longer
ship. (Hiring + EmptyWorkspace tests untouched — they already expect
nil for unrelated reasons.)
- internal/store/items_test.go: TestSeedCollectionsFromTemplate{Startup,Scrum,Product}RefSequence
and TestOnboardingFlow_FullWalkthrough_{Startup,Scrum,Product} deleted;
these locked the IDEA-1 ref-sequence + walkthrough behavior. Unused
helpers (findItemByTitle, extractStatus, safeFields, setItemStatus,
countItemsInCollection) deleted alongside them.
- internal/mcp/resources_test.go: TestReadItem_PreservesIDEAOneOnboardingBodyVerbatim
→ TestReadItem_PreservesBodyVerbatim. Property is the same (resource
pipeline doesn't mangle markdown), but the fixture is now synthetic
markdown instead of the IDEA-1 seed.
Note: handlers_dashboard.go still has the auto-discovery code path
(onboardingPrimaryCollectionSlugs map + the loop that probes for
item_number=1 + source="template"). It's now dead code — no item
will ever match the criteria after this PR. Left in place for a
follow-up cleanup pass to keep this PR focused.
Parent: PLAN-1496.
* docs: replace 'pad workspace onboard' references with /pad onboard (Codex round 1)
P2 finding on PR #577: README + CLAUDE.md still advertise the
'pad workspace onboard' subcommand in four places (README §Onboard
agents to a new codebase, README §3 Teach your agents the rules,
README CLI Reference, CLAUDE.md CLI). After this branch lands, those
instructions return "unknown command."
Replaced each with guidance pointing at /pad onboard (the playbook,
auto-seeded into every workspace). The library-list commands still
work and stay where they are.
Parent: PLAN-1496.
* docs: replace 'use pad to get IDEA-1' guidance with /pad onboard (Codex round 2)
P1 finding on PR #577: README.md:33-39 and CLAUDE.md:111-117 still
told users to 'use pad to get IDEA-1' after the post-init banner.
Since this branch deletes templates_onboarding.go and stops seeding
IDEA-1/PLAN-2/TASK-3/DOC-4, the quickstart instructions in both
top-level docs pointed at items that no longer exist.
Replaced each with /pad onboard guidance (the playbook is auto-seeded
into every new workspace by TASK-1500). CLAUDE.md's CLI reference
gets a one-line historical note explaining the pre-PLAN-1496 IDEA-1
pattern so readers reviewing older code/blame have context.
Parent: PLAN-1496.
* docs(skill): retire 'use pad to get IDEA-1' guidance in agent skill (Codex round 3)
P1 finding on PR #577: skills/pad/SKILL.md:175 still taught agents
that '"use pad to get IDEA-1"' should dispatch to 'pad item show IDEA-1'.
This branch deletes the seed items, so any agent following the
shipped skill in a fresh workspace would try to fetch a missing ref
instead of running /pad onboard.
Updated the routing entry to dispatch the legacy phrasing (kept as a
recognized intent so older docs/conversations still work) to the
/pad onboard playbook. Explicit "do NOT try to fetch IDEA-1
directly" to short-circuit the previously-trained behavior.
A broader skill cleanup — removing the standalone Onboarding
workflow section and adding the bootstrap nudge rendering — is
TASK-1505's scope. This PR's update is the minimal change needed to
unbreak the agent-facing routing.
Parent: PLAN-1496.
* docs(skill): add library-activation caveat to onboard routing entry (round 4)
P2 finding on PR #577: the routing entry said /pad onboard is
'always invokable because every workspace auto-seeds it.' True for
newly-created workspaces, but pre-existing workspaces (created before
PLAN-1496 lands) won't have it. Auto-upgrade is intentionally not
wired into SeedCollectionsFromTemplate for empty-template-name paths.
Mirrored the same activation-fallback caveat /pad plan and
/pad decompose carry: 'activate via library if the bootstrap's
playbooks array lacks invocation_slug=onboard, status=active.'
Parent: PLAN-1496.
|
||
|
xarmian
|
e59d3904c9 |
feat(server): refuse to mark item terminal while it has open children (IDEA-1494) (#571)
* feat(server): refuse to mark item terminal while it has open children (IDEA-1494)
Server-side guard inside handleUpdateItem that rejects a non-terminal →
terminal done-field transition when the item still has at least one
non-terminal child. Returns HTTP 409 with code=open_children plus a
structured details payload listing each blocking child's
{ref, title, status, collection_slug} so MCP-driven agents can
self-recover (ship the listed children, then retry) and the CLI can
render the same list verbatim.
Escape hatch: `--force` on `pad item update` / `pad item bulk-update`
and `force: true` on the MCP pad_item.action: update / bulk-update
inputs both forward into the same ItemUpdate.Force transport field
the handler consumes before any store mutation.
Trigger conditions are tight: the PATCH must change the done-field key
(resolved via TerminalValuesForDoneField against the parent's schema +
settings) AND the new value must be terminal AND the current value
must NOT already be terminal. Terminal → terminal and no-op terminal
transitions bypass the guard; only entering the terminal set is gated.
Per-child evaluation uses the child's own collection schema so
hierarchical workspaces with custom typed collections work without
extra plumbing.
Tests cover: rejection with one open child (with mutation-safety
assertion on the parent), no children, all-terminal children, --force
override, no-op terminal → terminal, terminal → terminal,
non-terminal → non-terminal, custom collection terminal_options
honored, and a parent task (not a plan) — IDEA-1494 optional extra #3.
MCP coverage asserts --force round-trips through both ExecDispatcher
and HTTPHandlerDispatcher and is omitted when force=false.
* fix(server): open-children guard round 2 — visibility, MCP pass-through, TOCTOU (IDEA-1494)
Three Codex round-1 issues, each fixed with the recommended shape:
P1 — visibility leak. The 409 response previously listed every blocking
child by ref/title/status, including children in collections the caller
couldn't see. The INVARIANT still evaluates against ALL children (it's a
data-integrity gate — a restricted user must not be able to close a
parent whose blockers they can't see), but the response payload now
filters to caller-visible children only. Hidden blockers surface as a
new `details.hidden_blocker_count` field plus an alternate human message
when every blocker is hidden ("blocked by N open children you don't
have access to"). Mirrors the visibility helpers (`visibleCollectionIDs`
+ `isItemVisibleToGuest`) used by the per-parent progress endpoint so
the two paths can't drift.
P2 — MCP code/details pass-through. The HTTP classifier was collapsing
409 into the generic `conflict` code and dropping `details`; the stdio
classifier was matching the human "cannot " message against the
validation regex and surfacing `validation_failed`. Both now surface
`open_children` with the structured details intact:
- HTTP: classifyHTTPStatusKind's 409 branch extracts the upstream
code; any non-empty, non-"conflict" code is passed through with
its `details` RawMessage. Generalizes beyond open_children — any
future structured 409 from a handler gets the same treatment.
- Stdio: the CLI writes a `pad-error: {json}\n` marker line on
stderr before the human-readable block (single source of truth for
both views), and classifyExecError detects the marker and lifts
the envelope verbatim. Marker is duplicated as a const between
internal/cli and internal/mcp to avoid pulling the cli package
into the classifier just for one string.
A new ErrOpenChildren error code constant + `Details json.RawMessage`
field on ErrorPayload back the wire shape.
P2 — TOCTOU. The guard previously ran in the handler before the store
transaction began; a concurrent child insert / child status flip could
slip between the children-list read and the parent's UPDATE. Fix:
- New `Store.UpdateItemWithPreCheck(id, input, precheck)` runs the
caller's invariant check inside the same tx, after acquiring the
workspace seq lock AND a new parent-children advisory lock keyed
on the parent ID. UpdateItem is now a thin wrapper passing nil.
- Every UpdateItem unconditionally acquires the parent-children
advisory lock for its own parent (if any) AND for itself-as-parent,
in a fixed order (parent first) so two updaters touching the same
parent always grab that key before the more-specific one — no
AB/BA deadlock.
- New `GetChildItemsTx` reads via the caller's tx; on Postgres the
advisory lock provides the snapshot guarantee (DISTINCT precludes
`FOR UPDATE`), on SQLite the global BEGIN IMMEDIATE write lock
serializes all writers.
- Handler now passes a precheck closure into UpdateItemWithPreCheck
at all three call sites (collab-snapshot path, applier-direct-write
path, main path). The guard's openChildrenGuardError sentinel is
unwrapped after each call so the 409 surfaces cleanly.
Tests:
- TestOpenChildrenGuard_VisibilitySanitizesPayload — restricted
editor sees parent + visible child, hidden child contributes to
hidden_blocker_count, no leak of ref/title/slug.
- TestOpenChildrenGuard_AllBlockersHiddenSurfaceGenericMessage —
open_children=[], hidden_blocker_count>0, message mentions "you
don't have access to."
- TestOpenChildrenGuard_TOCTOURace — 8 iterations of a child-flip
racing a parent-terminal update; asserts the forbidden outcome
(parent=completed AND child=open) never occurs.
- TestClassifyHTTPStatus_OpenChildrenPreservesCodeAndDetails +
inverse generic-409 test.
- TestClassifyExecError_OpenChildrenMarkerLiftsStructuredPayload +
no-marker-falls-through inverse.
* fix(server): open-children guard round 3 — 7 Codex findings closed (IDEA-1494)
P1 — visibility fail-closed. The handler was swallowing
visibleCollectionIDs errors, leaving visIDs==nil which the guard
treats as unrestricted, leaking hidden-child metadata. Now surfaces
the error as 500 BEFORE installing the precheck. Test:
TestOpenChildrenGuard_VisibilityLookupErrorFailsClosed closes the
store DB and asserts no 409+children leak.
P1 — link mutations acquire the advisory lock. SetParentLink,
ClearParentLink, CreateItemLink (when link_type ∈ childLinkTypes via
new isChildLinkType helper), DeleteItemLink (same condition), and
RestoreItem now take `pad:parent-children:<id>` in canonical sorted
order via new AcquireParentChildrenLocks helper. SetParentLink locks
BOTH old and new parents (re-parenting case). Race test
TestOpenChildrenGuard_LinkMutationRace asserts the forbidden
"link-committed-before-parent-flip AND parent flip succeeded" never
occurs by comparing link.created_at to parent.updated_at. Documented
semantics: status-wins + link-after-commit is legal under the
invariant "no open children EXIST AT THE MOMENT of transition" —
the post-condition variant ("no open child may EVER attach to a
terminal parent") is intentionally deferred.
P1 — MoveItem bypass closed. New MoveItemWithPreCheck mirrors
UpdateItemWithPreCheck — acquires workspace seq lock + parent-children
locks, re-reads in tx, runs caller precheck. handleMoveItem builds
the same guard closure using the DESTINATION schema for done-field
resolution (conservative — honors the schema the item moves INTO).
CLI gains `pad item move --force`, client gains MoveItemWithForce
that appends `?force=true` to the move endpoint. MCP catalog +
mapItemMove forward `force` through the route mapper. Tests:
TestOpenChildrenGuard_MoveItem_RejectsTerminalWithOpenChildren and
…_ForceOverrides.
P2 — pre-tx field-read TOCTOU. UpdateItemWithPreCheck and
MoveItemWithPreCheck now re-read the item via new getItemTx INSIDE
the tx (after locks) and pass that fresh snapshot to the precheck
closure; the precheck classifies the transition against the in-tx
view, not the handler-side pre-tx capture. Handler precheck closure
swaps `currentFieldsJS` from the in-tx snapshot. Test:
TestOpenChildrenGuard_PrecheckReadsInTxSnapshot stages a between-load
status mutation and asserts the precheck observes the post-mutation
fields.
P2 — bulk-update carries structured errors. cmd/pad/main.go's
updateFailure struct extended with Code + Details
(json.RawMessage). When client.UpdateItem returns *cli.APIError, the
row preserves the structured envelope. Human-text output also
renders the open-children list inline. Chose JSON-envelope route
over per-row stderr markers because bulk-update already produces a
structured envelope and ExecDispatcher returns stdout verbatim on
exit-0 — no classifier change needed. Test:
TestBulkUpdateStructuredFailuresCarryOpenChildrenDetails confirms
the wire shape the CLI lifts.
P3 — marker hardening. Marker bumped to versioned form
`pad-structured-error/v1:` (was `pad-error:`). cli.StructuredErrorMarker
+ mcp.structuredErrorMarker kept in lockstep with cross-references.
mcp.allowedStructuredErrorCodes whitelists known codes (currently
just open_children); unknown codes fall back to regex classification.
Marker must start the line after whitespace trim (embedded markers
ignored). Last-marker-wins to defeat pre-emption attacks. Tests:
TestClassifyExecError_{UnknownStructuredCode,OldMarkerVersion,
MarkerEmbeddedMidLine,LastMarker}.
P3 — soft-deleted collection schemas honored. New GetCollectionAnyState
mirrors childrenDoneFiltersForParent's inclusion rule; guard uses it
so a child still attached to a soft-deleted collection is evaluated
against ITS schema (custom terminal_options) instead of the default-
status fallback (which would mis-classify and false-block). Test:
TestOpenChildrenGuard_SoftDeletedCollectionSchemaHonored seeds a
custom collection, soft-deletes it while a child remains, and
asserts the terminal status is correctly recognized.
Comprehensive store-mutation audit results recorded in the PR
description (every method touching items.fields / items.collection_id
or item_links).
* fix(server): open-children guard round 4 — multi-parent locks, enum parity, PATCH atomicity (IDEA-1494)
Four Codex round-3 (blast-radius lens) findings, each fixed with the
recommended shape.
P1 — multi-parent lock set. acquireParentChildrenLocksForUpdate and
RestoreItem previously used `LIMIT 1` against item_links, so a child
with BOTH a `parent` link to P1 AND an `implements` link to P2 only
locked one of them. The other parent's open-children precheck could
race against the child's status flip and miss it.
Fix: new listParentChildLockKeys helper runs the same query
GetChildItems' inclusion rule uses (childLinkTypes), returns ALL
distinct parent target_ids, and feeds them into the canonical
multi-lock helper. Both UpdateItemWithPreCheck and RestoreItem now
acquire locks on {self} ∪ {all-parents-via-childLinkTypes}. Test:
TestOpenChildrenGuard_MultiParentChildLocksAll races a child status
flip against terminal-updates on both parents simultaneously.
P2 — lock-order asymmetry. The pre-fix codebase had multiple lock-
acquisition shapes: parent-then-self in acquireParentChildrenLocksForUpdate,
single-key in RestoreItem / CreateItemLink / DeleteItemLink /
ClearParentLink, and a sorted multi-key in SetParentLink. Two
concurrent callers using different ad-hoc orderings could AB/BA
deadlock.
Fix: removed the per-call-site AcquireParentChildrenLock helper
entirely. Every site now goes through AcquireParentChildrenLocks
(the canonical sorted multi-lock helper) — including ones that need
only one ID (the variadic call still sorts a one-element slice).
The helper's doc comment explicitly states the contract: "Ad-hoc
single-key acquisition outside this helper is FORBIDDEN — two call
sites taking distinct keys in different orders WILL deadlock."
Test: TestOpenChildrenGuard_NoDeadlockUnderReverseOrderConcurrency
runs reverse-order re-parents with a 5-second timeout; assertion
fails on hang.
P2 — HTTP/stdio code-surface parity. Round 2's HTTP pass-through
("any non-conflict upstream code") silently widened the ErrorCode
enum beyond stdio's allow-list (`open_children` only). Agents saw
different code surfaces depending on which dispatcher delivered
the response.
Fix: HTTP 409 branch in classifyHTTPStatusKind now consults the
same allowedStructuredErrorCodes whitelist stdio does. Codes
outside the set collapse to ErrConflict (no details), matching
what stdio does for an unknown-code structured marker. Doc on
allowedStructuredErrorCodes updated to make the dual-consumer
contract explicit: "Adding a new structured code is a TWO-WAY
change." Tests:
TestClassifyHTTPStatus_UnknownConflictCodeFallsBackToErrConflict
and TestStructuredErrorCodeParityAcrossTransports.
P3 — PATCH atomicity. A combined PATCH with `parent` + `status=terminal`
on an item with open children used to commit the parent-link change
INLINE (before the guard ran) and then reject the field write.
Caller saw 409 but the parent had already moved.
Fix: parent-link mutation is now DEFERRED — captured into outer-
scope vars during fields validation, executed AFTER
UpdateItemWithPreCheck succeeds. A guard rejection returns before
the link write block, so on rejection the link is untouched.
Documented choice: "reorder, don't tx-wrap" — wrapping SetParentLink
into the same store tx would require threading a *sql.Tx through
the SetParentLink API (which is also called from the
handler_item_links path); reordering is the smaller surgery and
gives the correct outcome on the failure direction. A residual
window remains in the OTHER direction (field write commits, link
write fails) — not made worse by the reorder, and called out
inline for a future tx-wrap pass.
Test: TestOpenChildrenGuard_PatchAtomicRejectionPreservesParentLink
sets up target → oldParent → openChild, sends PATCH {parent=newParent,
status=completed}, asserts 409 AND target.parent_link still points
at oldParent.
* fix(server): open-children guard — emit details.open_children as [] not null on hidden-only rejection (IDEA-1494)
|
||
|
xarmian
|
8094883869 |
feat(links): cross-workspace wiki-link resolution (IDEA-1492) (#568)
* feat(links): cross-workspace wiki-link resolution (IDEA-1492)
Adds [[workspace::REF]] and [[workspace::REF|Display]] wiki-link syntax
that resolves cross-workspace, plus a Go route GET
/{username}/{workspace}/ref/{REF} that 302-redirects to the canonical
item URL. 404 leaks no info about workspace existence — malformed refs
short-circuit before the workspace lookup, and access-denied returns 404
not 403.
Frontend (web/src/lib/utils/markdown.ts):
- renderMarkdown recognizes the workspace-prefix form and emits
cross-workspace anchors (class doc-link cross-workspace) pointing at
the resolver route. Same-workspace prefix is stripped and behaves
identically to the legacy [[REF]] form.
- wikiLinksToMarkdown emits the resolver URL for cross-workspace storage
and same-workspace items resolve through the in-memory list.
- markdownToWikiLinks rolls /<user?>/<ws>/ref/<REF> URLs back to
[[workspace::REF]] (or |Display when display text differs from the
ref). Legacy same-workspace round-trip is preserved.
Backend (internal/server/handlers_ref_resolver.go):
- Validates the ref shape before the DB hit (no oracle).
- Reuses resolveWorkspace + GetItemByRef for ACL + lookup.
- refResolverItemVisible mirrors requireItemVisible without depending
on RequireWorkspaceAccess middleware (this route is reachable
outside the workspace-scoped route group).
- Redirect target matches itemUrlId() so the post-redirect URL is
indistinguishable from a direct in-app navigation.
Tests: 302 success, 404 unknown-workspace, 404 unknown-ref, 404 on a
matrix of malformed refs (including url-encoded traversal). The
no-access matrix is partially covered — the existing test surface
doesn't compose a multi-user ACL fixture, so production-grade
"member of A probes B" is gated by the real auth middleware stack and
documented in TestRefResolver_NoAccess_DocumentsPreSetupBypass.
* fix(links): codex round-1 fixes for cross-workspace resolver (IDEA-1492)
P1.1 — Reserve "ref" as a collection slug. A collection slug of "ref"
would shadow every item URL under the resolver's /{u}/{ws}/ref/...
route. Added to reservedCollectionSlugs in internal/store/collections.go
so it auto-suffixes to "ref-collection", matching the existing
treatment of settings/activity/roles/etc.
P1.2 — Extract checkItemVisible as a context-free helper. The previous
refResolverItemVisible silently diverged from requireItemVisible by
ignoring direct collection grants and member_collection_access for
restricted members — a member with "specific" access on collection A
plus a direct grant on collection B would 404 through the resolver
even though they could see the item via the API. checkItemVisible now
replays the same rules requireItemVisible inlined; requireItemVisible
is now a thin wrapper, and the resolver derives its workspace role via
resolverWorkspaceRole and delegates to checkItemVisible. Drift between
the two paths is structurally impossible.
P2.1 — Cross-workspace round-trip preserves explicit display overrides.
The strip condition was `displayText === ref`, which would drop the
override on [[other::TASK-1|TASK-1]] — then re-rendering would emit
the default `other::TASK-1` and silently change visible link text.
Fixed: only strip when displayText matches the actual render default
`${ws}::${ref}`.
P2.2 — Two-segment route /{workspace}/ref/{REF}. TimelineCommentCard
and CommentThread call renderMarkdown without a username, so links in
timeline comments emit the two-segment href shape. Registered the
shorter route against the same handler; when the URL-path username is
absent the handler falls back to the workspace owner's username via a
new resolverOwnerUsername helper.
Sanity sweep:
- Dropped refItoa wrapper; use strconv.Itoa directly. The non-test
`itoa` collision was a test-only `itoa` in handlers_admin_users_test.go,
not a real symbol in non-test builds.
- Removed encodeURIComponent on workspace + ref in renderMarkdown.
parseCrossWorkspaceBody validates both against URL-safe regexes, and
wikiLinksToMarkdown doesn't encode — both functions now emit
identical bytes.
Tests:
- TestRefResolver_RejectsRefAsCollectionSlug — pins the reservation.
- TestRefResolver_TwoSegmentRouteResolves — both URL shapes resolve;
two-seg synthesizes the owner username.
- TestRefResolver_RestrictedMemberWithCollectionGrant — the
codex-flagged ACL case (restricted member + collection grant on a
different collection) now resolves to 302, not 404. Frontend test
for the round-trip override fix is documented as a gap (no vitest
infra in the repo).
* fix(links): codex round-2 fixes for cross-workspace resolver (IDEA-1492)
P1.1 — Tokenized roles bypass user-nil check. Pre-fix, checkItemVisible
rejected (nil user, "editor") tuples — exactly what RequireWorkspaceAccess
synthesizes for legacy workspace-scoped API tokens — false-404'ing every
requireItemVisible-gated handler hit by those tokens. Reordered the
checkItemVisible rules so any tokenized role (owner / editor) bypasses
the user-nil guard. checkItemVisible regression test (no HTTP layer)
pins the bypass.
P1.2 — System collections folded into item-grants branch. The pre-round-1
guestResourceFilterCore unioned ListSystemCollectionIDs into the
fullCollIDs set; the round-1 refactor dropped that union, so a
restricted member with conventions/playbooks (system collection) access
plus an unrelated item grant could LIST system items but 404 on
detail-fetch / ref-resolve. Restored the union inside checkItemVisible's
item-grants branch (non-guest path only — matches the original
guestResourceFilterCore semantics).
P1.3 — Empty owner-username 404s instead of emitting broken redirect.
When the workspace owner has no username on file (pre-setup ownerless
workspaces, legacy accounts), the synthesized redirect target became
`"/" + "" + "/" + slug + ...` → `//slug/...` — a protocol-relative URL
browsers interpret as a network-path reference. Now 404s via
refResolverNotFound rather than emitting the malformed Location header.
P1.4 — URL shape changed to /-/r/{workspace}/{ref} (Option B). Pre-fix,
the resolver lived at /{username}/{workspace}/ref/{ref}, which would
intercept item URLs in workspaces with pre-existing `ref`-slugged
collections (upgraded data; the round-1 reservation only blocks NEW
creates). Picked Option B over a migration because the feature is
unshipped, the new shape is more defensive (no future risk under any
collection slug), and the only cost is the frontend emit-shape change.
The leading `/-/r/` prefix can never collide with a user-namespace URL
because username + slug grammar both require letter-led. Frontend
renderMarkdown, wikiLinksToMarkdown, and markdownToWikiLinks all emit
and parse the new shape; the round-1 collection-slug reservation stays
as defense in depth.
Tests:
- TestCheckItemVisible_TokenizedRoleAllowsNilUser — P1.1 regression.
- TestRefResolver_RestrictedMemberWithSystemCollection — P1.2 regression.
- TestRefResolver_PreSetupBypass — P1.3 (ownerless workspace returns
404, not a broken `//slug/...` redirect).
- TestRefResolver_URLShapeNonOverlap — P1.4 (resolver doesn't intercept
`/{user}/{ws}/ref/{slug}` URLs).
- Existing TestRefResolver_* updated to the /-/r/ shape; the previous
two-segment fallback test is removed (the new URL shape has no
username component, so there's no two-segment vs three-segment
distinction to test).
* fix(links): scope round-2 bypass to nil user (Codex round-3 P1)
Round-2's checkItemVisible bypass for role in {"owner", "editor"} fired
unconditionally, including for real authenticated members. Result: a
member with workspace role "editor" and collection_access="specific"
short-circuited the per-collection filter — they could GET/PATCH/DELETE
items in collections their member_collection_access list excluded.
Scoped the bypass to the tokenized-nil-user case only:
if user == nil && (role == "owner" || role == "editor")
This is the exact set the bypass was supposed to address — fresh-install
mode (UserCount==0, role="owner") and legacy workspace-scoped API
tokens (tokenWorkspaceID matches, role="editor"). Both paths set
currentUser to nil; both are authorized by RequireWorkspaceAccess
before checkItemVisible runs.
Real authenticated members with the same roles now correctly fall
through to the existing per-collection visibility filter. Workspace
owners with default access still pass via the rule-4 "all access"
short-circuit (member.CollectionAccess == "all"); restricted editors
are now gated as intended.
Updated the rule-1 doc comment to make the scope-to-nil-user discipline
explicit — the prior wording conflated the tokenized and authenticated
paths, which is what led to the over-broad bypass.
Test: TestCheckItemVisible_AuthenticatedEditorWithRestrictedAccess
seeds a real editor with collection_access="specific" granting only
collection A, asserts visibility on a collection-B item returns false,
and adds a sanity assertion that the same editor sees collection-A
items. The existing TestCheckItemVisible_TokenizedRoleAllowsNilUser
still passes — it covers the (nil, "editor") tuple the corrected
bypass still allows.
Direct callers of checkItemVisible (grep): only requireItemVisible
(server.go) and resolverItemVisible (handlers_ref_resolver.go). Both
pass real (user, role) from request context, so the narrower scope
doesn't break any prior-green path.
* fix(links): allow digit-leading workspace slugs in xw wiki-links
Frontend WORKSPACE_SLUG_PATTERN was tighter than store.slugify (the
canonical rule): slugify keeps digit-leading inputs (e.g. "2026
Roadmap" → "2026-roadmap") but the frontend regex rejected them. Effect:
`[[2026-roadmap::TASK-1]]` fell through as a legacy title link, and
`/-/r/2026-roadmap/TASK-1` URLs didn't round-trip back to wiki syntax.
Two regex hunks, no behavior change beyond accepting the digit-led
case:
- WORKSPACE_SLUG_PATTERN: ^[a-z][a-z0-9-]*$ → ^[a-z0-9][a-z0-9-]*$
- markdownToWikiLinks reverse-regex workspace class: same widening
Stale doc-comment citing the old pattern updated to match.
Collection-slug grammar stays letter-led (the upstream rule differs;
only workspace slugs accept digit-led). Only functional consumer of
WORKSPACE_SLUG_PATTERN is parseCrossWorkspaceBody, which uses the
match boolean — no other downstream code relied on the leading-letter
constraint (Codex round-4).
|
||
|
xarmian
|
ec71903be7 |
feat(store): JSONB NOT NULL hardening on items/views + handler shape validation (IDEA-1486+1488) (#566)
* feat(store): NOT NULL hardening on items.fields/tags + views.config (IDEA-1486) Paired ship of IDEA-1486 (sibling-table JSONB NOT NULL hardening) and IDEA-1488 (handler-layer shape validation for ViewUpdate/CollectionUpdate). Generalizes the IDEA-1484 / collections.settings precedent (PR #562) to the remaining nullable JSON columns and closes the shape-validation gap that NOT NULL alone doesn't cover. Schema layer (IDEA-1486 floor): - migrations/056_items_jsonb_not_null.sql: rebuild items with fields TEXT NOT NULL DEFAULT '{}' and tags TEXT NOT NULL DEFAULT '[]', preserving all 7 indexes, recreating the 3 items_fts triggers, and rebuilding the FTS5 index. Foreign-keys-off / on bookends are lifted outside the IDEA-1485 atomic-tx wrapper. - migrations/057_views_config_not_null.sql: rebuild views with config TEXT NOT NULL DEFAULT '{}'. - pgmigrations/035 + 036: SET NOT NULL + SET DEFAULT on the three JSONB columns. Split per-table to mirror the SQLite per-table file granularity. Store layer (IDEA-1486 floor): - items.go UpdateItem and views.go UpdateView normalize "" -> "{}" / "[]" before writing. Same boundary pattern as CreateItem and the IDEA-1484 precedent at collections.go:248. - export.go ImportWorkspace coerces empty-string AND malformed JSON at import time on items.fields, items.tags, and collections.settings. Malformed input is coerce-and-log via slog.Warn (length only, never raw value) so legacy bundles don't fail-stop on one bad row. - remapFieldIDs early-returns "{}" on empty input so the second-pass UPDATE can't write "" verbatim. - Migrated the existing fmt.Printf at export.go:329 to slog.Warn for consistency. Handler layer (IDEA-1488 ceiling): - ViewCreate / ViewUpdate UnmarshalJSON via flexJSONToString with new ErrInvalidConfigType sentinel. - CollectionCreate / CollectionUpdate UnmarshalJSON with new ErrInvalidSettingsType sentinel. - handlers_views.go and handlers_collections.go surface both sentinels as 400 with the domain-level message (mirrors the BUG-1144 precedent at handlers_items.go:641). Tests: - internal/store/items_views_jsonb_test.go: store-coercion + import coercion + log-and-coerce-on-malformed + SQLite schema introspection (7 indexes + 3 FTS triggers + items_fts virtual table survival) + Postgres NOT NULL enforcement + migration re-apply idempotency + item_links round-trip after rebuild. - internal/server/handlers_views_collections_jsonb_test.go: PATCH/POST flexible-shape coverage for views.config and collections.settings, including domain-level 400 message assertions that the response does not leak Go unmarshal internals. Refs: IDEA-1486, IDEA-1488, IDEA-1484 (precedent), IDEA-1485 (substrate). * fix(store,models): codex R1 follow-ups for IDEA-1486 / IDEA-1488 Three concrete defects surfaced by codex R1 against the initial paired ship. All three close holes that defeated parts of the original contract. P1.1: migration 056 missed the playbook invocation_slug unique index. - migrations/056_items_jsonb_not_null.sql: recreate the partial UNIQUE index idx_items_invocation_slug_per_collection from migration 054 verbatim after the other 7 indexes. Without it, the application-layer pre-check in handlers_items.go:checkUniqueFields would be a TOCTOU race with no DB-level guard — the original index that 054 explicitly added as the actual uniqueness backstop would be silently dropped during the items rebuild. - items_views_jsonb_test.go: the schema-introspection test now asserts 8 indexes, not 7. Verified via `grep -rn "ON items(" migrations/` that no other items-touching indexes were missed. P1.2: flexJSONToString didn't validate inner content of JSON-encoded strings. Pre-fix, `{"config": "[]"}` / `{"settings": "not json"}` / `{"fields": "[]"}` / `{"tags": "{}"}` slipped past the shape validators because the `case '"'` branch unmarshalled the envelope and returned the inner string verbatim — bypassing the whole point of IDEA-1488. - models/item.go: after unmarshalling the JSON-encoded string, validate that the trimmed inner content's first byte matches expectedStart ('{' / '[') AND parses as JSON. Empty inner strings still pass through to the store-layer empty-string coercion (IDEA-1486 floor), so legacy "" → default normalization is preserved. - The pre-existing ItemUpdate fields/tags path inherits the same tightening because it routes through this helper — covered by new test file handlers_items_jsonb_inner_shape_test.go. - Parallel handler tests for views.config and collections.settings added to handlers_views_collections_jsonb_test.go. P2: coerceJSONForImport accepted JSON null as well-formed. - store/export.go: json.Unmarshal("null", &m) returns err=nil with m staying nil; the prior code returned the raw "null" string verbatim, which lands as JSONB null on Postgres (satisfies NOT NULL since SQL NULL ≠ JSONB null) or text "null" on SQLite. The non-nil check on the unmarshalled value routes JSON null to the existing log-and-coerce path with the rest of the malformed shapes. - items_views_jsonb_test.go: extended import test with an item carrying fields=null / tags=null; expects both coerced to "{}" / "[]" and the structured slog.Warn emitted. Verified: make test (SQLite) and the full ./... suite against the existing port-5445 Postgres container both pass cleanly. Refs: IDEA-1486, IDEA-1488, codex R1 review. * fix(store,server): codex R2 follow-ups for IDEA-1486 / IDEA-1488 Two defects surfaced by codex R2. P1 is a real ship-breaker; P2 closes a parity gap that R1 missed. P1: migration backfill normalized only SQL NULL, not malformed/wrong- shape JSON. The four new migrations originally wrote `WHERE x IS NULL`. Rows with fields = '' / 'null' / '[]' / 'not json' all survived the filter, then violated the post-migration NOT NULL+shape contract. Concrete ship- breaker on SQLite: 056 recreates the partial UNIQUE index on json_extract(fields, '$.invocation_slug') from migration 054, and json_extract errors on rows whose fields fails json_valid — a single bad row breaks CREATE INDEX mid-migration. Toggle-verified: with the NULL-only WHERE, the new SQLite test fails at exactly that CREATE INDEX with "SQL logic error: malformed JSON (1)". Widened the backfill clauses: - migrations/056: UPDATE items WHERE fields IS NULL OR json_valid(fields)=0 OR json_type(fields)!='object' (same trio for tags with 'array'). - migrations/057: same trio for views.config. - pgmigrations/035: WHERE fields IS NULL OR jsonb_typeof(fields)!='object'. JSONB rejects invalid JSON on write so the json_valid leg isn't needed on Postgres; only the shape check matters. - pgmigrations/036: same shape check on views.config. Regression tests in internal/store/items_views_jsonb_test.go: - TestItemsViewsJSONB_SQLiteBackfillRepairsMalformedShapes: applies migrations through 053 (skipping 054 which would itself error on malformed rows), seeds every observable shape pathology — SQL NULL, empty string, JSON null literal, wrong-shape JSON, non-JSON garbage — then applies 055/056/057. Asserts every malformed row is repaired AND the partial UNIQUE index actually fires on duplicate invocation_slug post-rebuild (proving the CREATE INDEX path executed end-to-end). - TestItemsViewsJSONB_PostgresBackfillRepairsMalformedShapes: parallel Postgres coverage; seeds JSONB null / array / primitive via direct ::jsonb cast and asserts the widened WHERE clause repairs each. P2: handleCreateItem didn't unwrap ErrInvalidFieldsType/ErrInvalidTagsType. R1's flexJSONToString tightening propagated the sentinels through every UnmarshalJSON path, but handleCreateItem (POST /items) still returned 'invalid JSON: <wrapped>' from decodeJSON. PATCH and the view/collection POST/PATCH handlers already unwrapped — POST was the outlier. - internal/server/handlers_items.go: mirror the PATCH-side errors.Is handling at the POST path. Brief, three-line diff. - handlers_items_jsonb_inner_shape_test.go: new TestCreateItem_ JSONEncodedStringInnerShapeValidated covers POST with fields=`[]`, fields=42, tags=`{}`, tags={"x":1}, plus a valid positive control. Asserts no "invalid JSON:" wrapper and presence of the sentinel message verbatim. Backfill-pattern audit (codex R2's grep prompt): only 055 / pg-034 (collections.settings, already shipped) exhibits the same NULL-only WHERE gap. Per the brief: NOT touched — retroactive repair belongs to a separate IDEA. Other NULL-only backfills (043/pg-023's oauth_providers, 044/pg-024's expires_at) handle their respective shapes correctly or aren't JSON columns. Verified: make test (SQLite) clean. Full ./... suite against the existing port-5445 Postgres container clean (one unrelated flake in internal/collab passed on rerun). Refs: IDEA-1486, IDEA-1488, codex R2 review. |
||
|
xarmian
|
e621eacb9b |
feat(server): POST /api/v1/import/url endpoint + integration tests (TASK-1472) (#556)
Wires internal/urlimport into the API: Fetcher → Detect → converters.
Side-effect-free; the editor's "Insert from URL" modal owns any item
mutation (TASK-1474).
Endpoint:
- POST /api/v1/import/url, body {"url"}, response {markdown,
detected_type, title?, source_url, fetched_at, content_type}.
- Status mapping: 400 invalid URL / SSRF / malformed body; 502 upstream
failure (incl. size cap); 504 fetch timeout; 422 conversion failure.
- Swagger 2.0 fallback: detected as "openapi" by the sniffer but
rejected by ConvertOpenAPI → falls through to ConvertGeneric and
re-classifies the response as "generic" so the UI shows the right
affordance.
- 30s wall-clock budget on the whole pipeline via context.WithTimeout
(Fetcher's own timeout is the HTTP-level cap).
Package-level Fetcher is memoized via the existing sync.Once-cached
safe transport (shared keep-alive pool, no per-request leak). The
handler skips the pre-flight ValidateURL when the package fetcher
has AllowLocal=true so tests can swap in a loopback-friendly fetcher
without bypassing the production guard.
Integration tests (handlers_import_test.go):
- HTML happy path (httptest upstream, asserts detected_type/title/
source_url/fetched_at/content_type all wired up).
- OpenAPI 3.x happy path (inline YAML upstream, asserts ConvertOpenAPI
was invoked and detected_type=openapi).
- Swagger 2.0 fallback (asserts detected_type re-classified to
generic, markdown non-empty).
- SSRF rejection (default fetcher, loopback URL → 400 mentioning
"private"/"reserved").
- file:// scheme rejected (400).
- Missing/malformed body rejected (400).
- Upstream 5xx surfaces as 502.
- Size cap exceeded surfaces as 502.
- No-side-effects check: item count unchanged before/after import.
Parent: PLAN-1467.
|
||
|
xarmian
|
38aa872864 |
fix(fields,activity): debounce typed-input field saves + collapse same-field activity runs (BUG-1466) (#549)
* fix(web): debounce typed-input field saves to stop per-keystroke activity rows (BUG-1466)
Text / number / URL fields in FieldEditor wired oninput directly to
onchange, so every keystroke became an item PATCH and an activity row.
Typing `ui/editor/tiptap` into a `component` field produced a 30-step
keystroke chain in the audit metadata (visible on BUG-1419's timeline).
Wrap the typed-input branches in a 500ms idle debounce, flush on blur
so tabbing away commits immediately, and flush on unmount so navigation
never drops a pending value. Discrete inputs (select / date / checkbox,
number ±1 buttons) keep firing on the user action — they aren't typing.
Mirrors the markdown content debounce pattern in the detail page.
* fix(activity): collapse same-field runs in merged changes metadata + unify diff separator (BUG-1466)
Follow-up to the web-side typing debounce. Two related changes:
1) collapseChanges() walks the merged "; "-delimited changes string and
collapses runs of consecutive same-field entries into a single
"field: first-old → last-new". Drops net no-ops (typed then backspaced).
When the web-side debounce in FieldEditor still produces multiple
PATCHes within the 5-minute coalesce window — or for older rows that
pre-date the debounce — the timeline now reads as one transition
instead of a chain. Run-based (not global) collapse so interleaved
edits on different fields keep their chronology.
2) diffFields now joins entries with "; " instead of ", " so the joiner
is consistent with mergeActivityMeta and TimelineActivityCard.svelte's
split delimiter. Multi-field PATCHes previously rendered as a single
unparseable blob in the web timeline because the parser only split on
";" — fixed as a side-effect.
Adds TestCollapseChanges (10 cases including the BUG-1419 repro) and
TestMergeActivityMeta_CollapsesSameFieldRun. Updates the existing
TestDiffFieldsPrimitives expectation to match the new joiner.
* fix(fields,activity): two follow-ups per Codex review (round 1)
[P1] FieldEditor.svelte::handleNumberStep
The ±1 buttons computed `(Number(value) || 0) + delta` AFTER calling
flushPendingSave(). But `value` is the parent prop — flushing fires
onchange asynchronously, so at the moment of the step computation
the prop still holds the pre-typed value. Typing 10 over 5 and
clicking + would flush 10 then send 6, overwriting the typed value.
Compute `base` from `pendingValue` (if hasPending) BEFORE clearing
the timer state, then send `base + delta` in one onchange call.
[P2] activities.go::collapseChanges
The drop-net-no-op step removed entries where `from == to`. But
diffFields intentionally emits same-display entries for
same-cardinality structured-field replacements like
`implementation_notes: (1 note) → (1 note)` (see
TestDiffFieldsSameCardinalityArrayChangeStillReported) — the labels
match because formatChangeValue summarizes by count, not content,
but the underlying data did change. Dropping them silently hides
real updates from the activity feed.
Track `mergedCount` per entry: increment when collapsing a run,
initialize to 1 on parse. Only drop when `mergedCount > 1 && from == to`
— i.e. only when the no-op resulted from collapsing multiple input
segments (the typed-then-backspaced case).
Tests: 2 new TestCollapseChanges cases (single structured-field
preserved, interleaved structured-field around a typed run).
* fix(fields,activity): two follow-ups per Codex review (round 2)
[P1] FieldEditor.svelte number stepper focus race
The number ±1 buttons race with the input's onblur handler: blur
fires before click in the natural focus-transfer flow, so
flushPendingSave clears hasPending → handleNumberStep reads stale
`value` from the parent prop → typing 25 over 10 and clicking +
sends 25 then 11, losing the typed value.
Add onmousedown={preventDefault} on both ±1 buttons. Mousedown
precedes blur, and preventDefault on mousedown suppresses the
natural focus transfer — the input keeps focus through the click,
so hasPending survives until handleNumberStep reads it.
[P2] collapseChanges still drops repeated same-display structured runs
Round 1's mergedCount>1 rule still dropped runs like
`implementation_notes: (1 note) → (1 note); implementation_notes: (1 note) → (1 note)`
— two real updates whose display strings happen to match because
formatChangeValue summarises array-valued fields by count. Each
PATCH represented a different underlying note (diffFields uses
reflect.DeepEqual to detect that), but the merged display showed
no transition.
Track `hadTransition` per entry: true iff the run had a display-
level transition (initial from != to, or a subsequent entry's `to`
differed from the anchored from). Drop only when
mergedCount > 1 && from == to && hadTransition — i.e. only true
net-cancellations (typed-then-backspaced). Same-display structured
repeats stay; real `foo → bar → foo` swings still drop.
Tests: 2 new TestCollapseChanges cases (repeated same-display preserved,
real foo→bar→foo swing still dropped).
* fix(fields,activity): two follow-ups per Codex review (round 3)
[P1] FieldEditor cross-item leak via debounce timer
When the parent reuses a FieldEditor instance across an item swap
(same schema, same field.key, different item — common when
navigating between items in the same collection), the parent's
`updateField` closure reads `item.id` at CALL time. A pending
timer set while item A was active would fire after item B mounted,
patching B with A's typed value.
Two-pronged fix in FieldEditor:
- Add a $effect that tracks the `value` prop and drops any pending
save the moment the parent re-props us. The user actively typed
for a now-stale context; aborting is safer than silently writing
to the new context. Also covers external collab/SSE rebases of
the same field on the same item.
- Switch the unmount cleanup from flush → drop. When the parent
navigates to an item whose schema lacks this field, the
FieldEditor unmounts AFTER the parent's `item` has already been
replaced, so a final onchange call would route through
updateField → wrong item. Blur is the supported commit gesture
(clicking elsewhere within the page, ±1 buttons, tab-out — all
flush eagerly); unmount-without-blur is treated as "user
abandoned the edit."
[P2] collapseChanges still dropped structured count-return swings
Round 2's hadTransition rule still dropped a run like
`implementation_notes: (1 note) → (2 notes); implementation_notes: (2 notes) → (1 note)`
— the user added a note then removed the original, ending with a
different single note. The merged display reads `(1 note) → (1 note)`
with hadTransition=true, indistinguishable from a typed-then-
backspaced cancellation. But formatChangeValue summaries are
LOSSY: same display label can wrap entirely different raw values.
Add a `hasLossySummary` flag per entry — true iff either `from`
or `to` matches the `(text)` format formatChangeValue emits for
structured fields. Propagate stickily across the run (any lossy
entry pins the whole run). Skip the drop step when the run
carries lossy summaries — we can't recover the raw delta from the
merged string, so we preserve the entry rather than risk hiding
a real change.
Tests: 2 new TestCollapseChanges cases (count-return swing
preserved, lossy-on-one-side still pins the run).
* fix(fields): demote hasPending from \$state to plain let — \$effect was cancelling every keystroke (round 4 [P1])
Critical regression in round 3: the value-track \$effect read
\`hasPending\` inside its body, which Svelte 5 promotes to a reactive
dependency. scheduleSave() setting \`hasPending = true\` retriggered
the same \$effect, whose body cleared the typing timer + pendingValue
+ hasPending before the debounce could fire. Net effect: typing into
any text / number / URL field was silently dropped — onchange never
ran, the field never saved.
hasPending is only read from imperative handlers (scheduleSave,
flushPendingSave, handleNumberStep, the value-track \$effect, the
unmount cleanup) — never from a template or other reactive context.
Demoting it to a plain \`let\` removes the unwanted subscription while
preserving the round-3 behaviour: external value-prop changes still
trigger the \$effect (it tracks \`value\`), the body reads hasPending
imperatively to decide whether to clear pending state.
No tests added — this is a Svelte reactivity edge case that can't
be unit-tested without a DOM. svelte-autofixer's pre-existing
"variable assigned inside \$effect" suggestion previously flagged the
hasPending mutation; that signal is gone now.
Per Codex review round 4.
|
||
|
xarmian
|
088ba2f839 |
fix(mcp): raise MCP per-token burst + classify 429 as ErrRateLimited (BUG-1430) (#546)
* fix(mcp): raise MCP per-token burst + classify 429 as ErrRateLimited (BUG-1430)
BUG-1409 reported an agent hitting "Pad backend 500s on parallel writes"
during workspace onboarding via remote MCP on Pad Cloud. Triage split
that umbrella into three children; this PR addresses BUG-1430 (the
parallel-writes symptom).
Root cause investigation showed the underlying write path is fine —
local SQLite handled 24 parallel item-create POSTs cleanly (busy_timeout
+ BEGIN IMMEDIATE + WAL serialize writers without errors). The most
plausible cause of the agent's "500 on parallel writes" report is the
MCP per-token rate limiter (burst 20, 60/min) rejecting requests 21-24
of an onboarding burst with HTTP 429, which the dispatcher's classifier
then collapsed into a generic ErrServerError envelope.
Changes:
- middleware_ratelimit.go: MCPPerToken burst 20 → 60. Sustained rate
unchanged at 60/min/token. Matches the general API limiter's burst-60
per-user cap so the MCP path no longer imposes a tighter ceiling than
the equivalent /api/v1 path. Comment expanded to record the rationale.
- internal/mcp/errors.go: add ErrRateLimited error code and an explicit
case http.StatusTooManyRequests in classifyHTTPStatusKind. 429s now
surface as a first-class rate-limited envelope with an actionable hint
pointing at Retry-After and the per-token cap, instead of landing in
the generic ErrServerError "other 4xx" bucket. Agents implementing
exponential backoff can switch on code without parsing free-form text.
- handlers_cloud.go: add slog.Error instrumentation to enforcePlanLimit
and enforceUserPlanLimit error paths. These are cloud-mode-only 500
candidates we couldn't exercise locally (local dev runs cloudMode=false);
the structured logs give operators a grep-able tag the next time the
symptom surfaces on real Pad Cloud, so we can rule the path in or out
empirically without another investigation pass.
- tests: bump iteration counts past the new burst (20 → 60), add 429
case to classifyHTTPStatus code-mapping table + envelope hint-shape
table.
Investigation context (full triage in BUG-1430):
- ../pad-cloud sidecar is NOT in the /api/v1 or /mcp request path
(nginx-router proxies those directly to pad backend).
- featureCount + advisory-lock contention on Postgres remain plausible
500 candidates under heavy bursts; the new logging is intended to
catch those if they fire.
Siblings BUG-1431 (status field placement) and BUG-1432 (tags field)
are tracked separately and not addressed here.
* fix(mcp): drop hardcoded cap from rate-limit hint per Codex review (round 1)
Codex round 1 [P2] caught that rateLimitHintFor's "the per-token cap is
60 req/min with a burst of 60" text was misleading: classifyHTTPStatusKind
handles 429s from the dispatcher's SYNTHESIZED /api/v1/... requests, which
come from the general API limiter (600/min, burst 60), the Search limiter
(30/min, burst 10), and potentially others — NOT the MCP per-token
limiter (which fires before the dispatcher runs and so never lands in
this classifier path).
Generalize the hint: point at Retry-After (which carries the correct
limiter-specific wait) and drop the cap from prose. Update the matching
test assertion to assert the generic shape ("burst-heavy" instead of
"60 req/min").
|
||
|
xarmian
|
d18a5d8140 |
refactor(bootstrap): omit redundant schema labels + omitempty sort_order (TASK-1424) (#543)
Two small additive trims to the BootstrapCollection projection
introduced in TASK-1412.
## 1. Omit redundant schema `label` when label == TitleCase(key)
A schema field's `label` is auto-fillable from its `key` (the CLI
and the MCP-side CreateCollection helper both apply
TitleCase(key) when label is empty — see titleCaseLabel in
internal/mcp/dispatch_http_routes.go). When the persisted label
matches that rule, it's redundant — the agent can reconstruct it
from the key. Examples on docapp:
- {"key":"status", "label":"Status"} ← redundant
- {"key":"due_date", "label":"Due Date"} ← redundant
- {"key":"trigger", "label":"When"} ← CUSTOM, preserved
Implementation: projectBootstrapCollection now passes the schema
bytes through trimRedundantSchemaLabels, which:
1. Unmarshals into a parallel bootstrapSchema/bootstrapFieldDef
struct purpose-built for the bootstrap shape.
2. Walks fields, clears any Label that equals TitleCase(Key).
3. Re-marshals — `omitempty` on bootstrapFieldDef.Label drops
the empty-string labels from the output.
Field ordering is preserved by struct-based marshalling.
## 2. Omitempty on BootstrapCollection.SortOrder
`sort_order` defaults to 0. Most collections never get an explicit
non-zero sort_order, so the field carried "sort_order":0 per entry
needlessly. Added ,omitempty to the struct tag.
## Drift detection
bootstrapFieldDef mirrors models.FieldDef field-for-field with two
deliberate differences: `Label` is omitempty, and `Default` is
json.RawMessage (so any default value round-trips verbatim
without re-parsing). The risk: if models.FieldDef gains a new
field, bootstrapFieldDef silently loses it from the bootstrap
schema response.
TestBootstrapFieldDefMirrorsModelsFieldDef catches this via
reflection — compares NumField + per-field JSON tags + has an
explicit allow-list for the Label tag delta. A new field added to
models.FieldDef without mirroring here fails the test with a
field-name-pointed error message.
## Test coverage
- TestBootstrapFieldDefMirrorsModelsFieldDef — drift detector.
- TestTrimRedundantSchemaLabels (4 subtests):
* drops-redundant-labels
* preserves-custom-labels (key="trigger", label="When" stays)
* multi-word-keys-titlecase-correctly (due_date → Due Date)
* malformed-schema-returns-raw (defensive: never block on parse error)
- Existing TestBootstrapSizeBudget shows fixture collections
section drop: 3,979 → 3,532 bytes (-11.2%).
## Measurements
Fixture (TestBootstrapSizeBudget): 7,823 → 7,376 bytes (-447 b / -5.7%).
Collections section alone: 3,979 → 3,532 bytes (-447 b / -11.2%) —
all of the win is concentrated in schema bytes via the label trim.
Live docapp expected savings: 9,384 → ~8,400 bytes (~10% drop on
collections), totaling roughly 600-900 b additional reduction on
the bootstrap response.
## Out of scope
- ToolSurfaceVersion 0.3 → 0.4 bump — TASK-1418 (the FINAL PR in
PLAN-1410). This is the last shape PR; TASK-1418 is now unblocked.
Parent: PLAN-1410. With this merged, PLAN-1410's bootstrap-shape
work is complete; only the contractual v0.4 announcement remains.
|
||
|
xarmian
|
ad87b960d7 |
refactor(bootstrap): slim Roles projection (BootstrapRole struct) (TASK-1423) (#542)
Same drop-pattern as TASK-1412's BootstrapCollection applied to the
bootstrap response's `roles` section.
## Struct + helper
New BootstrapRole struct purpose-built for the bootstrap response:
- Keeps: slug, name, description, icon, sort_order, item_count
- Drops: id, workspace_id, tools, created_at, updated_at
The `tools` field has no consumer outside the store CRUD
(grep confirms — only referenced in internal/store/agent_roles.go);
docapp has it set to "" on all three roles in practice. If it ever
becomes load-bearing for the agent, it can be added back.
New projectBootstrapRole(models.AgentRole) helper mirrors
projectBootstrapCollection.
## BuildAgentBootstrap reorder
Roles projection now happens AFTER the role-count recompute for
restricted callers (which keys lookups by AgentRole.ID, which the
projection drops). Same reorder pattern as TASK-1412's collections
refactor — the local `roles` slice stays in models.AgentRole shape
through the count recompute, then projects in a final pass alongside
the collections projection.
## Tests
New TestBootstrapRoleProjection seeds one role via the agent-roles
endpoint and verifies the wire shape:
- Positive: slug/name/description/item_count are present and correct.
- Negative: id/workspace_id/tools/created_at/updated_at are NOT
present in the marshalled JSON.
The negative check is the load-bearing part — without it, a future
refactor that "fixes" the projection by re-adding a UUID would pass
the positive assertions silently. Mirrors
TestBootstrapEmptyArraysNotNull's pattern for the recent_activity dedup.
Existing TestBootstrapEmptyWorkspace and TestBootstrapEmptyArraysNotNull
still pass — their `b.Roles != nil` and `roles != null` checks work
at slice-level regardless of element type.
## Measurements
Fixture (TestBootstrapSizeBudget) unchanged at 7,823 bytes — the
fixture seeds 0 roles so the projection change doesn't affect the
size budget (roles section was already at 2 bytes "[]").
Live docapp measurement deferred until make install — expected
savings: roles section was 1,066 bytes for 3 roles; drop is
~30-40% (~350 b) of that section.
## Out of scope
- Schema label + sort_order trim — TASK-1424.
- ToolSurfaceVersion 0.3 → 0.4 — TASK-1418 (final PR).
Parent: PLAN-1410.
|
||
|
xarmian
|
aaa581b105 |
refactor(bootstrap): extend dashboard caps to active_items/active_plans/by_role (TASK-1422) (#541)
* refactor(bootstrap): extend dashboard caps to active_items/active_plans/by_role/suggested_next (TASK-1422) Implements IDEA-1421 (absorbed into PLAN-1410's v0.4 envelope). Extends the BootstrapDashboard wrapper from TASK-1413 to cap four more dashboard sub-arrays with parallel overflow counts, same shape and semantics as the existing attention/recent_activity caps. ## Struct + caps Four new int fields on BootstrapDashboard (all `,omitempty`): - active_items_overflow_count - active_plans_overflow_count - by_role_overflow_count - suggested_next_overflow_count Four new cap constants alongside the existing two: - bootstrapActiveItemsCap = 5 - bootstrapActivePlansCap = 5 - bootstrapByRoleCap = 5 - bootstrapSuggestedNextCap = 5 capBootstrapDashboard extended with four parallel truncate-and-count blocks — same shallow-copy mutation pattern, source pointer untouched (the dashboard endpoint still returns its full-length arrays per its own contract). ## Tests TestCapBootstrapDashboard rewritten to cover all six caps under one contract. `mk` now takes a `dashCounts` struct (Att/Rec/Items/Plans/ Role/Sugg) so each subtest exercises specific caps without populating the others. Each of the four existing subtests (under-cap-no-overflow, over-cap-truncates-and-counts-overflow, source-pointer-unchanged, exact-cap-no-overflow) now asserts the new caps too. Added tiny assertLen/assertOverflow helpers to keep the per-array assertion noise from drowning the contract being tested. bootstrapSectionBytes extended to surface the four new cap-effect lines when triggered, table-driven so future caps drop in cleanly. seedBootstrapSizeFixture updated to seed 6 in_progress tasks (was 5 open) so the new active_items cap fires visibly in the per-section breakdown: "active_items capped: 5 shown, 1 overflow". The status flip is deliberate — dashboard.active_items filters on isActiveStatus(), which excludes initial/terminal statuses; open tasks never appeared in the section. ## Budget bootstrapSizeBudget 7 KiB → 9 KiB. Note that this is FIXTURE-side growth, not shape-side regression: the fixture now seeds enough active items to exercise the new cap (active_items section was 0 bytes when tasks were status=open). The cap itself is purely a SAVINGS — on docapp it drops active_items from 7 → 5 entries with overflow_count=2. Budget history note in handlers_bootstrap_test.go updated with the TASK-1422 line and an explicit "fixture-side, not shape-side" explanation so future readers understand why the budget moved up. ## Out of scope - Slim BootstrapRole projection — TASK-1423. - Schema label + sort_order trim — TASK-1424. - ToolSurfaceVersion 0.3 → 0.4 bump — TASK-1418 (final PR). Parent: PLAN-1410. Resolves IDEA-1421 once merged. * fix(bootstrap): drop unreachable suggested_next cap (TASK-1422 follow-up) Address Codex P1 finding on PR #541: `suggested_next_overflow_count` was unreachable in production responses because `buildDashboardResponse` already truncates `SuggestedNext` to 3 upstream (see "Take top 3" comment in handlers_dashboard.go:854-858), while my bootstrap cap was 5. The cap-and-overflow logic could only have fired against synthetic test state, never against the real dashboard pipeline. Two responses to consider: 1. Lower bootstrap's cap to a number smaller than 3 — defeats the upstream design choice (3 IS the intentional limit). 2. Drop the bootstrap-side cap — clean, no dead surface. Going with (2). If the upstream cap is ever raised or removed, that's the moment to add a suggested_next_overflow_count back. Removed: - SuggestedNextOverflowCount field on BootstrapDashboard - bootstrapSuggestedNextCap constant - The truncate-and-count block in capBootstrapDashboard - The suggested_next row in TestCapBootstrapDashboard's dashCounts helper and all five subtest assertions - The suggested_next entry in bootstrapSectionBytes's cap-line loop The fixture still seeds tasks and a plan, so the no-cap path on SuggestedNext is naturally exercised through TestBootstrapSizeBudget. The godoc on BootstrapDashboard now explicitly calls out the exclusion + the upstream-cap rationale so a future reader knows why suggested_next is missing from the otherwise-uniform cap set. Parent: PLAN-1410 / TASK-1422. * docs(skill): align SKILL.md dashboard cap description with TASK-1422 Address Codex P3 finding on PR #541: the SKILL.md `Context Loading` section described only the two original cap fields (attention_overflow_count, recent_activity_overflow_count). After TASK-1422 the bootstrap response carries three more (active_items_overflow_count, active_plans_overflow_count, by_role_overflow_count), and the agent needs to know to pull the full set via `pad project dashboard` when any of them are > 0. Updated the bullet to enumerate all five capped sub-arrays and state the overflow-field pattern generically rather than per-field. Same in-PR-sync pattern used for TASK-1413, TASK-1415, TASK-1416. Parent: PLAN-1410 / TASK-1422. * docs(bootstrap): fix three stale comments after dropping suggested_next cap (TASK-1422 follow-up) Address Codex P3 finding on PR #541 round 3: three stale doc strings referenced the old shape (with suggested_next) after the cap was dropped in the prior commit. Updated: 1. handlers_bootstrap_test.go budget-history line for TASK-1422 — removed `suggested_next` from the cap list and added the "deliberately excluded — already capped to 3 upstream" rationale so future readers know why the otherwise-uniform cap set is missing one. 2. handlers_bootstrap.go BootstrapDashboard godoc — changed "two overflow counts" to "five overflow counts (one per capped sub-array)". 3. handlers_bootstrap.go capBootstrapDashboard godoc — changed "both caps are untriggered" to "all caps are untriggered". Tidy-up only, no behavior change. Same skill-↔-code sync hygiene that has been the running theme across PLAN-1410's review loops. Parent: PLAN-1410 / TASK-1422. * docs(bootstrap): update remaining stale call-site comment for capBootstrapDashboard (TASK-1422 follow-up) Final stale-doc cleanup per Codex P3 on PR #541 round 4: the BuildAgentBootstrap dashboard-wrapping call-site comment still listed only attention + recent_activity. Updated to enumerate all five capped sub-arrays for parity with the godoc on BootstrapDashboard / capBootstrapDashboard. Same hygiene as the previous commit; no behavior change. Parent: PLAN-1410 / TASK-1422. |
||
|
xarmian
|
9c9aac3ea6 |
chore(bootstrap): tighten size budget 8 KiB → 7 KiB to lock in PLAN-1410's win (TASK-1417) (#540)
PLAN-1410's bootstrap shape work is complete on main (TASK-1411..1416). The fixture still measures 6,355 bytes against the 8 KiB budget — too much headroom for a ratchet whose purpose is to detect shape regressions. Tightened to 7 KiB: - Fixture: 6,355 bytes - Budget: 7,168 bytes (7 KiB) - Headroom: 813 bytes (~12.8%) Tight enough to catch any meaningful shape regression (reintroducing a duplicated field, un-capping a dashboard array, re-stringifying schema), loose enough to absorb routine schema reordering or single-field additions without false alarms. Budget-history note updated with the TASK-1417 line and a forward-looking pointer at IDEA-1421 (next-round dashboard sub-array caps) which will land its own win under its own ratchet. The companion measurement (per-section bytes for fixture + live docapp + pre/post per-invocation totals) was recorded directly into PLAN-1410's body via `pad item update PLAN-1410 --stdin`, which is the canonical home for plan results. Cumulative win: fixture: 13,861 → 6,355 (-54.2%) live docapp bootstrap: 52,033 → 33,375 (-35.9%) live SKILL.md: 40,193 → 29,840 (-25.8%) live combined per /pad: 92,226 → 63,215 (-31.5%) The original ~43% target was set against a static-workspace assumption; the live shortfall is workspace-scale-dependent (conventions on docapp are 11.2 KB vs 0.5 KB on the fixture). IDEA-1421 captures the next round (dashboard active_items / active_plans / by_role / suggested_next caps, estimated ~3 KB additional live win, backwards-compatible). Parent: PLAN-1410. Final remaining task: TASK-1418 (ToolSurfaceVersion 0.3 → 0.4 contractual announcement). |
||
|
xarmian
|
638a456f98 |
refactor(bootstrap): dedup recent_activity, drop convention slug, cap dashboard arrays (TASK-1413) (#536)
* refactor(bootstrap): dedup recent_activity, drop convention slug, cap dashboard arrays (TASK-1413)
Three bundled handler-level cleanups against PLAN-1410's bootstrap
shape. Total fixture savings: 8,992 → 6,355 bytes (-2,637 b / -29%).
1. Drop duplicate top-level `recent_activity`
AgentBootstrap.RecentActivity was bit-for-bit identical to
AgentBootstrap.Dashboard.RecentActivity. Removed:
- AgentBootstrap.RecentActivity field
- capRecentActivity() helper
- recentActivityWindow constant
- the time import (no longer used)
Fixture savings: -1,751 bytes.
2. Drop `slug` from AgentBootstrapConvention
Agents address convention items by ref (CONVE-N); slug was dead
weight. Removed the field + the population line in
collectAlwaysOnConventions.
Fixture savings: -78 bytes.
3. Cap dashboard.attention + dashboard.recent_activity to 5 in bootstrap
New BootstrapDashboard wrapper embeds *DashboardResponse (so the
wire shape stays compatible — same field names, same nesting) and
adds two overflow counts:
- attention_overflow_count (omitempty when zero)
- recent_activity_overflow_count (omitempty when zero)
The cap is applied via capBootstrapDashboard which shallow-copies
the DashboardResponse before truncating the slices, so callers
downstream of buildDashboardResponse (the dashboard endpoint
itself, the web UI) see their original full-length arrays
unchanged. `pad project dashboard` contract is preserved verbatim.
Fixture savings: -789 bytes (recent_activity capped 9 → 5;
attention untouched, fixture has 0 attention items).
Coverage:
- TestCapBootstrapDashboard (4 subtests): under-cap-no-overflow,
over-cap-truncates-and-counts-overflow, source-pointer-unchanged,
exact-cap-no-overflow. Locks in the cap contract independent of
the full bootstrap pipeline.
- TestBootstrapEmptyArraysNotNull updated: the top-level
recent_activity key was removed from the required-keys list,
with a separate assertion that guards against it reappearing.
- TestBootstrapEmptyWorkspace updated: removed the b.RecentActivity
nil-check; added a (defensive) check that dashboard's nested
recent_activity is non-nil when dashboard is present.
- bootstrapSectionBytes now surfaces the cap effect ("attention
capped: 5 shown, 4 overflow") when triggered, so the trim's
value is legible from CI output.
bootstrapSizeBudget tightened 11 KiB → 8 KiB to lock in the win.
Budget-history comment updated.
Out of scope (handled by later PLAN-1410 PRs):
- Skill-file trim (TASK-1414/1415/1416)
- Final measurement (TASK-1417)
- ToolSurfaceVersion 0.3 → 0.4 (TASK-1418, after all shape
changes land)
Parent: PLAN-1410.
* docs(skill): align SKILL.md bootstrap shape with PLAN-1410 / TASK-1413
The skill's `Context Loading` section described the old wire shape:
- `dashboard {...}` — active items, attention, suggested next, recent activity
- `recent_activity [...]` — capped to the last 24h
After TASK-1413 the top-level `recent_activity` field is gone (it was
a bit-for-bit duplicate of `dashboard.recent_activity`), and the
remaining `dashboard.recent_activity` is capped by COUNT (top 5) not
by TIME (24h window). The two cap fields (attention_overflow_count
and recent_activity_overflow_count) tell the agent how much was
trimmed so it can decide whether to follow up with a full
`pad project dashboard` query.
Per the Codex P2 finding on PR #536: documenting the new contract
in this PR keeps skill ↔ wire-shape strictly synchronized (no
window where the docs are wrong about the shape this PR ships).
Parent: PLAN-1410 / TASK-1413.
|
||
|
xarmian
|
8da0473e4e |
refactor(bootstrap): slim Collections projection — drop id/timestamps/settings, parse schema inline (TASK-1412) (#535)
Introduces BootstrapCollection — a purpose-built projection for the
bootstrap response that replaces []models.Collection on
AgentBootstrap.Collections. Drops fields the /pad skill never reads:
- id, workspace_id — agent addresses collections by slug
- created_at, updated_at, — irrelevant at context-load time
deleted_at
- settings — quick_actions + view defaults are
web-UI chat-prompt config
The remaining schema string is delivered as a nested JSON object
(json.RawMessage) rather than a JSON-encoded string, killing the
backslash-escape overhead so the agent sees real {}/[] structure
instead of double-encoded quotes. json.Valid() gates the emission
so a future migration leaving non-JSON in the column can't break
agent-side json.Unmarshal — invalid/empty schemas are simply
omitted (omitempty).
Measured against the bootstrapSizeBudget fixture (TASK-1411):
before after delta
collections 8,848 b 3,979 b -4,869 b (-55%)
total bootstrap 13,861 b 8,992 b -4,869 b (-35%)
Budget tightened from 16 KiB to 11 KiB to lock in the win. Later
PLAN-1410 PRs (TASK-1413's dedup + dashboard caps, TASK-1417's
final measurement) tighten further.
Wire-shape change details:
- BuildAgentBootstrap holds collections as []models.Collection
through the visibility-restricted role+count recompute (which
keys lookups by Collection.ID), then projects to []BootstrapCollection
at the end of that section. ID-keyed recompute logic is preserved
verbatim — only the final wire shape changes.
- printBootstrapMarkdown was already reading {slug, name, prefix}
via its own anonymous struct; those three are preserved.
- No web-UI consumers exist for /agent/bootstrap (grep confirms),
so no client-side churn.
Out of scope (handled by later PLAN-1410 PRs):
- Dedup'ing top-level recent_activity, dropping convention slug,
capping dashboard.attention/recent_activity (TASK-1413).
- ToolSurfaceVersion bump 0.3 → 0.4 (TASK-1418, after all shape
changes land).
Parent: PLAN-1410.
|
||
|
xarmian
|
91f1c0a017 |
test(bootstrap): add size-budget benchmark for the agent bootstrap response (TASK-1411) (#534)
Adds TestBootstrapSizeBudget which builds an AgentBootstrap blob against
a seeded representative fixture (default template seeds + 2 always-on
conventions with bodies + 1 slug-invocable playbook + 5 tasks + 1 plan)
and asserts the marshalled JSON byte count stays at or below
bootstrapSizeBudget (initially 16 KiB, against a current actual of
~13.8 KiB on the seeded fixture).
On every run — pass or fail — the test logs a per-section breakdown
(workspace / user / collections / conventions / roles / playbooks /
dashboard / recent_activity) so size regressions are diagnosable from
CI output alone, and so the cumulative trim across PLAN-1410 is visible
as the budget tightens.
This is the baseline ratchet for PLAN-1410's bootstrap-shape PRs:
- TASK-1412 (slim Collections projection) tightens the budget down
once schema-as-string and the redundant ids/timestamps come out.
- TASK-1413 (dedup top-level recent_activity, drop convention slug,
cap dashboard arrays) tightens further.
- TASK-1418 records the final v0.4 actual.
The docapp workspace currently measures ~52 KB / ~13K tokens on the
real bootstrap payload; the fixture is intentionally small but
exercises the same shape contributors so a regression in the projected
shape (per-collection settings, schema-as-string, duplicate
recent_activity, etc.) trips the budget at fixture scale.
Parent: PLAN-1410.
|
||
|
xarmian
|
fed4b60e65 |
feat(playbook): add pad playbook CLI (list/show/run) + endpoints (TASK-1382) (#520)
* feat(playbook): add pad playbook CLI (list/show/run) + endpoints (TASK-1382)
PLAN-1377 T5 — first-class invokable-procedure surface. Three HTTP
endpoints + three CLI subcommands, with a strict CLI arg parser and a
side-effect-free run path.
Endpoints
GET /workspaces/{ws}/playbooks — metadata array, same
projection as bootstrap
GET /workspaces/{ws}/playbooks/{ref} — full item, resolved by
invocation_slug | ref | slug
POST /workspaces/{ws}/playbooks/{ref}/run — bind args, return body +
bound + unbound. No
execution; the agent
owns step playback.
Resolution
resolvePlaybook walks invocation_slug first (the /pad <slug>
user-facing identifier), then falls back to ResolveItem (UUID / ref /
slug). A stray TASK-5 hitting /playbooks/TASK-5 returns 404 instead
of leaking a non-playbook into the surface.
Arg parsing
ParsePlaybookCLIArgs implements the strict rules from PLAN-1377:
- Required positional args first, in declared order.
- Flag types: bareword presence sets true.
- Other types: key=value form (number is parsed as float64,
enum is validated against declared options).
The server takes either pre-parsed args (MCP / programmatic callers)
OR raw CLI tokens (CLI caller); merge logic prefers explicit args
over raw_args. The CLI sends raw_args, so there is one parser
implementation and no drift risk.
CLI
pad playbook list — table or json
pad playbook show <slug|ref> [--format] — markdown / json
pad playbook run <slug|ref> [args...] — body + bound args
Client method
cli.Client.{ListPlaybooks, ShowPlaybook, RunPlaybook(args,
rawArgs)} — runtime callers pass args; CLI passes rawArgs.
Tests
TestPlaybookList, TestPlaybookShowByInvocationSlug,
TestPlaybookShowByRef, TestPlaybookShowRejectsNonPlaybook,
TestPlaybookRunBindsArgs (with-args, missing-required, with-raw-args),
TestParsePlaybookCLIArgsErrors, TestPlaybookListEmptyShape.
Parent: PLAN-1377.
* fix(playbook): tighten arg parsing per Codex round 1 (TASK-1382)
P2.1 — Positional binding now skips optional and flag-typed slots.
The PLAN-1377 contract says ONLY required args fill positional slots;
other typed args must be key=value. Without this, a spec with an
optional arg before a required one (e.g.
[merge-strategy?, target!]) bound the caller's bareword TASK-7 to
merge-strategy instead of target.
P2.2 — number coercion now uses strconv.ParseFloat with NaN/Inf
rejection. Sscanf(%g) was sloppy: it accepted '1abc' as 1 (partial
match) and accepted NaN/Inf, which json.Marshal then refused after
the handler had already written a 200 header.
P3 — empty-body run requests now decode cleanly. The decodeJSON
wrapper folds io.EOF into 'invalid JSON: EOF' so the previous
err.Error() != "EOF" check never fired. errors.Is(err, io.EOF) on
the unwrapped chain handles the wrapping correctly.
Tests: TestPlaybookRunAcceptsEmptyBody,
TestParsePlaybookCLIArgsOptionalNotPositional,
TestCoercePlaybookValueNumberRejectsBadInput.
Parent: PLAN-1377.
|
||
|
xarmian
|
24f0445efa |
feat(bootstrap): single-roundtrip /pad context-load endpoint (TASK-1379) (#518)
* feat(bootstrap): single-roundtrip /pad context-load endpoint (TASK-1379)
Implements the agent bootstrap surface for PLAN-1377. One HTTP call
replaces the four separate /pad context-loading invocations (workspace
+ collections + conventions + roles + playbooks) the skill used to
make, cutting ~200-400ms of startup latency on every /pad command.
Wire shape:
- GET /api/v1/workspaces/{ws}/agent/bootstrap returns AgentBootstrap:
workspace { slug, name, id }, user { name, email, id },
collections [...], conventions [...always-on, status=active],
roles [...], playbooks [metadata-only — no bodies], dashboard {...},
recent_activity [... 24h].
- pad bootstrap [--format json|markdown] CLI wrapper. JSON is the
canonical wire format that the /pad skill consumes; markdown is a
human-readable summary for quick terminal inspection.
Implementation notes:
- Single source of truth: Server.BuildAgentBootstrap. The HTTP handler
is a thin wrapper, and TASK-1380 will reuse it from three MCP
surfaces (resource + set_workspace embed + pad_meta tool action).
- Dashboard reuse: handleGetDashboard's body extracted into
buildDashboardResponse(workspaceID, r) returning (*DashboardResponse,
error). The HTTP handler is now a thin wrapper; bootstrap calls the
builder directly to embed dashboard data without a second roundtrip.
- Playbook bodies are deliberately NOT included — metadata only
(~80 bytes/entry vs 5-10KB) so the bootstrap stays small for
workspaces with many playbooks. Full bodies load on invocation.
- Convention bodies ARE included for the always-on/active subset (must
be agent-known up front); trigger-specific conventions stay
load-on-demand.
- Empty slices serialize as [] not null so the agent doesn't need
defensive nil checks.
Tests:
- TestBootstrapEmptyWorkspace, TestBootstrapEmptyArraysNotNull,
TestBootstrapIncludesPlaybookMetadata,
TestPlaybookSummaryPrefersFirstParagraph.
Parent: PLAN-1377.
* fix(bootstrap): respect collection visibility + guest grants (TASK-1379)
Codex round 1: BuildAgentBootstrap was bypassing visibility filters,
so a guest admitted by RequireWorkspaceAccess could read collections,
conventions, and playbook metadata they don't have access to.
Now mirrors handleListCollections + handleGetDashboard:
1. Resolve visibleCollectionIDs(r, workspaceID) once.
2. Filter the Collections array through isCollectionVisible.
3. Gate the conventions + playbooks sub-queries on whether the caller
can see those collections at all (presence in the filtered slice
implies visibility).
4. Empty slices for inaccessible sub-resources, so the wire shape
stays consistent — no missing keys to confuse the agent skill.
5. Pass-r=nil callers (future MCP in-process dispatchers) keep the
'full visibility' shortcut, but the doc comment now spells out that
those callers MUST verify access out-of-band.
Parent: PLAN-1377.
* fix(bootstrap): apply guest item-level grants + recompute role counts (TASK-1379)
Codex round 2:
P1 — Item-level guest grants now flow into the convention + playbook
sub-queries. visibleCollectionIDs alone admits the conventions /
playbooks collection if a guest has ANY item grant inside it; without
ItemIDs filtering, those queries then return the whole always-on
convention body set or every playbook's metadata. Now mirrors the
handleListItems shape: resolve (fullCollIDs, grantedItemIDs) via
guestResourceFilter, and pass the (collIDs, itemIDs) tuple through to
collectAlwaysOnConventions + collectPlaybookMetadata so a guest with a
single grant only sees that one item.
P2 — AgentRole.ItemCount is now recomputed from the visible item set
for restricted callers, matching handleListAgentRoles. Without this,
a guest could read role counts computed across all workspace items and
infer hidden activity per role.
Helper signatures updated to accept (collIDs, itemIDs); the doc
comments explain the nil/non-nil semantics so future callers can't
silently regress this.
Parent: PLAN-1377.
* fix(bootstrap): rewrite collection counts from visible set for guests (TASK-1379)
Codex round 3: Collection.item_count + active_item_count are computed
across the whole collection by ListCollections, so a guest with one
item grant in a collection still received the collection in the
filtered list but with hidden counts. Reuse the visible item set
(already computed for role counts) to recompute collection item_count
for restricted callers. active_item_count is set equal to item_count
to avoid a separate done-rules buildup the bootstrap consumers don't
depend on — better a self-consistent number than a leaked one.
Parent: PLAN-1377.
|
||
|
xarmian
|
c38b3bf5cd |
feat(playbooks): add invocation_slug + arguments schema fields (TASK-1378) (#517)
* feat(playbooks): add invocation_slug + arguments schema fields (TASK-1378)
Foundational change for PLAN-1377 — playbooks become first-class invokable
procedures. Two new optional fields land on the Playbooks collection
schema:
- `invocation_slug` (text, kebab-case, unique-per-workspace among non-null
values): enables `/pad <slug>` direct invocation. Nullable so
trigger-only playbooks (e.g. on-release checklists) don't need one.
- `arguments` (json, array of {name, type, required, default, description}):
declares the playbook's argument contract; mirrors the body's
`## Arguments` section in queryable form.
Plumbing pieces:
- `models.FieldDef` grows two general-purpose options — `Pattern` for
regex validation and `UniqueScope` for collection-level uniqueness.
Both are opt-in; existing schemas are unaffected.
- `items.ValidateFields` learns the `json` field type (accepts any
JSON-decodable value) and applies `Pattern` to string-typed values.
- `handlers_items.checkUniqueFields` queries `Store.ListItems` to enforce
`UniqueScope == "workspace_collection"` on create + update.
- Two migrations (SQLite 054, Postgres 033) JSON-patch the playbooks
schema on existing workspaces so the new fields show up without a
workspace re-init.
- TypeScript `FieldDef` mirrors the Go side.
Parent: PLAN-1377.
* fix(playbooks): address Codex review round 1 findings (TASK-1378)
P1 — EditCollectionModal now round-trips opaque pattern/unique_scope
metadata. EditableField carries the new keys; the load + save paths
preserve them so re-saving the playbooks collection from the UI doesn't
strip server-side validation rules the modal doesn't yet expose
dedicated controls for. fieldFromDef mirrors the change for templates.
P2 — checkUniqueFields' pre-write ListItems check is now backed by a
partial unique index (idx_items_invocation_slug_per_collection,
SQLite + Postgres) scoped to non-empty, non-deleted rows. The pre-check
still gives users a friendly error message in the common case; the
index closes the TOCTOU race between two concurrent writers. The
create-conflict error message is now generic enough to cover both the
slug constraint and the new invocation_slug index.
P2 — `json` field type now rejects raw strings, numbers, and bools. Only
objects, arrays, and null are accepted, so a generic web text input
can't silently corrupt a structured field by emitting "[]" instead of
an actual array. FieldEditor.svelte routes `json` fields to a
read-only summary in both readonly and edit modes; dedicated editors
(like TASK-1384's playbook editor that owns `arguments`) own the
structured form.
P3 — invocation_slug regex now requires a minimum of two characters
(`^[a-z0-9][a-z0-9-]*[a-z0-9]$`) in the Go const, the SQLite migration,
the Postgres migration, and the validate tests. Single-letter slugs
would shadow plausible NL tokens (e.g. `/pad a ...`) and the doc
comment already claimed the two-char floor; this aligns code with
intent.
Parent: PLAN-1377.
* fix(playbooks): address Codex review round 2 findings (TASK-1378)
P2.1 — checkUniqueFields no longer passes IncludeArchived=true. The
application-layer pre-check now matches the partial unique index's
`deleted_at IS NULL` predicate so a soft-deleted playbook releases its
slug back to the pool and reclaiming it succeeds instead of 409'ing.
P2.2 — handleUpdateItem now maps UNIQUE constraint / duplicate key
errors from UpdateItem to HTTP 409, mirroring the create path. A true
concurrent-update race that slips past checkUniqueFields and trips the
partial unique index used to surface as a misleading 500.
(Not addressed in this round: Codex's third finding — concern about the
partial unique index applying to "every collection" — is, on close
reading, not what the index does. `ON items(collection_id, json_extract(...))`
scopes uniqueness to the (collection_id, slug) pair, so two items in
different collections with the same `invocation_slug` value coexist
fine. The migration-failure risk is theoretical: `invocation_slug` is
a brand-new field key, so no pre-existing items can have it set, and
no migration-time duplicates can exist. If a future custom collection
adopts the same field name, opting into per-collection uniqueness is
exactly the intended semantic of FieldDef.UniqueScope.)
Parent: PLAN-1377.
* fix(playbooks): map restore-path UNIQUE violations to 409 (TASK-1378)
Codex round 3: restoring an archived playbook can hit the partial
unique index on invocation_slug if a replacement item already claimed
the slug. Map UNIQUE constraint / duplicate key errors from RestoreItem
to HTTP 409 with a targeted message, matching the create + update paths.
Parent: PLAN-1377.
* fix(playbooks): map collab-snapshot UNIQUE violations to 409 (TASK-1378)
Codex round 4: the collab-snapshot PATCH branch under
`s.collab.UnderItemLock` ran its own UpdateItem call and fell through
to writeInternalError on any non-stale-snapshot error. A concurrent
edit racing the invocation_slug partial unique index would surface as
500 instead of 409. Mirror the main UpdateItem error mapping.
Codex's other round-4 finding — the partial unique index applying to
"every collection" — is not addressed because the index IS already
collection-scoped: `ON items(collection_id, json_extract(fields,
'$.invocation_slug'))`. Two items in different collections with the
same slug coexist; only same-collection duplicates conflict. Migration
duplicates are impossible because `invocation_slug` is a brand-new
field key with no pre-existing items setting it. A custom collection
that later adopts the same field name opts into per-collection
uniqueness, matching the FieldDef.UniqueScope="workspace_collection"
semantic.
Parent: PLAN-1377.
|
||
|
xarmian
|
c2351d861d |
fix(ci): lower test-only bcrypt cost + re-enable -race on PRs (BUG-1371) (#513)
The full `internal/server` test suite under `-race` had grown past the 30m CI timeout, failing every push to main since ~TASK-1354. Diagnosis: bcrypt at the production cost (12) takes ~3s per call under the race detector, and dozens of tests now bootstrap a user via the loopback HTTP path (`bootstrapFirstUser` → `store.CreateUser` → `bcrypt.GenerateFromPassword`). Cumulative cost dominated the budget. Two coordinated changes: 1. Lower bcrypt cost in test binaries. `bcryptCost` becomes a package var (still package-private), and a new `SetBcryptCostForTesting` helper lets each test binary's `TestMain` drop it to `bcrypt.MinCost`. Production stays at 12 — only the test process ever mutates the value. 2. Re-enable `-race` on pull requests. The `if: github.ref == 'refs/heads/main'` gate was originally a GitHub Actions minutes cost-control; the repo is public now, so PR minutes are free, and we'd rather catch race regressions on the contributing branch than after merge. Measured impact: - `go test -race ./internal/server`: 1800s timeout → 830s (13m51s). - `go test ./internal/store`: 808s → 35s. - `go test ./internal/server`: 192s → 60s. The 30m timeout stays — it's headroom for genuine deadlocks, which would still hit the goroutine-dump panic the way BUG-851 did. Prior art: BUG-851 (10m → 30m bump, ipRateLimiter goroutine drain). This is a different cause (bcrypt cumulative time) so the fix is different. |
||
|
xarmian
|
bd8667eadf |
feat: seq-stamped SSE events + stale-event short-circuit (TASK-1358) (#505)
* feat: seq-stamped SSE events + stale-event short-circuit (TASK-1358)
## Server
- events.Event gains a `Seq int64` field (omitempty). Server populates
it on item lifecycle events so SSE consumers can reason about
ordering and contiguity against their /items-changes cursor.
- publishItemEventWithName takes seq as a parameter; all call sites
in handlers_items.go pass the item's current seq:
- item_created → item.Seq
- item_updated → updated.Seq
- item_archived → re-fetched via GetItemIncludeDeleted (DeleteItem
bumps seq but doesn't return the updated row)
- item_restored → restored.Seq
- move target → moved.Seq
## Web
- ItemEvent gains optional `seq` (matches the server's omitempty).
- localIndex.classifySSEEvent(ws, event) returns
'no-seq' | 'stale' | 'contiguous' | 'gap'. The collection page
uses this to short-circuit duplicate / replayed events the
server's replay buffer re-delivers after tab-resume. Non-stale
events still call deltaSync because the SSE wire payload only
carries metadata, not the row data; classify does NOT advance
the cursor (applyDelta with real row data is the only path that
does — preserving the IDB invariant from TASK-1356).
Parent: PLAN-1343.
* fix(events): version-restore item_updated event carries seq (Codex round 1)
Version-restore in handlers_item_versions.go was publishing the
item_updated event directly via events.Publish without Seq, bypassing
the new seq-stamped SSE contract from TASK-1358. localIndex's
classifySSEEvent would always return 'no-seq' for those events,
forcing a generic /items-changes refetch instead of allowing the
stale/gap fast paths.
Now passes updated.Seq from the store response.
Parent: PLAN-1343.
|
||
|
xarmian
|
a5b93c17c9 |
feat(api): add /items-changes?since=<seq> delta endpoint (TASK-1354) (#494)
* feat(api): add /items-changes?since=<seq> delta endpoint (TASK-1354) Adds the delta-fetch sibling of /items-index for the local-first read model (PLAN-1343 / DOC-1342 design decision #1). Clients track the workspace-scoped monotonic seq cursor returned by /items-index (TASK-1353) and poll /items-changes?since=<cursor> to apply just the rows that have mutated — without re-downloading the entire workspace. ## Endpoint GET /api/v1/workspaces/{ws}/items-changes?since=<seq>&limit=<n> - `since`: exclusive seq lower bound (returns `seq > since`). Defaults to 0 → full delta == /items-index modulo ordering. Bad input → 400. - `limit`: cap on rows. Defaults to 5000, clamped to 50000. Bad input → 400. ## Response { "changes": [...skinny rows with `deleted: bool`...], "cursor": "<decimal MAX(seq) or unchanged since when empty>" } Soft-deleted rows propagate (no `deleted_at IS NULL` filter on the backing scan) so a delta consumer can remove them from its local index without a second roundtrip. Parent metadata enrichment matches /items-index: the underlying GetItem filters soft-deleted parents so we never leak parent title/ref for an archived parent. Cursor contract: - Sorted ASC by seq → re-passing the response's cursor as `since` on the next poll is no-overlap, no-gap (strictly monotonic seq invariant from TASK-1352). - Empty response preserves the caller's `since` so position isn't lost. - Truncated-by-limit responses set cursor to the last row's seq. ## Tests - FullDeltaFromZero — three creates, since=0, ascending seq, every row deleted=false, cursor=MAX(seq). - IncrementalUpdateAndDelete — typical resume flow: snapshot cursor, mutate, delta returns exactly the mutated + tombstoned rows with the right `deleted` flag. - CursorRoundtripsCleanly — empty-poll after consuming, cursor preserved. - LimitTruncatesAndCursorResumes — paging contract holds end to end with no overlap. - InvalidParams — bad since / limit values rejected with 400. - EmptyWorkspace — cursor round-trips caller's since unchanged. ## Web TypeScript: `ItemChangeRow = ItemIndexRow & { deleted: boolean }`, `ItemChangesResponse = { changes, cursor }`. API client gains `api.items.changes(ws, sinceCursor, opts?)` with the same defensive content-strip as listIndex so a stray `content: ""` key from a Go zero-value can never clobber the canonical store. Parent: PLAN-1343. Depends on TASK-1352 (seq column) and TASK-1353 (seq cursor on /items-index). Unblocks the future client-side localIndex.applyDelta integration task. * fix(api): surface tombstones for item-grant users in /items-changes per Codex review (round 1) Codex round 1 caught that handleListItemsChanges was building its ItemIDs filter from guestResourceFilter, which itself uses GuestVisibleResources whose item-grant query filters out soft-deleted items. The result: a guest or restricted member with an item-level grant on a single item would see that ID disappear from the lookup as soon as the item was soft-deleted — and /items-changes would never emit a `deleted:true` tombstone, so the client would keep the stale row in its local index forever. Fix: - New Store.GuestVisibleResourcesIncludeDeleted that drops the `i.deleted_at IS NULL` / `c.deleted_at IS NULL` filters on both collection and item grants so tombstone IDs flow through. - New Server.guestResourceFilterIncludeDeletedItems delegate pointing at the new store helper. Implementation is shared with the live variant via guestResourceFilterCore so the member-collection-access + system-collection merge logic stays in one place. - handleListItemsChanges swaps to the include-deleted variant. Test: TestGuestVisibleResourcesIncludeDeleted_SurfacesTombstones covers both variants side-by-side — live drops the soft-deleted grant, include-deleted preserves it. * fix(store): assign per-row unique seqs in MigrateItemFieldValues per Codex review (round 2) Codex round 2 caught that the bulk UPDATE inside MigrateItemFieldValues gave every affected row the SAME MAX(seq)+1. A /items-changes?limit=N poll that cut through that equal-seq group would advance the cursor to the shared seq, and the next `seq > cursor` poll would silently miss the rest of the group — the cursor contract requires strict monotonicity. Switched to a per-row loop inside the migration transaction so every UPDATE re-reads MAX(seq) and each affected row ends up with a strictly unique seq. The workspace advisory lock makes the read-modify-write race-free on Postgres; SQLite's single-writer rule handles it implicitly. Trade-off: O(N) statements instead of O(1) for the bulk path. Option-rename is an admin one-off so the cost is acceptable (~1s/1000 rows on a warm SQLite connection). If future use cases demand a larger row budget, a single-statement UPDATE..FROM with ROW_NUMBER() CTE assigning per-row seqs would also work. Test: TestMigrateItemFieldValues_PerRowUniqueSeq confirms 5 rows in a single migration step all get unique seqs. |
||
|
xarmian
|
974472799a |
feat(api): wire real workspace seq into /items-index cursor + rows (TASK-1353) (#493)
* feat(api): wire real workspace seq into /items-index cursor + rows (TASK-1353)
Replaces the placeholder `updated_at`-derived cursor on the
/items-index response with the real workspace-scoped MAX(seq)
introduced by TASK-1352. Each returned row carries its own `seq`
field so clients can reason about ordering without parsing the
cursor.
When the requested scope returns zero rows but the workspace has
items (e.g. ?collection=docs on a workspace whose docs collection
is empty but whose tasks/ideas are not), the cursor falls back to
the workspace's true MAX(seq) via a new Store.MaxItemSeq helper.
That way the client's next /items-changes?since=cursor poll starts
at the right floor instead of replaying every prior mutation from 0.
Empty workspaces collapse to "0".
Encoding: cursor is the decimal-encoded MAX(seq). Treated as opaque
on the wire (clients re-pass it as ?since=). String form leaves
room to switch to base32/etc later without an API break.
TypeScript: `ItemIndexRow` (via `Item`) adds optional `seq?: number`;
`ItemIndexResponse.cursor` docstring updated to reflect the real
seq cursor semantics. `api.items.listIndex` docstring updated.
Tests:
- TestListItemsIndex_SkinnyProjectionAndShape: cursor now asserts
decimal-encoded MAX(seq); per-row seq is non-zero.
- TestListItemsIndex_EmptyResultFallsBackToWorkspaceMax: new test
covering the cursor fallback on filtered-but-empty results.
- TestListItemsIndex_CursorMonotonicAcrossMutations: new test
confirming cursor advances after every mutation.
Parent: PLAN-1343. Depends on TASK-1352 (seq column). Unblocks
TASK-1354 (/items-changes endpoint).
* fix(api): snapshot workspace MAX(seq) before list to close cursor race per Codex review (round 1)
Codex round 1 caught a real race in /items-index cursor computation:
ListItemsIndex ran first, then MaxItemSeq ran in a separate query.
A concurrent INSERT visible to a future /items-changes call could
land between them — the response would be `items: []` with cursor =
the new seq, and a subsequent /items-changes?since=cursor poll
(seq > cursor) would never return that row.
Fix: capture MaxItemSeq BEFORE the list query. Per the workspace's
monotonic counter invariant (TASK-1352) any insert after that
snapshot has seq > captured M, so /items-changes?since=M will see
it. Rows the list DOES observe may have seq > M (a concurrent
insert the list query happened to commit-snapshot); MAX(rows.seq)
bumps the cursor for that case so the client never re-fetches what
was already in the response.
Long-form comment on the handler captures the race scenario and the
invariant that makes the snapshot order safe.
|
||
|
xarmian
|
d6894def4f |
feat(web): collection page fetches via skinny /items-index endpoint (TASK-1349) (#491)
* feat(web): collection page fetches via skinny /items-index endpoint (TASK-1349)
Replaces every \`api.items.listByCollection(ws, coll)\` call in the
collection page with \`fetchSkinnyItems(ws, coll, includeArchived)\`,
which calls the local-first \`/items-index\` endpoint (TASK-1344)
through the typed client wrapper (TASK-1345). Items now ship
without the rich-text \`content\` body — the bulk of the per-row
wire size — until the user opens an item detail page, which still
goes through its existing full-item fetch.
Call sites updated:
- loadCollection — primary load + plans-names lookup
- SSE handler for item_created / item_archived / item_restored / item_updated
- Sync coordinator's full-refresh fallback
The skinny rows are widened to \`Item[]\` at the boundary by setting
\`content: ''\` on each row. This keeps the existing view component
type contract unchanged and means existing call sites that read
\`item.content\` see an empty string — already a "nothing to do"
sentinel in the markdown-checklist progress branch.
Documented regression — out of scope for this task: non-plans
collections used to display checklist progress derived from item
content's markdown checkboxes. With \`content\` no longer fetched
for the list view, that progress no longer appears. Plans
progress is unaffected (uses /plans-progress, not content
parsing). Re-introducing the feature requires either server-side
progress on the index endpoint or a separate lazy fetch — a
follow-up rather than a blocker for the bandwidth win.
In-scope behavior preserved:
- Item create/update flow: server still returns full items, dropped
into the array as-is; sync coordinator's incremental updates
similarly use the full-item type from the changes feed
- Server-side FTS search via \`searchResultIds\`: still id-keyed,
works against skinny rows
- List / Board / Table view components: already only read fields
present on the skinny row (title, fields, tags, sort_order…)
- Detail page fetch: unchanged — still goes through
\`api.items.get\` which returns the full Item with content
Parent: PLAN-1343.
* fix(api+web): add /collections/{coll}/checkbox-progress endpoint to preserve list-view checklist progress per Codex review (round 1)
Codex round 1 [P2] flagged that the original PR shipped a real
regression: non-plans collections used to compute markdown-checkbox
progress client-side from `item.content`, and the skinny
`/items-index` endpoint dropped `content` from the payload — so
list/board/table progress badges silently stopped appearing on
docs/tasks/custom collections.
This commit closes that gap with a new server endpoint that
computes the same `{item_id, total, done}` counts via
LENGTH/REPLACE arithmetic on the stored content, returning only
the small derived counts. No item bodies cross the wire.
Server (Go):
- `store.CollectionCheckboxProgress(workspaceID, collectionID)` —
SQL: `(LENGTH(content) - LENGTH(REPLACE(content, '- [ ]', '')))
/ 5 + (LENGTH(content) - LENGTH(REPLACE(content, '- [x]', '')))
/ 5` for total, the second clause alone for done. Same trick on
SQLite and PostgreSQL.
- `handleCollectionCheckboxProgress` — collection-visibility +
item-grant filter so guests / restricted members can't enumerate
items they shouldn't see. Mirrors `guestResourceFilter` exactly.
- Route: `GET /api/v1/workspaces/{ws}/collections/{coll}/checkbox-progress`.
- Test `TestCollectionCheckboxProgress` covers the math (open +
done counts), zero-result rows are filtered, unknown collection
→ 404, empty result → 200 + `[]`.
Web:
- `api.items.collectionCheckboxProgress(ws, coll)`
- Both call sites in `+page.svelte` (initial `loadCollection`
non-plans branch + `refreshProgress` non-plans branch) now
pull from the endpoint instead of parsing `item.content`.
- Drops the previous "documented regression" comment — the
feature is fully preserved.
Sub-100-byte response per item (vs. the full content body) so the
bandwidth win from `/items-index` is preserved. The endpoint scans
content server-side, but doesn't transmit it — the original
listByCollection call both scanned AND transmitted content.
Parent: PLAN-1343.
* fix(api+web): plumb include_archived through checkbox-progress per Codex review (round 2)
Codex round 2 [P2] caught that the Archived toggle path lost
checklist progress badges: `CollectionCheckboxProgress` hard-coded
`deleted_at IS NULL`, but the page-side fetch is called with the
same `showArchived` flag that toggles whether archived items
render. With the toggle on, archived non-plan items appeared in
the list but had no `itemProgress` row — the old client-side parse
would have counted them.
Fix: thread `includeArchived` through the call chain.
- store.CollectionCheckboxProgress(workspaceID, collectionID,
includeArchived bool) — appends `AND deleted_at IS NULL` only
when includeArchived is false. Default match the original
archived-off behavior.
- handleCollectionCheckboxProgress reads
?include_archived=true and forwards.
- api.items.collectionCheckboxProgress(ws, coll, { includeArchived })
on the client.
- +page.svelte's two call sites pass `showArchived` /
`includeArchived` exactly.
TestCollectionCheckboxProgress now archives one of the seeded
items and asserts:
- default response excludes the archived item (1 row)
- ?include_archived=true response includes it (2 rows)
Also clarified the const-doc on `checkboxCountSQL` to reflect the
dynamic deleted-at clause.
|
||
|
xarmian
|
e83e7ef413 |
feat(api): add /items/index skinny-projection endpoint (TASK-1344) (#486)
* feat(api): add /items/index skinny-projection endpoint (TASK-1344)
Foundation for PLAN-1343 (local-first read model). Adds a new
GET /api/v1/workspaces/{ws}/items/index endpoint that returns
every item in a workspace minus the rich-text `content` body,
so the client can hydrate an in-memory + IndexedDB index from a
single request and render every collection page from local
state without re-fetching.
Response: {items, total, cursor}. The cursor placeholder is the
max(updated_at) across the result set — Phase 2 replaces it with
a monotonic `seq` cursor. Sort is updated_at DESC, id ASC for
deterministic, cursor-friendly ordering.
Auth uses the same collection-visibility + item-grant filter as
handleListItems. Optional ?collection=<slug> filter for use by
collection pages. ?include_archived=true mirrors the existing
list behavior.
Parent: PLAN-1343.
* fix(api): move skinny-projection endpoint to /items-index per Codex review (round 1)
Codex round 1 [P2] flagged that the original `/items/index` path
shadowed the detail URL of any item whose slug is `index` — slugify
emits `index` for a title of "Index", and chi's static-over-wildcard
preference would route `GET /items/index` to the new index handler
instead of the existing `/items/{itemSlug}` detail handler.
Move the endpoint up to the workspace level as `/items-index`, sibling
to the existing `/plans-progress` route. Slugs cannot contain hyphens
adjacent to identifiers in a way that would collide with a static
workspace-level path, so this URL space is permanently safe.
New test `TestListItemsIndex_DoesNotShadowItemSlug` locks in the
contract: a real item titled "Index" still resolves through
`/items/{itemSlug}`, while `/items-index` returns the index wrapper.
|
||
|
xarmian
|
bdcb62e902 |
fix(api): accept nested object/array for PATCH items fields/tags (BUG-1144) (#485)
The PATCH /api/v1/workspaces/{ws}/items/{ref} endpoint previously
demanded `fields` and `tags` arrive as JSON-encoded strings, because
models.ItemUpdate declares them as *string to mirror the storage shape.
Sending the natural nested-object shape any reasonable HTTP client
would produce returned HTTP 400 with a leaked Go unmarshal error
naming the internal struct field — confusing for anyone integrating
against Pad over HTTP (webhook reactors, custom dashboards, non-CLI
agents, third-party MCP bridges).
This is the symmetric input-side counterpart to BUG-991, which was
fixed at the MCP boundary in PR #364 with dual-emit normalization
rather than the full Plan-sized models.Item migration.
Fix: add a custom ItemUpdate.UnmarshalJSON that accepts either shape
on the wire and normalizes to the canonical string internally. The
struct field type stays *string, so the validation/storage/web/CLI
pipeline is untouched. All in-process Go callers construct ItemUpdate
literals (15 grepped call sites) and never hit UnmarshalJSON, so the
change is invisible to them.
Wrong shapes (e.g. `{"fields":42}`, `{"tags":{"x":1}}`) now return a
domain-level 400 — `"fields" must be a JSON object or a JSON-encoded
string` — surfaced via sentinel errors (ErrInvalidFieldsType /
ErrInvalidTagsType) that the handler unwraps from decodeJSON's
"invalid JSON: %w" wrapper.
Coverage:
- models/item_test.go: 10 sub-tests covering object, array, string,
null, absent, and wrong-type cases for both fields and tags.
- server/handlers_items_test.go: 6 PATCH integration sub-tests
asserting back-compat, the BUG-1144 repro now returns 200, and
that error responses no longer leak Go struct field names.
Smoke-tested against the live server with the exact repro curl from
BUG-1144 (HTTP 200), plus malformed (HTTP 400 with clean message)
and stringified-string back-compat (HTTP 200).
|
||
|
xarmian
|
18087463ce |
feat(collab): op-log cursor protocol — force-refresh + watermark advance (TASK-1319) (#472)
* feat(collab): op-log cursor protocol — force-refresh + watermark advance (TASK-1319)
Closes both holes left by TASK-1309:
1. Long-disconnected tab + external-write race. A reconnecting client
announces its highest applied item_yjs_updates.id via `?since=<id>`.
If that id is below MIN(id) for the item, rows it expected to
replay have been pruned and the server sends a `force_refresh`
control frame and closes the conn. Client recreates the Y.Doc
and lazy-seeds from items.content. Without this, Tab A's stale
state would silently overwrite an external CLI/MCP write on the
next 5s flush.
2. Browser-only-edited items never GC'd. Browser collab-snapshot
PATCHes now carry an op_log_cursor body field. The store advances
items.content_flushed_op_log_id only when the cursor matches the
current MAX(op-log.id) — proving the markdown captures every
persisted op. SQL CASE clause re-evaluates MAX at COMMIT time so
a peer op landing between client-side cursor capture and the
UPDATE leaves the watermark untouched (no over-advancement).
Combined cursor mechanism:
- Server attaches op_log_cursor JSON control frames after replay,
after every successful AppendYjsUpdate (originator), and to every
peer's binary fan-out (so all peers stay in lockstep without a
round trip).
- Client persists per-tab in sessionStorage (NOT localStorage —
avoids cross-tab cursor leakage that would force-refresh stable
sessions).
- Server's MIN(id) check + force_refresh fires only when a non-zero
`since` is below MIN; `since=0` is treated as a fresh client.
New store methods: MinOpLogID, MaxOpLogID. New ItemUpdate field:
OpLogCursor *int64. New control message types: op_log_cursor,
force_refresh. New OpEvent.OpLogID for cursor piggyback. Existing
collab tests updated to drain TextMessage cursor frames.
Tests cover: initial cursor frame after replay (populated + empty
op-log), force_refresh fires when since<MIN, delta replay when
since>=MIN, cursor broadcast to originator + peers on append, and
watermark advancement gated on cursor==MAX.
Parent: PLAN-1248. Builds on TASK-1309.
* fix(collab): skip stale-Ydoc flush on force_refresh teardown per Codex review (round 1)
A force_refresh tear-down means the local Y.Doc cursor is below the
server's MIN(item_yjs_updates.id) — its derived markdown is stale.
Without this guard the collab $effect cleanup runs flushCollabNow
on the way out and silently PATCHes that stale markdown back to
items.content, overwriting the canonical content the fresh provider
is supposed to lazy-seed from. Per Codex round 1 [P1] of TASK-1319.
* fix(collab): force_refresh on empty op-log + cancel pending flush per Codex review (round 2)
Two P1 fixes:
1. Manager.Join now force_refreshes when since>0 and the op-log is
empty (hasMin==false), not just when since<MIN. After
PruneAndApply wipes the entire op-log, MIN is undefined; the
original predicate would have admitted the stale tab and let its
on-open Y.encodeStateAsUpdate write resurrect the pre-prune
document.
2. The +page.svelte onForceRefresh handler now also clears
collabFlushTimer. Without this a 5s timer that armed before the
force_refresh frame arrived can still fire AFTER the cleanup
ran, PATCHing stale Y.Doc-derived markdown to items.content.
New test: TestRoomManagerForceRefreshOnEmptyOpLogWithSince covers
the empty-op-log branch.
Per Codex round 2 [P1] of TASK-1319.
* fix(collab): include forceRefreshNonce in Editor key so it remounts on force_refresh per Codex review (round 3)
The collab $effect cleanup runs on forceRefreshNonce bump, but the
<Editor> {#key} was `${item.id}:true` — itemID doesn't change, so
the keyed Editor wasn't unmounting. The Tiptap Collaboration
extension only binds in onMount, so the editor stayed wired to the
stale (destroyed) Y.Doc while a fresh provider+doc were set up
in parallel. Edits would either be unsynced or eventually flush
stale markdown again.
Adding forceRefreshNonce to the key forces the Editor to remount
in lockstep with the doc swap. Per Codex round 3 [P1] of TASK-1319.
* fix(collab): refetch item.content before lazy-seed on force_refresh per Codex review (round 4)
After force_refresh the collab $effect rebuilds the Y.Doc and the
lazy-seed (TASK-1261) seeds it from item.content. But item.content
was the cached page-state copy — possibly stale relative to the
server (the WS force_refresh can beat the SSE/visibility refresh
that would otherwise update it). Lazy-seeding stale content into
a fresh op-log re-introduces exactly the staleness force_refresh
was supposed to clear: the next 5s flush PATCHes that stale view
back to canonical items.content.
onForceRefresh now does an api.items.get() before bumping the
nonce so the rebuild's lazy seed reads server-fresh content. A
failed fetch falls through to the bump anyway (an editor on
possibly-stale content is still better than a broken editor).
Per Codex round 4 [P1] of TASK-1319.
* fix(collab): suppress cursor during replay + move force_refresh check before getOrCreate per Codex review (round 5)
Two more findings:
1. [P1] writeLoop sends op_log_cursor frames for live ops broadcast
during the replay window. A client disconnecting after one of
those cursors lands but BEFORE the rest of replay completes
would persist a cursor pointing past unreplayed rows. On
reconnect with since=that-cursor, server replays nothing — the
client's Y.Doc would be missing causally-required ops.
Fix: per-roomConn replayDone atomic.Bool. writeLoop suppresses
cursor frames while it's false. runConn flips it after the
post-replay initial cursor is on the wire. Live binary frames
continue to flow during replay (Yjs CRDT commutativity); only
the cursor metadata is gated.
2. [P2] Force-refresh path leaked an empty room. getOrCreate
inserted into m.rooms before the force_refresh bail-out left
an orphan entry that PruneSweep would later treat as 'active'
and skip indefinitely.
Fix: schema-rebuild + force_refresh checks now run BEFORE
getOrCreate. Both are store-only mutations and the per-item
lock is held throughout, so concurrency is unchanged.
New test: TestRoomManagerCursorSuppressedDuringReplay regression-
guards the cursor-suppression behaviour.
Per Codex round 5 [P1+P2] of TASK-1319.
* fix(collab): tighten initial cursor + sync-destroy provider on force_refresh per Codex review (round 6)
Two more P1 fixes:
1. runConn's empty-replay fallback used MaxOpLogID() to anchor
the initial cursor. A live op landing between replayTo
returning and the cursor write would be reflected in MAX
but its binary frame might not have flowed through this
conn's writeLoop yet — the cursor would advertise an id
the client hasn't received. Initial cursor is now strictly
max(highestReplayed, since); MaxOpLogID is removed from
the opLogStore interface.
2. Provider.handleControlMessage's force_refresh branch now
calls this.destroy() SYNCHRONOUSLY before invoking the
onForceRefresh callback. Previously the consumer's recovery
path (async items.get refetch) would race the provider's
own onClose-triggered reconnect, which would re-open with
since=0 and push Y.encodeStateAsUpdate of the stale Y.Doc
— recreating the corruption force_refresh was meant to
prevent. destroy() sets destroyed=true so scheduleReconnect
short-circuits.
Per Codex round 6 [P1] of TASK-1319.
* fix(collab): block flush scheduling during force_refresh recovery per Codex review (round 7)
Previously, after onForceRefresh fires:
1. Provider is destroyed synchronously.
2. Async items.get refetch is in flight.
3. forceRefreshNonce bumps after refetch resolves.
4. $effect cleanup runs, then rebuild.
But during steps 2-3 the editor component is still mounted with
the stale Y.Doc, and a local edit fires handleContentUpdate which
calls scheduleCollabFlush. clearTimeout earlier in onForceRefresh
only canceled the timer at THAT moment; a new edit during the
refetch window arms a fresh timer that fires before cleanup. That
PATCHes stale Y.Doc-derived markdown back to canonical content,
recreating the corruption force_refresh was meant to prevent.
Fix: forceRefreshInFlight flag set in onForceRefresh, blocks
scheduleCollabFlush, resets after the fresh provider is wired
(end of $effect run). Per Codex round 7 [P1].
* fix(collab): gate runCollabFlush itself on force_refresh in-flight per Codex review (round 8)
scheduleCollabFlush blocked the 5s timer path, but direct callers
of flushCollabNow / runCollabFlush (beforeunload handler,
rich-to-raw toggle) bypassed the guard. A page reload or raw
toggle DURING the force_refresh recovery window still PATCHed
stale Y.Doc-derived markdown to canonical items.content.
Pulling the guard into runCollabFlush covers every caller in one
spot and returns 'deduped' so the result-shape contract holds.
Per Codex round 8 [P1] of TASK-1319.
* fix(collab): distinct 'skipped' result for force_refresh path; raw-toggle aborts per Codex review (round 9)
runCollabFlush returning 'deduped' on the force_refresh-blocked
path was indistinguishable from a legitimate same-content dedupe.
The rich→raw toggle treats 'deduped' as 'server already has this
markdown' and seeds rawSeedMarkdown from it — letting the user's
next raw edit overwrite canonical items.content with content
derived from the stale Y.Doc.
Add a distinct 'skipped' result for the force_refresh path. Raw
toggle aborts on it (with a 'try again in a moment' toast); other
callers fall through unchanged because no other call site
behaviorally depends on 'deduped' vs 'skipped'.
Per Codex round 9 [P1] of TASK-1319.
* fix(collab): server-side gate + post-await client guard against stale collab-snapshot per Codex review (round 10)
A force_refresh frame can arrive WHILE a collab-snapshot PATCH is
already mid-flight to the server. The client-side
forceRefreshInFlight check at PATCH-start can't catch this race;
the request lands at the server with stale Y.Doc-derived markdown.
Two-pronged fix:
1. Server: handler now checks op_log_cursor against MIN(op-log.id)
for collab-snapshot PATCHes and returns 409 Conflict when
cursor < MIN. Such cursors prove the flushing tab's Y.Doc was
built on rows that have been pruned (PruneAndApply, schema
rebuild, dormant GC). The markdown is, by construction, stale.
2. Client: post-await check on forceRefreshInFlight returns
'skipped' instead of 'flushed' so saveStatus / lastFlushedContent
don't seed from a known-stale base even if the server happened
to accept the PATCH (e.g. MIN advanced after handler validation).
New tests: TestCollabSnapshotRejectsCursorBelowMin (gate fires),
TestCollabSnapshotAcceptsCursorAtOrAboveMin (negative path).
Also de-leak an unused slice in the round-5 cursor-suppression test
so staticcheck stays clean.
Per Codex round 10 [P1] of TASK-1319.
* fix(collab): reject collab-snapshot when cursor>0 and op-log empty per Codex review (round 11)
The HTTP-layer gate I added in round 10 mirrored only PART of the
WS-upgrade force_refresh predicate. Round 5 had already taught us
that 'op-log entirely pruned' is a separate stale path from
'cursor below MIN' (PruneAndApply, schema rebuild, dormant GC all
leave hasMin=false), and the WS check now uses
`since > 0 && (!hasMin || since < minID)`. The HTTP gate had
only the second clause.
Mirror the WS predicate at the handler so a stale collab-snapshot
PATCH against an empty op-log gets a 409 too. New regression:
TestCollabSnapshotRejectsCursorOnEmptyOpLog.
Per Codex round 11 [P1] of TASK-1319.
* fix(collab): reject collab-snapshot cursor=0 on non-empty op-log per Codex review (round 12)
Round-11 gate accepted cursor=0 unconditionally. But a stateful tab
whose previous session disconnected BEFORE receiving the
post-replay cursor frame (network blip during the writeMu burst
between replay binaries and the cursor) ends up with sessionStorage
cursor=0 + a non-empty Y.Doc populated by prior replay binaries.
On reconnect with since=0 the server treats it as fresh, replays
nothing if the op-log was meanwhile pruned, and the client's
on-open Y.encodeStateAsUpdate resurrects pre-prune ops. The next
flush carries cursor=0 + stale-derived markdown.
The gate now refuses any incompatible cursor:
- cursor>0 + empty op-log (prior rule)
- cursor<MIN + non-empty op-log (prior rule, now naturally
catches cursor=0 too because 0 < any positive MIN)
The WS replay path is unchanged — full replay from since=0 is
the recovery for clients that genuinely lost their cursor; the
corruption manifested through the flush PATCH which we now gate.
New test: TestCollabSnapshotRejectsCursorZeroOnNonEmptyOpLog.
Per Codex round 12 [P1] of TASK-1319.
* fix(collab): close cursor=0 client/server gaps + lock validation+write atomically per Codex review (round 13)
Four P1 issues addressed:
1. Client always sends op_log_cursor (including 0) so the server
gate sees the field. Previously cursor=0 was omitted, which
silently bypassed the server's stale-snapshot rejection.
2. Provider construction now resets sessionStorage cursor to 0
when the Y.Doc is empty. The Y.Doc isn't persisted across
page reload, so a stored cursor=N + fresh empty Y.Doc would
announce since=N to the server and miss rows 1..N from
replay (server only replays id > N).
3. onOpen skips Y.encodeStateAsUpdate when lastOpLogID === 0.
A populated Y.Doc + cursor=0 is the network-blip-during-cursor-
write failure mode; pushing that state can resurrect ops the
server has pruned. Server replay + lazy-seed handle recovery
without our push.
4. Server gate now runs INSIDE the per-item collab setup lock
(new RoomManager.UnderItemLock helper) so a concurrent prune
(PruneAndApply, schema rebuild, dormant GC) cannot land
between the MIN check and the items.content write. Without
this, a tight race let stale snapshots overwrite canonical
content the prune just installed.
Per Codex round 13 [P1] of TASK-1319.
* fix(collab): gate handleDocUpdate on cursorAnchored to close stale-Ydoc edit path per Codex review (round 14)
Round 13 fix skipped on-open send for lastOpLogID===0, but local
edits via handleDocUpdate still propagated. A populated Y.Doc +
no-cursor-yet client could type, the edit would land in the
op-log with id N, server would send originator cursor=N, and
the next 5s flush would carry an 'anchored' cursor that passed
the server's MIN check — overwriting items.content with stale-
Y.Doc-derived markdown.
Add a cursorAnchored boolean. Set on first op_log_cursor frame
receipt (including cursor=0 against an empty op-log — that's a
legitimate 'server has nothing' signal). handleDocUpdate refuses
to send before this. Local edits buffer in the editor; once the
cursor arrives (or force_refresh rebuilds the provider), the
existing reconnect/edit paths catch them up.
Per Codex round 14 [P1] of TASK-1319.
* fix(collab): buffer + flush pre-anchor local updates per Codex review (round 15)
Round 14 silently dropped local Yjs updates fired before the
first op_log_cursor frame anchored the session. Yjs updates are
incremental: a dropped keystroke leaves later ops referencing
structs no peer can resolve, breaking convergence.
Buffer pre-anchor updates in a Uint8Array[] (capped at 1000 to
prevent unbounded growth in pathological 'anchor never arrives'
scenarios — overflow triggers force_refresh-style recovery).
On the first cursor frame, flush the buffer in order so the
server gets every causally-required struct before any post-
anchor updates land.
Per Codex round 15 [P1] of TASK-1319.
* fix(collab): destroy provider before force_refresh on pre-anchor buffer overflow per Codex review (round 16)
Round 15 overflow path called onForceRefresh but didn't destroy
the provider synchronously. A late op_log_cursor arriving before
the page-level rebuild (the recovery callback is async — refetches
items.content) would flip cursorAnchored=true, the partially-
populated buffer would flush, but the DROPPED prefix (the
overflowed entries) would leave server-side ops causally
incomplete — exactly the bug the buffer was supposed to prevent.
destroy() sets destroyed=true, removes message listener,
short-circuits scheduleReconnect, closes the socket. Late cursor
frames can no longer anchor a doomed provider.
Per Codex round 16 [P2] of TASK-1319.
* fix(collab): refuse rebuild on refetch fail + broaden on-open gate to cursorAnchored per Codex review (round 17)
Two findings:
[P1] force_refresh recovery bumped forceRefreshNonce in finally
even when the item.content refetch failed. The rebuild then
lazy-seeded from the cached (possibly-stale) item.content, and
the next flush would PATCH that stale view back to the server.
Move the bump into .then() so a failed refetch surfaces a
'please reload' toast and leaves the editor effectively
read-only (forceRefreshInFlight stays true, blocking flushes).
[P2] Send-on-open gate was lastOpLogID > 0, which silently
dropped local edits made during a brief offline window after a
legitimate 'cursor=0' anchor (empty op-log session). Switch to
cursorAnchored — the boolean specifically distinguishes
'unanchored' (stale Y.Doc + no server confirmation) from
'anchored at cursor=0' (legitimate empty op-log).
Per Codex round 17 [P1+P2] of TASK-1319.
* fix(collab): force_refresh on cursor=0 against non-empty Y.Doc per Codex review (round 18)
cursor=0 means the server's op-log is currently empty. A
non-empty Y.Doc at first-cursor receipt implies the ops came
from an earlier connection within this provider's life that
never reached its post-replay cursor frame, followed by a
server-side prune (PruneAndApply, schema rebuild, dormant GC)
during our disconnect. Anchoring at cursor=0 in that state
would mark a stale Y.Doc as authoritative; the next on-open
state push or flush would resurrect pre-prune state and
overwrite canonical items.content.
Detect the configuration via Y.encodeStateVector length and
invoke the same force_refresh-style recovery the explicit
server frame triggers: destroy provider, clear sessionStorage,
fire onForceRefresh so the page rebuilds from items.content.
Per Codex round 18 [P1] of TASK-1319.
* fix(collab): gate cursor=0 force_refresh on remoteSyncApplied per Codex review (round 19)
Round 18 force_refreshed the provider whenever cursor=0 arrived
against a non-empty Y.Doc. But local pre-anchor edits (user typed
before the initial cursor=0 of a legitimate empty-op-log session
arrived) ALSO populate Y.Doc — yet those edits live in
preAnchorUpdates and were supposed to flush on anchor. The
predicate spuriously triggered force_refresh, dropping the
buffered local edits.
Track remoteSyncApplied (set when readSyncMessage applies
anything to Y.Doc — replay binary or live peer op). Only force_
refresh on cursor=0 when remoteSyncApplied is true: that's the
true 'remote replay landed but server now reports empty op-log
=> mid-session prune' signature.
Per Codex round 19 [P1] of TASK-1319.
* fix(collab): repair brace mis-merge in wsProvider cursor=0 guard
The round-19 patch overlapped the round-18 inner block, producing
an extra brace + over-indented body. Collapsing into a single
clean block restores parseability without changing semantics
beyond what round 19 already documented.
* fix(collab): gate syncStep2 reply on cursorAnchored per Codex review (round 20)
readSyncMessage writes an inline syncStep2 reply when it receives
a peer's syncStep1. That reply embeds our current Y.Doc state.
If a peer's syncStep1 arrives before our first op_log_cursor
(pre-anchor window), the reply path bypasses handleDocUpdate's
cursorAnchored gate and lets potentially-stale Y.Doc state reach
the server before the cursor=0 + remoteSyncApplied force_refresh
recovery has a chance to fire.
Suppress the reply while unanchored. Peer state propagation
still works: the buffered preAnchorUpdates flush on anchor, and
the lazy-seed rebuild after a force_refresh seeds canonical
content from items.content.
Per Codex round 20 [P1] of TASK-1319.
* fix(collab): fold mid-replay live op ids into post-replay cursor + remoteSyncApplied only on apply per Codex review (round 21)
Two more findings:
[P1 server] writeLoop suppresses cursor frames during replay to
prevent the client persisting a cursor past unreplayed rows.
But binary frames for those live ops still go through
(commutativity), so the client APPLIES them to its Y.Doc. The
post-replay initial cursor only covered max(highestReplayed,
since), leaving the cursor below the highest applied op. On
empty-replay sessions this trips the client's
'cursor=0 + remoteSyncApplied' force_refresh path and discards
buffered pre-anchor edits.
Track maxLiveOpLogIDDuringReplay on the roomConn (atomic
compare-and-swap) and fold it into the post-replay cursor.
[P1 client] remoteSyncApplied was set on every MESSAGE_SYNC,
including syncStep1 (which only carries a state vector — it
doesn't apply state). A peer's syncStep1 arriving pre-anchor
would falsely flag remote-sync-applied and trip the cursor=0
force_refresh on legitimate empty-op-log sessions. Set the
flag only after readSyncMessage returns, and only for
syncStep2 / update subtypes.
Per Codex round 21 [P1] of TASK-1319.
* fix(collab): widen writeMu critical section + drop omitempty on op_log_id per Codex review (round 22)
Two more P1s:
[P1 server] writeLoop's mid-replay record-max happened OUTSIDE
writeMu, so runConn's post-replay read could race the record:
runConn loads → writeLoop's atomic store of higher value →
runConn sends cursor below the live id. Move the entire
per-event sequence (binary write + replayDone observation +
record-or-send) inside writeMu, and have runConn acquire
writeMu around its read+cursor-write+replayDone-flip. The lock
serializes the two paths cleanly: writeLoop events that ran
first have already recorded; events that arrive after replayDone
flips emit their own cursor frames.
[P1 protocol] OpLogID had `omitempty` JSON tag — a legitimate
cursor=0 (empty op-log session) serialized as
`{"type":"op_log_cursor"}` with no op_log_id field. The
client's strict-type check then rejected it as malformed,
leaving the session unanchored and local edits buffered
forever. Drop omitempty so 0 is wire-visible. Other control
types (applier_request/ack) carry an extra op_log_id:0 in
their JSON, which their client dispatches ignore.
Per Codex round 22 [P1] of TASK-1319.
* fix(collab): route originator cursor through writeLoop FIFO per Codex review (round 23)
readLoop sent the originator's op_log_cursor directly via
sendOpLogCursor right after AppendYjsUpdate, bypassing the bus/
writeLoop ordering. With a peer op already queued in rc.bus, the
sequence on the wire could be:
1. originator cursor=N (newer local op)
2. peer binary (older op)
3. peer cursor=M < N (rejected by client's max-take logic)
Client persists cursor=N. If the client then disconnects before
applying the peer binary, reconnect with since=N replays nothing
(server has nothing > N) and the older peer op is lost forever
to this client's Y.Doc.
Fix: writeLoop now processes self events too — skipping the
binary echo (the originator already has Y.Doc state) but routing
the cursor frame through the same FIFO bus channel as peer ops.
The originator's cursor=N now arrives strictly AFTER all
older-id peer events on the same channel.
Per Codex round 23 [P1] of TASK-1319.
|
||
|
xarmian
|
028db39217 |
feat(collab): periodic op-log GC sweeper for dormant items (TASK-1309) (#471)
The Yjs collab dumb-relay accumulates op-log rows indefinitely in item_yjs_updates. DOC-1307 surfaced 45-second p50 cold-reconnect latency on a single item with 5000 accumulated rows. Without GC, busy items keep growing. This adds a periodic background sweeper that prunes the entire op-log for items that are both DORMANT (no recent activity) AND FULLY FLUSHED (items.content has captured every op-log row). Whole-log only — Yjs op streams are causally linked, prefix-pruning corrupts replay; future cold connects lazy-seed from items.content. Components: - Store.ListDormantOpLogItemsBefore (joins items, filters watermark) - Store.PruneItemOpLogIfDormantBefore (atomic conditional DELETE) - Store.GetItemContentFlushedOpLogID (per-item watermark getter) - RoomManager.PruneSweep (per-item-locked, active-room-skip) - Server.StartOpLogGC / stopOpLogGC (mirrors orphan_gc.go pattern) - cmd/pad/main.go env vars PAD_OPLOG_GC_INTERVAL / PAD_OPLOG_GC_MIN_AGE - New (item_id, created_at) index for the dormancy query - New items.content_flushed_op_log_id column (id-based watermark, monotonic, no clock-skew or second-granularity false positives) + content_flushed_at (informational timestamp) Watermark policy: - Server-driven full-content writes (CLI / MCP / version restore / PruneAndApply) advance content_flushed_op_log_id to MAX(op-log.id) via subquery, atomic with the content UPDATE - Browser collab-snapshot 5s flushes do NOT advance the watermark — they can't prove their markdown captured every peer's ops, so letting them stamp would risk later GC-pruning unsynced peer edits - Schema-mismatch rebuild (TASK-1268) logs a WARN when it drops unflushed ops (data loss is unavoidable on schema bumps but visible) Stop ordering: collab.Close() now runs BEFORE bg.Wait() so a GC goroutine waiting on an itemLock behind an active Join can drain. Migration backfill: items WITH existing op-log rows keep NULL watermark (don't certify); items WITHOUT op-log rows get a synthetic 0 watermark (vacuous, harmless — no rows to compare against). Tests: - 6 RoomManager.PruneSweep tests (dormant prune / default minAge / empty / bails-on-Close / skips-active-room / skips-row-added-mid- sweep via fakeOpLog hook) - 5 Server.OpLogGC tests (prunes-dormant / start-idempotent / preserves-unflushed / backfill-doesnt-certify-unflushed / no-collab-noop) - TestCollabSnapshotDoesNotAdvanceOpLogWatermark in store - TestCollabSnapshotQueryOverridesBodyVersionSource in server (regression for body-attacker bypass) Seven rounds of Codex review — caught 5 P1s and 4 P2s I would have shipped under self-review: 1. Prefix-prune corrupts Yjs replay 2. Stop ordering deadlock 3. Missing index 4. Best-effort flush ⇒ data loss 5. Backfill over-certifies via metadata-PATCH 6. Second-granularity timestamp comparison 7. Schema-mismatch path drops unflushed silently 8. Browser flush stamps watermark beyond Y.Doc 9. Body version_source bypasses server policy |
||
|
xarmian
|
9b46be915a |
feat(collab): schema-version handshake + mismatch rebuild (TASK-1268) (#466)
Adds a client→server schema-version handshake on every WS connect and a per-item op-log rebuild path for the case where the server ships a new SCHEMA_VERSION and finds older rows persisted in the op-log. Client side - New web/src/lib/collab/schemaVersion.ts exporting `SCHEMA_VERSION` (currently '1') with a documented bump rule covering Tiptap extension changes, coordinated multi-package bumps, and Y.Doc fragment-shape changes. - wsProvider's defaultCollabUrl appends ?schema_version=... Server side - handlers_collab.go validates ?schema_version against RoomManager.SchemaVersion() BEFORE upgrading the WS; mismatch returns HTTP 400 with code "schema_mismatch". An empty query is treated as legacy '1' for graceful deploys; once the server bumps past v1, missing query becomes a 400 too. - New RoomManager.SchemaVersion() getter. - RoomManager.Join's setup-phase (under itemLock) now calls maybeRebuildOnSchemaMismatch: if the latest persisted op-log row's schema_version disagrees with the manager's current version, the entire item op-log is pruned via PruneYjsUpdatesBefore. items.content is canonical and untouched, so the lazy-seed path (TASK-1261) re-encodes it into ops at the new schema on the next idle tick. - New store method LatestYjsUpdateSchemaVersion. Tests - internal/collab/manager_test.go: three new tests (mismatch prunes, clean version preserves op-log, post-rebuild connects are clean) + fakeOpLog gets LatestYjsUpdateSchemaVersion + PruneYjsUpdatesBefore. - internal/server/handlers_collab_test.go: rejects-schema-mismatch (400), accepts-explicit-match (101). One round of Codex review (CLEAN with two NITs, both fixed). |
||
|
xarmian
|
191b887e23 |
feat(versions): VersionSource attribution + collab coexistence (TASK-1267) (#465)
The collab 5s-flush PATCH (TASK-1260) sends
`?source=collab-snapshot` with a body of just `{ content }`. Without
a handler-side stamp, `Store.UpdateItem`'s default coerced empty
input.Source to "web" on the version row and the per-(actor, source)
throttle suppressed every collab-driven snapshot following the user's
last manual web edit — version-diff effectively went silent during
co-edit sessions.
Adds:
- ItemUpdate.VersionSource: overrides per-version-row Source
attribution WITHOUT mutating items.source. The latter feeds
WorkspaceHasAgentActivity's `source IN ('cli', 'mcp')` filter,
so a CLI/MCP-created item the user opens in the editor would
otherwise silently flip out of the agent-activity tally on every
auto-flush.
- Store.UpdateItem prefers VersionSource over Source for version
row creation; falls back to Source then "web" if neither set.
- handlers_items.go stamps `input.VersionSource = "collab-snapshot"`
for `?source=collab-snapshot` PATCHes (when not already set).
Tests:
- internal/store/items_collab_versions_test.go: store-level
reverse-patch reconstruction over a CLI→web→collab-snapshot
edit sequence; verifies IsDiff=true on at least one row.
- internal/server/handlers_items_collab_versions_test.go: full
HTTP-level test of the route; asserts a collab-snapshot version
row is created AND that items.source stays "cli".
Four rounds of Codex review.
|
||
|
xarmian
|
fdc6b221d0 |
test(collab): regression guard for share-page collab isolation (TASK-1266) (#464)
The public /s/{token} share page must keep rendering markdown via
marked + DOMPurify and never open a WebSocket to /api/v1/collab —
anonymous viewers can't authenticate, and exposing per-item Y.Doc
traffic to the public internet would be a security regression.
This is a verification task: zero production code changes. Adds
TestSharePageDoesNotImportCollab in internal/server which:
- Walks the import closure rooted at the share-page +page.svelte
(resolves $lib/ aliases, relative paths, conventional extensions,
static AND dynamic `import(...)` forms; skips external packages)
- Asserts no file in the closure contains forbidden tokens:
wsProvider, CollabProvider, @tiptap/extension-collaboration,
@tiptap/y-tiptap, 'yjs' / "yjs", y-protocols, Editor.svelte
(suffix), WebSocket, /api/v1/collab
- Asserts the route file still imports + invokes marked and
DOMPurify.sanitize (catches a renderer swap)
- Strips comments before all checks so leftover commented-out
imports can't bypass
Verified by injection: direct AND transitive AND dynamic-import
regressions all fail the test.
Four rounds of Codex review.
|
||
|
xarmian
|
9b1a91ab00 |
feat(collab): 5s-idle + on-disconnect markdown flush (TASK-1260) (#458)
* feat(collab): 5s-idle + on-disconnect markdown flush (TASK-1260) Replaces the temporary handleContentUpdate suppression introduced in TASK-1259 (PR #457) with a proper flush mechanism. Under collab, the Y.Doc + op-log are canonical for live state but items.content needs to stay reasonably fresh for downstream consumers (search index, share-page, exports, plain API readers). ## Mechanism 1. **5s idle timer.** Every editor onUpdate (local OR remote) resets a 5s timer. On fire, PATCHes items.content via the new `?source=collab-snapshot` query param. 2. **Server-side bypass.** handleUpdateItem inspects the source query param. When set, skips the applyContentViaCollab routing entirely and writes directly. Without the bypass, the PATCH would loop back through the applier protocol (the same tab gets asked to apply, acks, server strips input.Content) and leave items.content unchanged forever. The flag is trustworthy because the caller already has edit access. 3. **Dedupe across peers.** Track lastFlushedContent. If our last successful flush already landed this exact markdown, skip the PATCH. Multiple connected tabs would otherwise each fire a redundant flush after every shared edit converges, multiplying server load by the peer count. 4. **On-disconnect flush.** $effect cleanup (item swap or page unmount) calls flushCollabNow(true) BEFORE provider.destroy(). A separate beforeunload listener catches close-tab / reload / external-nav. Both use fetch keepalive: true so the request outlives the page lifecycle. 5. **Item-id race guards.** runCollabFlush captures reqItemId before await; ignores response if item swapped. loadData() clears collabFlushTimer + lastFlushedContent on navigation. ## Files - internal/server/handlers_items.go — accept `?source=collab-snapshot` - web/src/lib/api/client.ts — add api.items.flushCollabContent - web/src/routes/.../[slug]/+page.svelte — handleContentUpdate gains scheduleCollabFlush + runCollabFlush + flushCollabNow; wired to $effect cleanup + beforeunload + loadData reset. Parent: PLAN-1248 * fix(collab): capture ws+itemId at provider mount + apply unescapeDocLinks per Codex review (round 1) Two findings from round 1: 1) [P1] runCollabFlush resolved item.id and wsSlug at execution time, not at schedule time. During item navigation the timer could fire (or $effect cleanup could run) AFTER `item` was already updated to the new item, causing the OLD editor's markdown to be PATCHed against the NEW item's URL — cross-item content corruption. Fix: introduce activeCollabContext = { wsSlug, itemId }, captured at $effect-body time (when the provider is minted). scheduleCollabFlush, runCollabFlush, and flushCollabNow all take their target identity from this captured context, never from live reactive state. Cleared in the $effect's own cleanup (defensive `=== ctx` slot guard so a fast-navigation churn doesn't clobber a successor context). 2) [P2] The disconnect flush read raw editor.storage.markdown .getMarkdown() without unescapeDocLinks, unlike the regular onUpdate path. Closing/navigating before the idle flush could persist escaped wiki links like \[\[TASK-1\]\] which then wouldn't be converted by markdownToWikiLinks. Fix: apply unescapeDocLinks() at the start of runCollabFlush (covers both the timer-driven idle path and the unmount path). * fix(collab): gate UI mutations on foreground+current-item per Codex review (round 2) [P2] runCollabFlush mutated page-scoped state (saveStatus, editorStore.lastSaveTime, lastFlushedContent) before checking whether the captured itemId still matches the foreground item. On item navigation, the cleanup-driven keepalive flush could stamp 'saving' onto the NEW page's saveStatus, leaving it pinned indefinitely (and pollute lastFlushedContent for the new item's dedupe state). Fix: introduce isForegroundCurrent() = !keepalive && item.id === itemId. Gate saveStatus / setLastSaveTime / showSaved on it so background cleanup flushes never touch UI state. Gate lastFlushedContent on item.id === itemId regardless of keepalive so a stale flush can't seed the wrong item's dedupe. * fix(collab): skip cleanup flush on rich→raw transition per Codex review (round 3) [P1] $effect cleanup fires the keepalive flushCollabNow on every provider teardown, including rawMode toggles. The raw-button onclick already pre-populated rawPendingMarkdown with the live editor markdown (which the 1.2s raw debounce will land), so the keepalive PATCH from cleanup is redundant — and worse, can land AFTER the raw save and clobber newer raw edits with the older Y.Doc snapshot. Fix: gate the cleanup flush on `!rawMode`. If rawMode is true at cleanup time, the user just toggled to raw and the raw-mode codepath owns items.content from here. The other cleanup triggers (item nav, canEdit flip, page unmount) all keep firing the flush as before. Note: rawMode === true at cleanup time unambiguously means "transitioning into raw" — the inverse case (already in raw and the cleanup fires for some other reason) is impossible because collabKey gates on !rawMode, so the provider $effect never runs while rawMode is true. * fix(collab): synchronously flush Y.Doc state on rich→raw toggle per Codex review (round 4) [P1] Rich → raw → navigate-without-typing-or-toggling-back never PATCHed the live Y.Doc state to items.content. The previous seed mechanism only set rawPendingMarkdown, which only fires the 1.2s debounce on a subsequent handleRawContentUpdate call — which never happens if the user doesn't type. Fix: await runCollabFlush(ws, itemId, md, true) inside the raw button's async onclick BEFORE flipping rawMode = true. This: - Lands items.content with the live Y.Doc state synchronously (one PATCH, awaited, with keepalive: true so it survives a fast post-toggle navigation). - Seeds lastFlushedContent so any cleanup-driven re-flush is deduped. - Avoids populating rawPendingMarkdown — the raw debounce now only fires for actual user edits in raw mode, eliminating the race where a stale debounce fired after navigation could clobber state. The Round 3 cleanup-skip on rawMode is kept as defense-in-depth (also makes the no-op-when-already-flushed semantics explicit). * fix(collab): loop-flush until stable + cancel timer on rich→raw toggle per Codex review (round 5) Two HIGH findings from round 5: 1) Round 4's single-flush captured md BEFORE the await; concurrent peer edits (e.g. same user's other tab) during the await were lost from the seed and could be overwritten by subsequent raw-mode saves. Fix: loop-flush until stable. Re-read editor markdown after each PATCH; if it changed, flush again. Capped at 3 iterations to bound the transition under aggressive concurrent typing. 2) An onUpdate during the await could schedule a 5s collab flush timer that survived the rawMode flip. The cleanup skipped flushCollabNow on rawMode, but the timer fired its own runCollabFlush — which then PATCHed stale rich markdown on top of subsequent raw saves. Fix: explicitly clearTimeout(collabFlushTimer) at the end of the rich→raw onclick (after the loop-flush, before flipping rawMode). Belt-and-braces with the Round 3 cleanup skip. * fix(collab): seed raw mode from lastFlushed (not unflushed Y.Doc) per Codex review (round 6) [HIGH] Round 5's loop-flush could exit at the 3-iteration cap with md still differing from the last-PATCHed value, then seed rawSeedMarkdown with that unflushed md. An immediate navigation without typing would lose the unpersisted state. Fix: track lastFlushed inside the loop. After the loop, seed rawSeedMarkdown = lastFlushed (the markdown we actually PATCHed), NOT md (potentially a never-flushed in-memory value). If peer edits keep arriving past our cap, items.content lags Y.Doc briefly — but the peer's own 5s flush will catch up shortly, and at least raw mode shows state consistent with items.content rather than holding a value the server never received. * fix(collab): three corner-case fixes per Codex review (round 7) 1) [HIGH] lastFlushed = md was set unconditionally inside the loop-flush, even when runCollabFlush returned false (PATCH failed). rawSeedMarkdown could then be seeded with markdown the server never received. Fix: gate `lastFlushed = md` on runCollabFlush returning true. Failed PATCHes leave lastFlushed at its prior value. 2) [HIGH] lastFlushedContent (the collab-flush dedupe key) was never invalidated by raw-mode direct saves. Scenario: collab flushes A. Raw saves B. User returns to rich + edits back to A. Next collab flush dedupes (lastFlushedContent === A) and skips, leaving items.content stuck on B. Fix: reset lastFlushedContent = null after every successful raw save (both the regular handleRawContentUpdate path and the flushRawIfPending drain loop) so subsequent collab flushes always re-PATCH. 3) [MEDIUM] The async rich→raw onclick applied rawSeedMarkdown + rawMode = true after multiple awaits without verifying the user was still on the same item. A navigation during the loop-flush could let item A's handler resume and seed raw mode on item B. Fix: before mutating component state (rawSeedMarkdown, rawMode), check `item?.id === itemId` (the captured target). Bail with `return` if mismatched. * fix(collab): differentiate flush outcomes + foreground keepalive=false per Codex review (round 8) Two findings from round 8: 1) [P1] runCollabFlush returned `false` for both PATCH failure AND dedupe-skip. The rich→raw toggle treated `false` as "didn't flush" and didn't seed rawSeedMarkdown — but a dedupe means items.content already matches our markdown (the prior successful flush put it there). Raw mode then seeded from the page's stale `item.content` field, and a subsequent raw save could overwrite the current server content with the pre-collab snapshot. Fix: change runCollabFlush's return type to a discriminated string: 'flushed' | 'deduped' | 'failed'. The toggle treats 'flushed' and 'deduped' equivalently for seeding (both mean "server has this markdown") and only bails on 'failed'. 2) [P2] The toggle path used keepalive=true for the awaited flush. Browser keepalive requests can reject for bodies larger than the per-origin keepalive quota (~64KB). On reject, the catch silently fell through and raw mode activated with rawSeedMarkdown null. Fix: switch the toggle path to keepalive=false. The await is synchronous and user-initiated; navigation isn't imminent, so the keepalive escape hatch isn't needed (and risks losing the explicit save). Also added an `aborted` short-circuit so a 'failed' result returns early WITHOUT entering raw mode — user can retry. Cleanup-driven flushes (which DO need to survive page lifecycle) still use keepalive=true. |