mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-10-27 14:57:15 +00:00
dbbbefda75d4fc4aaeac7bdaecd65eab772f00a9
14 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xarmian
|
fa66181b88 |
fix(webhooks): a DNS failure is retried, and a redirect may not turn the POST into a bodyless GET (BUG-3661, BUG-3662) (#2163)
* fix(webhooks): a DNS failure is retried, and a redirect may not turn the POST into a bodyless GET (BUG-3661, BUG-3662) BUG-3661: the owner path's pre-delivery URL check returned the resolver's error like a policy refusal, and attemptDeliver and checkRedirect made every validation error permanent, so a DNS timeout or SERVFAIL ended the endpoint's part in the event for good. The app path's dial turned resolver errors into ErrRefused, which attemptApp maps to permanent. Now ValidateWebhookURLContext returns a typed ResolveError from a lookup bound to the attempt's timeout (a seam, lookupIP), attemptDeliver and checkRedirect treat it as transient, and appfetch's dial returns resolver errors as a network error (ErrNotSent from Poster.Post: transient, unsent). Policy refusals stay permanent. BUG-3662: net/http follows a 301, 302 or 303 after a POST as a bodyless GET and keeps the headers, so the receiver got a GET with a signature for a body it never saw, and a 2xx was recorded as delivered. checkRedirect now refuses a hop that changes the method or drops the body (permanent: the receiver's configuration). 307 and 308 keep both and are followed. Tests: resolver failure transient at delivery, at a redirect and on the app poster, with policy controls staying permanent / refused; 301/302/303 refused without sending the GET; 307/308 deliver the POST with its body and signature. Each fix's mutant turns its test red. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(webhooks): a dial the address screen refuses is permanent, and one deadline covers the whole attempt (BUG-3661, codex r1) Medium: screenDialAddr refusals (a DNS answer that changed between the URL check and the dial) were plain errors, so the Do error counted as transient and a policy refusal was retried. They now wrap errDialBlocked, which attemptDeliver treats as permanent. Low: the pre-delivery lookup had its own deliveryTimeout before the request got another; both now run under one attempt context. TestBUG3661_DialTimeBlockIsPermanent (validation sees a public answer, the dialer resolves localhost) is red with the classification removed. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
f6938b33a5 |
feat(webhooks): owner webhooks keep per-endpoint delivery state; 408/425/429 are retried (TASK-3409) (#1809)
* fix(webhooks): 408, 425 and 429 are transient on the owner and app delivery paths, through one shared RetryableStatus (TASK-3409) * feat(webhooks): owner hooks keep per-endpoint delivery state: a retry goes only to endpoints still owed, a 4xx is final for the event, and endpoints still owed when retention gives up are counted as dropped (GET /webhooks, pad webhook list) (TASK-3409) * fix(webhooks): an owner endpoint deactivated while an event is still owed to it is recorded and counted as dropped, not silently skipped (TASK-3409, codex r1) * fix(store): the automatic deactivation settles the endpoint's owed deliveries as dropped and counts them, in its own transaction (TASK-3409, codex r2) |
||
|
xarmian
|
be54575954 |
feat(apps): per-endpoint webhook delivery state, a per-install rate cap, and a 24 h drop (TASK-3408, SPEC-6 U10c) (#1783)
* feat(apps): per-endpoint webhook delivery state, a per-install rate cap, and a 24 h drop (TASK-3408, SPEC-6 U10c) webhook_deliveries records each app hook's decision per outbox event. A terminal row (delivered, permanent, refused, skipped, dropped) is never retried, so a retry goes only to app endpoints still owed, and an event an app was refused once (held, disabled, not visible) stays refused, which closes U10b's deliver_from precision and clock-skew residuals for any event refused once. A pending row (transient, deferred, rate_limited) keeps the event owed. Owner hooks record only "delivered", used only to skip re-sending an event an owner endpoint already received while it stays owed to an app (lead ruling, day 87). Owner failure, retry and ack are unchanged; full owner per-endpoint state is TASK-3409. A per-install cap of 600 a minute (burst 60, per instance) defers the excess without charging an attempt. A delivery still owed 24 h after its event is dropped, counted on the hook and shown to the owner as undelivered_dropped. Migrations 124 / pg 098. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): rate cap counts every attempt and only subscribed targets; limiter pruned; bulk skips recorded (codex r1 on U10c, TASK-3408) Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): only owed deliveries are dropped; the rate cap is asked after admission; the ledger counts requests sent (codex r2 on U10c, TASK-3408) Admission's checks become one function run under the install lock for admission and as a plain read for AppDeliveryRefusal, so a 24 h-old delivery admission would refuse is recorded refused, not dropped and counted. The rate gate moves after admission, so a refused attempt spends nothing. DeliverAppEvent returns the requests it sent, which the ledger adds. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): a request that failed before sending is not counted (codex r3 on U10c, TASK-3408) appfetch.Poster wraps dial and TLS-handshake failures in ErrNotSent; the dispatcher counts a request as sent only when the failure is neither that nor a policy refusal. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): the drop decision runs under the install lock; a write failing before its first byte is unsent (codex r4 on U10c, TASK-3408) DropOwedDelivery takes the install row and runs admission's checks in the same transaction that records the outcome, so a disable or upgrade cannot land between the check and the drop. The Poster counts bytes written through its connection; a write that fails at zero bytes is ErrNotSent. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): a late delivery replaces a drop and takes it off the count (codex r5 on U10c, TASK-3408) A request admitted before another drainer dropped its event (the first drainer's outbox claim expired mid-request) did reach the app. RecordDelivery now reads the row under its own transaction: delivered replaces dropped and decrements dropped_count; every other terminal decision still stands. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): RecordDelivery inserts before it locks, so a concurrent drop is seen (codex r6 on U10c, TASK-3408) SELECT ... FOR UPDATE locks nothing when the row does not exist, so a drop committing between the read and the upsert was overwritten with the count left raised. RecordDelivery now inserts with ON CONFLICT DO NOTHING (which waits for a concurrent inserter), and only when the row already existed locks and reads it before deciding. A deterministic test drives a drop between the two statements; run against the previous code on Postgres it ends delivered with dropped_count 1. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * test(apps): the drop/record race test waits for the drop to reach its write (codex r7 on U10c, TASK-3408) Against the previous RecordDelivery on Postgres it fails 3 of 3 (delivered, dropped_count 1); with the fix it passes on both dialects. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
68e306c76c |
feat(apps): app webhook delivery behind an in-flight fence, signature v2 (TASK-3408, SPEC-6 U10b) (#1780)
* feat(apps): app webhook delivery behind an in-flight fence, signature v2 (TASK-3408, SPEC-6 U10b) The outbox drain delivers each app event a second time, to the workspace's app hooks, with a body built from the event-time app-projection block alone (the DOC-3371 §5 table); an event without one is skipped and counted. The envelope's via_app is set on create events only (lead ruling, day 87); actor attribution is U10d. Every attempt is admitted in one transaction that takes the install row FOR SHARE, checks install active, hook released, event subscribed and its collection still the app's, and records the attempt in app_delivery_inflight with a database-time expiry (12 s; the attempt's 10 s deadline starts before admission). Disable, rotate and uninstall now drain: phase 2 waits until no unexpired record remains. https only, never a redirect, dial-time screening with the admin's webhook-flagged private origins (appfetch.Poster). Signature v2: X-Pad-Signature-256 t=,v1= over "t.body", plus X-Pad-Install. Backoff waits are context-aware timers for owner and app hooks alike; the server cancels the dispatcher's context at Stop. Rotate holds the hook again with a fresh secret. Migrations 123 / pg 097. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * test(apps): U10b mutant kills: skipped events are acked, Stop ends a retry, a stopped dispatcher sends nothing, an unsendable admission releases its record; NUL baseline (TASK-3408) Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): app webhook fence holds until the body is written; path prefix by segment; deleted workspaces refused (codex r1, TASK-3408) Poster.Post cancels and waits for the transport to close the request body before returning, so a server answering early cannot let the in-flight record end while body bytes still go out. A private origin's path prefix is matched by whole segments, refusing dot segments and percent-encoded paths. Admission refuses an install whose workspace is soft-deleted. The database-clock-step residual is stated where the expiry is defined. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): seal the app webhook connection before Post returns; keep every private-origin path (codex r2, TASK-3408) Waiting for the request body's Close was not enough: the transport closes the body before flushing its buffered writer. Every Poster connection is now wrapped in a write fence: writes go through a lock, and Post seals the connection (unblocking and then waiting out any write in progress) before it returns, so no byte can follow. Keep-alives are off for app hooks, so a connection carries one request. Duplicate private-origin entries keep every path; an entry without one leaves the origin unrestricted. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): app webhooks write the whole request before reading the answer; bulk members never reach apps (codex r3, TASK-3408) The Poster no longer uses net/http's Transport, which may return a response while still writing the request. It dials through the same screen (the screened IP itself), runs crypto/tls with SNI, hostname verification, the system roots and a TLS 1.2 floor, writes the request with (*http.Request).Write, flushes, then parses the answer with http.ReadResponse under a 64 KiB bound, and closes the connection, all on the caller's goroutine. A 2xx therefore means the whole body was sent, and nothing can write after Post returns. This replaces r2's write fence. Events of a bulk operation (batch_id set) are not delivered to apps, whether claimed alone or folded, and are counted skipped_bulk: which members arrive alone depends on claim timing (DOC-3371 §5: bulk edits are invisible to apps in v1). Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): interim answers skipped; private pins stay paired with their path; app comments carry their install; folded bulk counted (codex r4, TASK-3408) Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): an event the app skipped is never delivered later because an owner hook kept it pending (codex r5, TASK-3408) webhooks.deliver_from is the first instant an app hook may receive events: set at redeem, at re-enable, and when an upgrade changes the subscriptions. Admission refuses an event that occurred earlier, so an event skipped while the hook was held, the install disabled, or the type unsubscribed is not delivered when a failing owner hook keeps its outbox row pending. Both stamps are whole seconds; an event in the same second as the boundary is admitted (a stated one-second residual). Per-endpoint state (U10c) replaces the shared retry. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(apps): hooks released under migration 122 deliver from their release; clock residuals stated (codex r6, TASK-3408) Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
c52f64e263 |
fix(netpolicy): one outbound address policy for URL import and webhooks (#1738)
* fix(urlimport): one address policy for URL import and webhooks (BUG-3358) URL import refused only loopback, RFC 1918, link-local and CGNAT, so it reached 198.18.0.0/15, 0.0.0.0/8, 240.0.0.0/4 and the other reserved ranges webhooks already refused, and the import door returns the fetched body. Both now call internal/netpolicy.Blocked, checked at dial time against the address being connected to, which also adds IPv6 forms that embed IPv4 (NAT64, 6to4, IPv4-compatible) and the remaining special-purpose IPv6 blocks. Regression legs cover a literal address, a DNS answer and a redirect. * fix(netpolicy): registry gaps, public exceptions, NAT64 by embedded address, malformed input (BUG-3358) Codex r1: add 192.88.99.0/24, 100:0:0:1::/64, 3fff::/20 and 5f00::/16; allow the globally reachable exceptions inside 192.0.0.0/24 and 2001::/23; judge the NAT64 well-known prefix by the IPv4 address it embeds instead of refusing the prefix; refuse any slice that is not 4 or 16 bytes. The redirect test now requires the policy refusal, and the dialer's literal screen has its own leg. |
||
|
xarmian
|
7ee0bddc45 |
fix(webhooks): whole-workspace access to manage; no delivery for a deleted workspace (#1727)
* fix(webhooks): whole-workspace access to manage them; none for a deleted workspace (BUG-3340) A webhook receives item snapshots from every collection, but creating or test-firing one required only the owner role, so an owner restricted to specific collections could route every collection's content to a URL of their choosing. Create and test now require unrestricted workspace access, through the same check full export uses (BUG-1922), factored into requireUnrestrictedAccess. Delete stays open: it is revocation. ListWebhooks now returns nothing for a soft-deleted workspace. Both dispatcher delivery paths (Dispatch and DeliverEvent) list through it, so a deleted workspace's events are dropped instead of delivered during the restore window; a restore brings the webhooks back. Co-Authored-By: Claude <noreply@anthropic.com> * fix(webhooks): re-check the workspace before every send; hide URLs from restricted owners (BUG-3340) Codex r1 (no P0): - P1: ListWebhooks checked deletion once, but a delivery reuses the hook list and payload across inline retries, later hooks in the fan-out, and redirect hops; codex reproduced POSTs after the delete committed. The dispatcher store gains WorkspaceLive, checked before every attempt (a new deliverySuppressed result: nothing sent, not the endpoint's failure, not counted as matched) and before every redirect hop (the request carries the hook's workspace). A lookup error is transient, so a blip delays rather than drops or sends unchecked. - P2: the list still showed restricted owners every destination URL, and a URL can be the credential to a receiver's stored request bodies. A restricted owner now sees each webhook without its URL (the id is kept, so they can still delete it). Tests: retry, later hook and redirect after a mid-delivery deletion each send nothing (all three red with the checks disabled); the list hides URLs from the restricted owner only. Co-Authored-By: Claude <noreply@anthropic.com> * fix(webhooks): check the workspace on every connection; defer on lookup errors (BUG-3340) Codex r2 (no P0): - P1: net/http can send more than once inside one Do (it re-dials and replays the body after a reused connection fails with nothing written), and a deletion can land during dial or TLS setup, both after the per-attempt check. An httptrace GotConn hook now re-checks the workspace on every connection the request is about to be written on, after TLS and before writing, and cancels the request if it is gone. - P2 (two): a liveness lookup ERROR was treated as a deletion at the redirect check (the outbox then acked an undelivered event) and as an endpoint failure before a send (charged to the hook, which could auto-deactivate it). A new deliveryDeferred result sends nothing, keeps the event owed (Transient, so the outbox does not ack), and charges nothing to the hook. Suppression is reserved for a confirmed deletion. Tests: deletion before the connection is ready sends nothing; a pre-send lookup error and a redirect lookup error each defer without sending or charging. Each goes red when its own fix is reverted. Co-Authored-By: Claude <noreply@anthropic.com> * fix(webhooks): close the connection in GotConn, a real write barrier (BUG-3340) Codex r3 (no P0/P2): cancelling the request context in GotConn is not a write barrier for HTTP/1: the transport queues the request before it checks the context, so its writer could still send the whole request (codex measured 10/10,000 in the zero-send test). GotConn now also closes the connection it was handed when the workspace is gone or its liveness unknown: nothing can be written to a closed socket, and the cancel stops any transport retry on a fresh one. Measured here: the cancel-only version fails the zero-send test 2/10,000; with the close, 0/10,000 for both the deletion and lookup-error tests, and the webhooks package passes under -race. Co-Authored-By: Claude <noreply@anthropic.com> * fix(webhooks): the connection check decides the outcome even if Do returns a response (BUG-3340) Codex r4 (no P0/P1) P2: after GotConn closes and cancels, net/http can still return an already-buffered response, and the suppressed/deferred flags were only consulted when Do returned an error, so such a response counted as a delivery. The flags are now checked right after Do, whatever it returned, closing any response body first. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
xarmian
|
6a37512227 |
feat(server): outbox drain — webhooks delivered from the choke point (TASK-2714) (#1173)
* test(store): pin the events/1 taxonomy as an independent copy (TASK-2714)
TestCanonicalEventsAreFullyDeclared iterated kernelevents.Canonical() and
asserted each entry resolved something non-empty. That check cannot fail for
any table the compiler accepts: eventSpec requires both fields, so a corrupted
table — an entry deleted, an entry added, item.deleted quietly rebased onto the
ref-only payload — passed its own validation. A test that agrees with whatever
the table says is not a test of the table.
The sixteen name/subject/family triples are now written out as literals, so the
test DISAGREES with the table when the table moves. The wire strings behind the
name constants are pinned separately, because the triple map is keyed on
literals and a renamed constant would otherwise slip through as long as the
constant and the table moved together.
Ordered as this unit's first commit because TASK-2714 edits that table (the
handler-path bulk mapping): an independent copy earns its keep at the moment of
the edit, not before.
Mutation matrix, 4/4 caught: drop member.joined (17 -> 15 count mismatch and a
missing-name error), rehome item.deleted onto ref_only (family mismatch),
rename ItemMoved's wire string to item.move (constant leg), add an undeclared
item.frobnicated entry (count + undeclared-name + non-canonical legs). The
fourth reported "survived" on its first run because the sed never matched the
table's alignment — the mutation was verified present in the file before the
result was believed.
TASK-2714 requirement 4 (lead pass on #1172).
* feat(store): max-age prune for undispatched outbox rows (TASK-2714)
Requirement 3's missing half. PruneDispatchedOutbox filters on dispatched_at
IS NOT NULL, so a row that can never be delivered — a workspace whose only
webhook was deleted, an endpoint that 4xxs forever — is unreachable by it and
keeps its frozen payload indefinitely.
That matters because SPEC-3 makes payload privacy TEMPORAL. An outbox payload
is a frozen snapshot and account deletion's de-identify posture reaches only
live rows, so the retention window is the whole privacy claim; a window only
one of its two halves can close is not a window.
The trade is stated in the doc comment rather than left to be inferred:
at-least-once holds WITHIN the retention window and not past it, which is why
the caller's max-age must be far larger than any retry schedule. Deleting
rather than stamping the rows dispatched is deliberate — a dispatched stamp
would be a lie in the durable record, and this table is the only evidence of
what the kernel emitted.
Mutation matrix, 2/2 caught: drop the dispatched_at IS NULL clause (prunes the
aged DISPATCHED row too, handing retention two owners with different windows),
drop the occurred_at cutoff (prunes a young pending row a retry is still
owed). The test asserts its own premise — all three seeded rows are confirmed
present before the survivor checks, which would otherwise pass for a reason
unrelated to the prune.
No caller yet: the drain loop wires it up in the next commit.
* feat(events): derive SSE names from the taxonomy; retire item.updated_with_comment (TASK-2714)
SPEC-3 §"the choke point owns the canonical→surface name mapping". SSE's
snake_case vocabulary and the webhook dot-form vocabulary drifted because
nothing tied them together — each was hand-passed at its own call sites. This
ties them.
v1.5 pins what "derive" means: NAME derivation, not delivery path. SSE stays
direct-published at the mutation site, because it carries request-scoped
attribution (Actor / ActorName / Source) that a frozen outbox payload
deliberately does not hold; only its NAME now comes from the taxonomy. Moving
SSE behind the drain is TASK-2722.
- eventSpec gains an `sse` field — ONE table, not a second map, for the reason
round 11 of the last unit established: a separate map can disagree with the
first and fails open exactly when it matters. Empty is a real value (attachment,
member and pack events have no SSE surface) and SurfaceSSE reports false for it,
so silence can't be mistaken for a name.
- Several canonical events derive the SAME SSE name — status_changed and moved
both surface as item_updated — because the SSE vocabulary is coarser than
events/1 and the UI never distinguished them. The finer name is what the
webhook wire and bindings get.
- The 12 canonical SSE publish sites take their names from derived package vars,
resolved AT INIT. Every call site is a compile-time constant, so a missing
surface is a startup panic rather than a per-request decision between "log and
drop" and "publish under an empty name".
- handlers_item_links.go keeps the events.ItemUpdated literal, commented: link
mutations are silent in events/1 (v1.5), so there is no canonical name to
derive from. TASK-2723 carries link.created / link.removed.
- item.updated_with_comment retired (v1.2, Dave's ruling). One producer deleted;
the events.ItemUpdatedWithComment constant deleted with it — it had no producer
and no web consumer (grepped .go/.ts/.svelte), so leaving it would leave a name
a future publisher could reach for.
The compat guard is what makes this a refactor rather than a wire change:
TestDerivedSSENamesMatchTheLegacyWireVocabulary asserts each derived name equals
the events.* constant clients are pinned to. A derivation producing
"item.created" or "item_deleted" would break the live UI while every other Go
test still passed.
Mutation matrix, 3/3 caught: rename item.deleted's SSE surface to item_deleted
(both the taxonomy test and the compat guard fail), split item.moved onto its own
SSE name (same), make SurfaceSSE return (spec.sse, ok) so no-surface events fail
open (the taxonomy test's silence leg names all four). Running total 9/9.
go test ./internal/server ./internal/store ./internal/events: all green.
* feat(webhooks): synchronous DeliverEvent seam with per-endpoint outcome (TASK-2714)
Requirements 1 and 2. Dispatch returns once its per-hook goroutines are
spawned and reports nothing, so a drain built on it would stamp rows
dispatched while the HTTP requests were still in flight — losing exactly the
events the outbox exists to make unlosable. DeliverEvent blocks and tallies.
- Delivery carries WorkspaceID / EventID / Event / OccurredAt / Payload.
OccurredAt is the EVENT's timestamp, not dispatch time: SPEC-3 pins
time-relative binding predicates to it, so stamping time.Now() would make
every consumer's notion of when a mutation happened depend on how backed up
the queue was. Payload is json.RawMessage — []byte would base64 the snapshot
into a string that is valid JSON and completely unusable.
- WebhookPayload gains ID, the consumer dedupe key SPEC-3 §Delivery guarantees
already told consumers to use. Before this, that instruction named a field
nobody could see. omitempty, because the "webhook.test" ping is not a kernel
event, has no outbox row, and must not invent an id.
- DeliveryOutcome counts rather than a status, because one event fans out to N
endpoints and the answers differ. Three distinctions the drain branches on:
Matched==0 is SUCCESS (a webhook-less workspace is owed nothing; reading it
as undelivered would back up every event in every such workspace until
retention deleted it); Permanent does not hold the event pending (re-sending
to an endpoint that will reject it again costs the queue its progress);
Transient does. Retryable() states the ack rule once instead of letting each
caller re-derive it.
- A returned error is reserved for the SERVER's failures — listing hooks,
marshalling. Those must not ack: nothing was attempted, so the event is
still owed in full.
- Dispatch keeps its async shape for its one remaining caller and says so.
deliver() now returns the outcome it always computed; the async path
discards it.
Mutation matrix, 6/6 caught: stamp dispatch time instead of occurred_at; drop
the envelope id; pass the payload as []byte (base64); deliver asynchronously
and assume success (the synchronous leg names it exactly); count a permanent
rejection as transient; swallow a store failure into a zero outcome (the test
prints the outcome that would have acked an undelivered event).
Running total 15/15. go test ./internal/webhooks green.
* feat(store): batch_id correlation for handler-path bulk mutations (TASK-2714)
F2's write half. A lane-wide bulk action is a handler LOOP over per-item store
mutations with no enclosing transaction, so each member writes its own
canonical outbox row — which is what keeps SPEC-3's per-member binding
evaluation free, and also means that without a marker the drain would put 200
item.deleted events on the webhook wire for a 200-item lane archive: exactly
the flood TASK-1668's batch event exists to prevent.
RECORDED, NEVER INFERRED (SPEC-3 v1.5). The schema-free alternative was
grouping pending rows by workspace and a time window, which would fold two
unrelated single updates into somebody's bulk event whenever they landed in
the same tick. A wire event saying "these five items changed together" is only
true if something recorded that they did.
- migrations 082 / pgmigrations 060: nullable event_outbox.batch_id, no FK
(a batch is not a row anywhere, it is a name the handler minted), plus a
partial index on the pending set.
- store.MutationOption / WithEventBatch: variadic, because every existing call
site is a single-item mutation with nothing to declare and making all of them
pass a zero value would bury the one case that matters.
- The handler mints one id per bulk OPERATION, before the loop and
unconditionally — deciding mid-loop whether a run "counts as" a batch would
make the correlation depend on how far the loop got.
POPULATION CORRECTED: my escalation said four store methods; it is FIVE.
archive (DeleteItem), restore (RestoreItem), move (MoveItemWithPreCheck), field
update (UpdateItemWithPreCheck) and assign (UpdateItem) are the complete set of
mutating store calls handlers_items_bulk.go makes — restore was the one I
missed, which is CONVE-18's exact lesson arriving one level up. The test drives
all five rather than sampling, because the failure is per-method: a signature
that accepts the option and never threads it compiles, passes everything else,
and silently un-batches one of the six bulk verbs.
Mutation matrix, 5/5 caught across the four distinct emit sites: drop the stamp
on the update path (both Update legs fail), on delete, on restore, on move. The
delete mutation first read as SURVIVED — it had made the package fail to BUILD
(opt then unused), and the grep for test-level FAIL lines printed nothing. The
compiler catch is the stronger result, but the instrument mis-reported it, so
it was re-run with opt kept alive and the test named it directly.
go test ./internal/store ./internal/server green.
* test(server): anchor the SSE compat guard to the client's literal strings (TASK-2714)
The guard compared the derivation against events.* — the Go side. A
coordinated rename of the taxonomy AND the constants passes that, and is
exactly the change that breaks the browser: the client is pinned to the
STRINGS, in web/src/lib/services/sse.svelte.ts's ITEM_EVENTS.
The wanted column is now a literal copy of what the client listens for, with
the file named. events.* is asserted alongside as a second leg, so a drift
between the Go constants and the client is attributed rather than merely
reported. Same disagree-with-the-table principle as the taxonomy test, one
layer out: this file has to be edited by hand when the wire vocabulary
intentionally changes, and that edit is when someone goes and changes the
client too.
Mutation matrix, 2/2, each hitting only its own leg: rename events.ItemCreated
to the dot-form with the taxonomy untouched (drift leg fires), and make the
taxonomy publish the dot-form on SSE (browser leg fires). Running total 22/22.
Lead's catch on the day-49 review of commit
|
||
|
xarmian
|
aeb80883f0 |
fix(webhooks): track delivery goroutines + bounded retry (BUG-2012) (#864)
* fix(webhooks): track delivery goroutines + bounded retry (BUG-2012) Webhook deliveries ran in untracked `go d.deliver(...)` goroutines that write to the store — the BUG-842 shutdown-race class that goAsync was built to prevent — and had no retry. - Inject a `spawn func(func())` into Dispatcher (SetSpawn). Server wires s.goAsync via SetWebhookDispatcher so deliveries are tracked on s.bg (Stop() waits for in-flight deliveries) and inherit goAsync's panic recovery (BUG-2011). Nil spawn falls back to a plain goroutine, so standalone Dispatcher usage is unchanged. - Add a bounded in-goroutine retry: up to 3 attempts with linear backoff on transient failures (network error / timeout / 5xx). Permanent failures (4xx, SSRF block, malformed URL) stop immediately. The final outcome is recorded once via UpdateWebhookFailure. - Tests: delivery runs on the injected spawn; transient 5xx retries to the cap; permanent 4xx does not; a recovered transient records success. Claude-Session: https://claude.ai/code/session_019knGmnHcx5rrgWXQ8V8DZS * fix(webhooks): classify redirect-block + non-5xx as permanent (Codex review) Second Codex review of PR #864 found two retry-classification gaps: - P2: an SSRF-blocked (or looping) redirect surfaces as an error from client.Do (via CheckRedirect), which the retry loop treated as transient — so a redirect to an internal target was retried 3x with backoff. Wrap a sentinel (errRedirectRejected) in checkRedirect and match it with errors.Is (url.Error unwraps to it) to classify these as permanent — attempted once, no retries. - P3: the status switch treated every non-2xx/non-4xx as transient. Narrow transient to 5xx only; 4xx/3xx-no-Location/1xx are permanent, matching the stated "network error / timeout / 5xx" retry policy. Adds TestDispatcher_RedirectBlockIsPermanent. Claude-Session: https://claude.ai/code/session_019knGmnHcx5rrgWXQ8V8DZS |
||
|
xarmian
|
ce46b6f190 |
fix(webhooks): enforce SSRF guard at dial time + widen deny-list (#836)
The webhook SSRF guard only validated the literal URL string at parse time, so HTTP 302 redirects and DNS rebinding both reached internal IPs (cloud metadata, RFC1918, docker services). Enforce the guard at connect time: the delivery client's dialer Control callback re-checks the actual resolved IP before the socket connects (closing the DNS-rebind TOCTOU), and CheckRedirect re-runs ValidateWebhookURL on every hop with a redirect cap. Proxy is pinned nil so HTTP(S)_PROXY can't bypass the dialer's check. Widen isPrivateIP to also deny CGNAT (100.64/10), IETF protocol assignments (192.0.0/24), benchmarking (198.18/15), Class E (240/4), broadcast, all multicast (224/4, ff00::/8), and the TEST-NET / IPv6 documentation ranges. Fixes BUG-1993. Claude-Session: https://claude.ai/code/session_01BoPkYhKqMiWPYmxQigeWsA |
||
|
xarmian
|
7cda0d7896 |
feat: rebrand to Perpetual Software + new tagline (IDEA-832) (#273)
Migrates from xarmian/pad to PerpetualSoftware/pad across the entire
repo and updates the product subtitle to "Collaborate with your AI
agents".
Go module rename
- go.mod: github.com/xarmian/pad → github.com/PerpetualSoftware/pad
- All Go imports updated across cmd/pad, internal/{cli,server,store,
models,collections,items,events,metrics,webhooks} (~130 files)
- Test fixtures with the literal repo slug ("xarmian/pad" in JSON
shapes, SSH/HTTPS git URL strings, workspace_context fixtures)
also updated, including the secondary repo entry
(xarmian/pad-web → PerpetualSoftware/pad-web — pad-web was also
moved to the org per branch context)
Docs / config
- README badges, install instructions, brew tap, Docker image, source
build path, sponsor link (sponsor link kept as personal @xarmian)
- Subtitle: "Project management for developers and AI agents." →
"Collaborate with your AI agents." (README, manifests, web layout
meta, .goreleaser homebrew description)
- CONTRIBUTING.md, SECURITY.md, skills/INSTALL.md
- .goreleaser.yaml: homebrew_casks owner, GHCR image, release github
owner, cosign cert-identity regex, comments
- .github/workflows/release.yml: tap/release comments
- deploy/k8s/deployment.yaml: container image
- docs/deployment.md: clone URL
- web/static/{site.webmanifest,manifest.json}: description
- web/src/routes/+layout.svelte: meta description + og:description
Brew tap path is PerpetualSoftware/tap/pad (CamelCase, matches
GitHub user case). GHCR image is ghcr.io/perpetualsoftware/pad
(lowercased per GHCR's URL normalization). CODEOWNERS @xarmian and
FUNDING.yml github: xarmian intentionally retained — those are the
personal maintainer / sponsor account, separate from the org repo.
Verification: go build ./..., go test ./... (all pkgs pass), web
build, and make install all clean (TASK-844, TASK-845).
|
||
|
xarmian
|
157ca4e88f |
chore: bump Go toolchain to 1.26 (TASK-763) (#247)
* chore: bump Go toolchain to 1.26 (TASK-763) Bump Go from 1.25 to 1.26 across all toolchain pins: - go.mod — go 1.25.0 → go 1.26.0 - Dockerfile — golang:1.25-alpine → golang:1.26-alpine - .github/workflows/ci.yml — three setup-go steps (Go, Go-Postgres, E2E jobs) - .github/workflows/release.yml — release pipeline No `toolchain` directive: the repo is pre-launch with no external contributors yet, so we set the floor where we want it (hard requirement). Verified locally before commit: - golangci-lint v2.11.4 builds and runs under Go 1.26.2 (pinned in CI) - golang:1.26-alpine and 1.26.2-alpine images present on Docker Hub - go build ./... clean - go vet ./... clean - go test ./... all pass Parent: PLAN-644 (OSS Repo Hygiene and Launch Polish). * chore: gofmt -w under Go 1.26 (TASK-763) Apply Go 1.26's gofmt to the codebase. ~41 files reformatted, all struct-tag whitespace realignment — no semantic changes. Verified: - gofmt -l ./cmd ./internal returns empty after - go build ./... still clean - go test ./... still passes (run before commit) Bundling the gofmt diff with the toolchain bump in the same PR because the formatting drift is a direct consequence of moving from 1.25 to 1.26; splitting them creates a mandatory two-PR ordering for no value. Parent: PLAN-644. * docs: bump documented Go floor to 1.26 (TASK-763) Match go.mod's hard 1.26.0 requirement in the source-build instructions. Caught by Codex review round 1 on PR #247. - README.md:158 — "Go 1.25+" → "Go 1.26+" - CONTRIBUTING.md:9 — "Go 1.25+" → "Go 1.26+" |
||
|
xarmian
|
e7f4448028 |
feat: add readiness probe and structured logging (TASK-161)
- Add /health/live (liveness) and /health/ready (readiness with DB check) endpoints - Add Store.Ping() for database connectivity verification - Create internal/logging package using stdlib log/slog - Support PAD_LOG_LEVEL (debug/info/warn/error) and PAD_LOG_FORMAT (text/json) env vars - Add structured request logging middleware replacing chi's default Logger - Migrate all log.Printf calls to slog with proper levels and key-value attrs - Exempt health probe endpoints from auth middleware |
||
|
xarmian
|
8aa6481421 |
PHASE-12: Security Hardening for Pad Cloud (#67)
* feat: enforce RBAC role checks on all mutation endpoints (TASK-150) Add requireMinRole helper and role enforcement to 30+ mutation handlers. Viewers are now blocked from all state-changing operations, editors can mutate items/docs/comments/views but not collections/webhooks/workspace settings, and only owners can perform administrative operations. Includes 11 integration tests with real auth covering viewer/editor/owner access across items, collections, documents, comments, agent roles, item links, and workspace operations. * fix: scope search results to user's workspaces (TASK-151) Search without a ?workspace= param previously returned results from all workspaces in the database. Now the handler resolves the authenticated user's workspace memberships and passes their IDs to the store query, ensuring results only include items from workspaces the user belongs to. Fresh installs (no users) retain unscoped search for backward compat. Includes integration test proving cross-workspace isolation. * fix: add webhook URL validation and SSRF protection (TASK-152) Webhook creation now validates URLs before accepting them: only HTTP(S) schemes allowed, embedded credentials rejected, private/reserved IPs blocked (loopback, RFC1918, link-local, cloud metadata 169.254.169.254), and hostnames are DNS-resolved to verify they don't point to private IPs. Defense-in-depth check also added to the dispatcher's deliver function so existing webhooks with unsafe URLs are blocked at delivery time. * feat: add CSRF protection with double-submit cookie pattern (TASK-153) Implements CSRF middleware that validates X-CSRF-Token header matches the pad_csrf cookie on all state-changing API requests. Bearer token auth, auth endpoints, and fresh installs are exempt. The frontend client reads the CSRF cookie and attaches the header on mutations. * feat: add per-endpoint rate limiting middleware (TASK-154) Adds IP-based rate limiting for auth endpoints (5/min login, 3/hr password reset, 5/hr registration) and user-based limits for API (100/min) and search (30/min). Uses golang.org/x/time/rate with automatic stale-entry cleanup. Adds chi RealIP middleware for correct client IP behind proxies. Returns 429 with Retry-After. * fix: sanitize error responses and remove PII from logs (TASK-155) Replace all writeError(500, err.Error()) calls with writeInternalError that logs the real error server-side and returns a generic message to clients. Remove email addresses, user IDs, and password reset tokens from log output to prevent PII leakage. * feat: add security headers, configurable CORS, and secure cookies (TASK-160) Add SecurityHeaders middleware (CSP, X-Frame-Options, nosniff, Referrer-Policy, Permissions-Policy). Make CORS origins configurable via PAD_CORS_ORIGINS env var. Add PAD_SECURE_COOKIES for TLS deployments (sets Secure flag on session/CSRF cookies and enables HSTS). Also adds X-CSRF-Token to CORS allowed headers. * fix: address PR review — lazy router init and trusted IP for rate limits Fix two issues flagged by Codex: 1. CORS/HSTS config was ignored because setupRouter() ran in New() before SetCORSOrigins/SetSecureCookies were called. Now uses sync.Once to lazily build the router on first ServeHTTP/Listen. 2. Rate limiter read X-Real-IP directly from untrusted headers, allowing clients to spoof IPs. Now uses RemoteAddr only (which chimiddleware.RealIP already sanitizes from trusted proxy headers). |
||
|
xarmian
|
86722d7607 |
feat: Add webhooks, saved views, enhanced dashboard, and dependencies backend
Webhooks: - Full subsystem with HMAC-SHA256 signing, event filtering, auto-disable after 10 failures, test delivery endpoint - Migration 010_webhooks.sql, model, store CRUD, dispatcher with tests - Wired into item create/update/delete/move and comment create handlers Dashboard API: - active_items: Returns actual in-progress items with refs and priorities - active_item_count on collections (excludes terminal statuses) - Blocked item detection in attention (via dependency links) - isDoneStatus expanded to include cancelled/rejected/fixed/implemented Saved Views: - Store CRUD, API handlers, routes for per-collection saved views - View config stores filters, sort, view_type Activity: - Source filtering support (web/cli/agent) in activity list endpoint |