Files
pad/internal/webhooks/bug3340_midflight_test.go
T
xarmian 7ee0bddc45 fix(webhooks): whole-workspace access to manage; no delivery for a deleted workspace (#1727)
* fix(webhooks): whole-workspace access to manage them; none for a deleted workspace (BUG-3340)

A webhook receives item snapshots from every collection, but creating or
test-firing one required only the owner role, so an owner restricted to
specific collections could route every collection's content to a URL of
their choosing. Create and test now require unrestricted workspace
access, through the same check full export uses (BUG-1922), factored
into requireUnrestrictedAccess. Delete stays open: it is revocation.

ListWebhooks now returns nothing for a soft-deleted workspace. Both
dispatcher delivery paths (Dispatch and DeliverEvent) list through it,
so a deleted workspace's events are dropped instead of delivered during
the restore window; a restore brings the webhooks back.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(webhooks): re-check the workspace before every send; hide URLs from restricted owners (BUG-3340)

Codex r1 (no P0):
- P1: ListWebhooks checked deletion once, but a delivery reuses the hook
  list and payload across inline retries, later hooks in the fan-out,
  and redirect hops; codex reproduced POSTs after the delete committed.
  The dispatcher store gains WorkspaceLive, checked before every attempt
  (a new deliverySuppressed result: nothing sent, not the endpoint's
  failure, not counted as matched) and before every redirect hop (the
  request carries the hook's workspace). A lookup error is transient, so
  a blip delays rather than drops or sends unchecked.
- P2: the list still showed restricted owners every destination URL, and
  a URL can be the credential to a receiver's stored request bodies. A
  restricted owner now sees each webhook without its URL (the id is
  kept, so they can still delete it).

Tests: retry, later hook and redirect after a mid-delivery deletion each
send nothing (all three red with the checks disabled); the list hides
URLs from the restricted owner only.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(webhooks): check the workspace on every connection; defer on lookup errors (BUG-3340)

Codex r2 (no P0):
- P1: net/http can send more than once inside one Do (it re-dials and
  replays the body after a reused connection fails with nothing
  written), and a deletion can land during dial or TLS setup, both after
  the per-attempt check. An httptrace GotConn hook now re-checks the
  workspace on every connection the request is about to be written on,
  after TLS and before writing, and cancels the request if it is gone.
- P2 (two): a liveness lookup ERROR was treated as a deletion at the
  redirect check (the outbox then acked an undelivered event) and as an
  endpoint failure before a send (charged to the hook, which could
  auto-deactivate it). A new deliveryDeferred result sends nothing, keeps
  the event owed (Transient, so the outbox does not ack), and charges
  nothing to the hook. Suppression is reserved for a confirmed deletion.

Tests: deletion before the connection is ready sends nothing; a
pre-send lookup error and a redirect lookup error each defer without
sending or charging. Each goes red when its own fix is reverted.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(webhooks): close the connection in GotConn, a real write barrier (BUG-3340)

Codex r3 (no P0/P2): cancelling the request context in GotConn is not a
write barrier for HTTP/1: the transport queues the request before it
checks the context, so its writer could still send the whole request
(codex measured 10/10,000 in the zero-send test). GotConn now also closes
the connection it was handed when the workspace is gone or its liveness
unknown: nothing can be written to a closed socket, and the cancel stops
any transport retry on a fresh one.

Measured here: the cancel-only version fails the zero-send test 2/10,000;
with the close, 0/10,000 for both the deletion and lookup-error tests,
and the webhooks package passes under -race.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(webhooks): the connection check decides the outcome even if Do returns a response (BUG-3340)

Codex r4 (no P0/P1) P2: after GotConn closes and cancels, net/http can
still return an already-buffered response, and the suppressed/deferred
flags were only consulted when Do returned an error, so such a response
counted as a delivery. The flags are now checked right after Do,
whatever it returned, closing any response body first.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-10-02 13:20:09 -04:00

197 lines
7.1 KiB
Go

package webhooks
import (
"errors"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"github.com/PerpetualSoftware/pad/internal/models"
)
// BUG-3340 (codex r1 P1): a delivery that read its hook list before the
// workspace was soft-deleted must stop sending at the next boundary: the
// next retry, the next hook in the fan-out, and any redirect hop. Each test
// deletes the workspace from inside the first request the receiver sees.
func TestBUG3340_RetryAfterDeletionSendsNothing(t *testing.T) {
var hits int32
var st *mockStore
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&hits, 1)
st.deleteWorkspace("ws1")
w.WriteHeader(http.StatusInternalServerError) // transient: would retry
}))
defer srv.Close()
st = newMockStore([]models.Webhook{{ID: "h1", WorkspaceID: "ws1", URL: srv.URL, Events: `["*"]`, Active: true}})
d := NewDispatcher(st)
d.SkipSSRF = true
d.retryBackoff = 0
out, err := d.DeliverEvent(Delivery{WorkspaceID: "ws1", EventID: "e", Event: "item.created", Payload: []byte(`{}`)})
if err != nil {
t.Fatal(err)
}
if n := atomic.LoadInt32(&hits); n != 1 {
t.Errorf("receiver got %d POSTs, want 1 (no retry after the deletion)", n)
}
if out.Matched != 0 || out.Transient != 0 || out.Permanent != 0 {
t.Errorf("outcome %+v: a suppressed delivery owes nothing and is not a failure", out)
}
}
func TestBUG3340_LaterHookAfterDeletionSendsNothing(t *testing.T) {
var first, second int32
var st *mockStore
a := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&first, 1)
st.deleteWorkspace("ws1")
w.WriteHeader(http.StatusOK)
}))
defer a.Close()
b := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&second, 1)
w.WriteHeader(http.StatusOK)
}))
defer b.Close()
st = newMockStore([]models.Webhook{
{ID: "h1", WorkspaceID: "ws1", URL: a.URL, Events: `["*"]`, Active: true},
{ID: "h2", WorkspaceID: "ws1", URL: b.URL, Events: `["*"]`, Active: true},
})
d := NewDispatcher(st)
d.SkipSSRF = true
if _, err := d.DeliverEvent(Delivery{WorkspaceID: "ws1", EventID: "e", Event: "item.created", Payload: []byte(`{}`)}); err != nil {
t.Fatal(err)
}
if atomic.LoadInt32(&first) != 1 {
t.Fatalf("control: the first hook should receive the event")
}
if n := atomic.LoadInt32(&second); n != 0 {
t.Errorf("the second hook got %d POSTs after the workspace was deleted, want 0", n)
}
}
func TestBUG3340_RedirectAfterDeletionIsNotFollowed(t *testing.T) {
var target int32
var st *mockStore
dst := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&target, 1)
w.WriteHeader(http.StatusOK)
}))
defer dst.Close()
src := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
st.deleteWorkspace("ws1")
http.Redirect(w, r, dst.URL, http.StatusTemporaryRedirect)
}))
defer src.Close()
st = newMockStore([]models.Webhook{{ID: "h1", WorkspaceID: "ws1", URL: src.URL, Events: `["*"]`, Active: true}})
d := NewDispatcher(st)
d.SkipSSRF = true
if _, err := d.DeliverEvent(Delivery{WorkspaceID: "ws1", EventID: "e", Event: "item.created", Payload: []byte(`{}`)}); err != nil {
t.Fatal(err)
}
if n := atomic.LoadInt32(&target); n != 0 {
t.Errorf("the redirect target got %d POSTs after the workspace was deleted, want 0", n)
}
}
// Codex r2 P1: net/http can send inside a single Do on a connection obtained
// after the per-attempt check (a re-dial and replay, or a deletion during
// dial or TLS setup). The per-connection check must catch it: the workspace
// is live at the attempt check and deleted by the time a connection is ready.
func TestBUG3340_DeletionBeforeConnectionReadySendsNothing(t *testing.T) {
var hits int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&hits, 1)
w.WriteHeader(http.StatusOK)
}))
defer srv.Close()
st := newMockStore([]models.Webhook{{ID: "h1", WorkspaceID: "ws1", URL: srv.URL, Events: `["*"]`, Active: true}})
var calls int32
st.liveFn = func(string) (bool, error) {
// First call: the per-attempt check. Every later call: the
// per-connection check, after the deletion.
return atomic.AddInt32(&calls, 1) == 1, nil
}
d := NewDispatcher(st)
d.SkipSSRF = true
out, err := d.DeliverEvent(Delivery{WorkspaceID: "ws1", EventID: "e", Event: "item.created", Payload: []byte(`{}`)})
if err != nil {
t.Fatal(err)
}
if n := atomic.LoadInt32(&hits); n != 0 {
t.Errorf("receiver got %d POSTs after the workspace was deleted before the connection was ready, want 0", n)
}
if out.Matched != 0 || out.Succeeded != 0 {
t.Errorf("outcome %+v: want a suppressed delivery (nothing matched, nothing sent)", out)
}
}
// Codex r2 P2s: a liveness LOOKUP ERROR is neither a deletion nor the
// endpoint's failure. It must leave the event owed (Transient, so the outbox
// does not ack it) and must not be charged to the webhook.
func TestBUG3340_LivenessLookupErrorDefersWithoutCharging(t *testing.T) {
var hits int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&hits, 1)
w.WriteHeader(http.StatusOK)
}))
defer srv.Close()
st := newMockStore([]models.Webhook{{ID: "h1", WorkspaceID: "ws1", URL: srv.URL, Events: `["*"]`, Active: true}})
st.liveFn = func(string) (bool, error) { return false, errors.New("db down") }
d := NewDispatcher(st)
d.SkipSSRF = true
d.retryBackoff = 0
out, err := d.DeliverEvent(Delivery{WorkspaceID: "ws1", EventID: "e", Event: "item.created", Payload: []byte(`{}`)})
if err != nil {
t.Fatal(err)
}
if n := atomic.LoadInt32(&hits); n != 0 {
t.Errorf("receiver got %d POSTs with liveness unknown, want 0", n)
}
if out.Transient != 1 || out.Matched != 1 {
t.Errorf("outcome %+v: want the event still owed (Matched 1, Transient 1)", out)
}
st.mu.Lock()
_, charged := st.failures["h1"]
st.mu.Unlock()
if charged {
t.Error("a liveness lookup error was recorded against the webhook")
}
}
func TestBUG3340_RedirectLivenessLookupErrorDefers(t *testing.T) {
var target int32
dst := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&target, 1)
w.WriteHeader(http.StatusOK)
}))
defer dst.Close()
var failRedirect atomic.Bool
src := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
failRedirect.Store(true)
http.Redirect(w, r, dst.URL, http.StatusTemporaryRedirect)
}))
defer src.Close()
st := newMockStore([]models.Webhook{{ID: "h1", WorkspaceID: "ws1", URL: src.URL, Events: `["*"]`, Active: true}})
st.liveFn = func(string) (bool, error) {
if failRedirect.Load() {
return false, errors.New("db down")
}
return true, nil
}
d := NewDispatcher(st)
d.SkipSSRF = true
out, err := d.DeliverEvent(Delivery{WorkspaceID: "ws1", EventID: "e", Event: "item.created", Payload: []byte(`{}`)})
if err != nil {
t.Fatal(err)
}
if n := atomic.LoadInt32(&target); n != 0 {
t.Errorf("redirect target got %d POSTs with liveness unknown, want 0", n)
}
if out.Transient != 1 {
t.Errorf("outcome %+v: a redirect liveness error must leave the event owed", out)
}
}