mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-10-26 22:37:03 +00:00
dbbbefda75d4fc4aaeac7bdaecd65eab772f00a9
106 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xarmian
|
7825f3afc5 |
fix(mcp): MCP's long-lived streams keep alive and take a stream admission slot (BUG-3835) (#2197)
* fix(mcp): MCP's long-lived streams keep alive and take a stream admission slot (BUG-3835) An idle MCP event stream made no write, so #2182's write deadline never fired on it: a client that stopped reading but held TCP open pinned a handler goroutine and its session until a notification happened to be written, and a signed-in client could open any number of them. - A ': keepalive' comment at the /events cadence on any response the transport commits as 200 text/event-stream, written through the same bounded writer. mcp-go writes each SSE event with one Write, so a mutex at Write boundaries keeps keepalives between events. No keepalive after a JSON-RPC response (the final event of a POST stream) or after the handler returns. mcp-go's own heartbeat is not used: it sends ping requests every client must answer. - The shared streamAdmission gate, keyed by streamPrincipal, taken at request entry for the two long-lived doors: GET /mcp and a subscriptions/listen POST. A refusal is the same plain 429 sse_limit_exceeded with Retry-After 5, sent before the transport runs. A tool call upgraded to SSE mid-call is not counted: a refusal there would land after its side effects. Listen is detected by decoding the whole body exactly as mcp-go does (encoding/json into a "method" field), not by a fixed-size peek that key order could evade. - Docs, config, the startup log and the metric help say the budget now covers MCP's long-lived streams. Credential expiry for /mcp streams is BUG-3837. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * docs(mcp): the keepalive's shutdown window, the event-store premise and the chatgpt mount, stated precisely (BUG-3835) Codex r1 (Medium, proposed declined): a keepalive tick between the transport's return and end() can add one comment frame; it is still inside mcpBoundWrites, which joins the keepalive goroutine before returning, so net/http never sees a write after the handler. The comments now say so. Codex r1 (Low): GET /mcp/chatgpt is admitted too; the docs named only /mcp. Hunter r1: the one-Write premise holds only without an event store (writeSSEEventRaw writes twice); stated at the premise. Detection of a stalled idle client takes up to about three keepalive intervals; stated in docs/api.md. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): a panicking transport stops the keepalive; MCP streams are stated long-lived without a credential re-check (BUG-3835) Hunter r2 (Low-Medium): the keepalive shutdown ran after next.ServeHTTP without defer, so a panicking transport left it running past the handler, with chi's Recoverer swallowing the panic. Now deferred; pinned by TestMCP_BUG3835_PanickingTransportStopsTheKeepalive (red when not deferred). Hunter r2 (Medium): docs/api.md and two code comments said no MCP route is long-lived. The MCP transport's GET and subscriptions/listen streams are, and do not yet re-check their credential; that is BUG-3837. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * docs(mcp): the keepalive catches dead peers; a live non-reader is bounded by the budget, not detected (BUG-3835) r3 hunter Medium: with default socket buffers a comment every 30s takes days to fill them, so the "caught within about three keepalive intervals" claim held only in the shrunk-buffer tests. api.md, the file header comment and the stalled-reader test comment now say what the keepalive catches and what the budget bounds. Codex r3 Low: deployment.md's stream gauge and checklist now include MCP's long-lived streams. The end()/ended comments state the one-frame window between the transport's return and end(). Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): the admission body read is in the body census; dead-peer detection is stated as TCP gives it (BUG-3835) r4 blind hunter: - M1: mcpLongLivedRequest reads the request body and was not in the body reader census, so TestEveryRequestBodyReaderIsAccountedFor failed. It is now listed, with the reason and the second-buffer cost the lead accepted. - M3: a half-open peer accepts keepalive writes into the socket buffer until TCP gives up on it; "within one keepalive plus the write deadline" held only in the shrunk-buffer tests. api.md and the header comment now say so. - L3: the panicking-transport test waits for its first keepalive, read under the writer's mutex, instead of sleeping a fixed 10ms (M7 still red). Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): admission reads a POST body only when its protocol header is modern; r6 fixes (BUG-3835) Pad serves the legacy MCP protocol versions only, and mcp-go serves a subscriptions/listen only on a modern-era POST whose Mcp-Protocol-Version header is modern and matches _meta. So a POST without a modern header is never a listen, and admission no longer reads its whole body a second time (r6 hunter M1, gate approved by the lead). The docs, the metrics help, the startup log and the census reason say listen is counted once the modern era is served. api.md states the keepalive outlives a proxy idle timeout until BUG-3837, the deployment note says the per-user bucket is shared with the browser's /events, and a test pins isFinalSSEEvent against the real transport's response frame. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): admission looks for a listen only on a protocol version the transport serves; r7 fixes (BUG-3835) The listen gate now also requires the header's version to be one the mount's transport serves, read through mcp-go's SupportsProtocolVersion (forwarded through the ChatGPT wrapper). A client probing an unserved modern version at a full budget reaches mcp-go and gets the 400 it falls back on, not a 429 (r7 ruling). Pad serves the legacy versions only, so no POST body is read for admission today, which the docs now say. New tests pin that mcp-go never serves a listen without a modern header (with a control that it serves a well-formed one), that an unserved modern version is not read or counted, and that both production mounts report their served versions. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * test(mcp): the served-version gate is checked against the transports pad server mounts (BUG-3835) r8 L2 (lead ruling): internal/mcp's wiring test builds its own transports, so a shim around either production binding would silently send admission back to counting every modern version. Server.MCPMountServesVersion exposes the gate's read of the transport the server holds, and a cmd/pad test drives wireMCP and asserts both /mcp and /mcp/chatgpt report their versions. Shim mutants on each binding turn it red. Claude-Session: https://claude.ai/code/session_012QxKpjqXTpU84auY5zZ8FR * docs(mcp): the bus and mcpBoundWrites comments name the MCP stream slot and keepalive (BUG-3835) r5 hunter 3 and hunter 6's doc half, accepted by the lead (16:04Z): internal/events/bus.go said streamAdmission is acquired by "both endpoints", falsified once MCP's long-lived streams acquire it too, and mcpBoundWrites' doc comment did not mention the admission slot or the keepalive. Comment-only. Claude-Session: https://claude.ai/code/session_012QxKpjqXTpU84auY5zZ8FR * docs(api): the stream budget's 429 is stated once, and the MCP keepalive has its own paragraph (BUG-3835) r5 hunter 2, accepted by the lead (16:04Z): the admission paragraph stated the 429 refusal twice, the second time after the keepalive text it no longer fitted. Docs-only. Claude-Session: https://claude.ai/code/session_012QxKpjqXTpU84auY5zZ8FR |
||
|
xarmian
|
cfc5391581 |
fix(events): a corrupted event sequence counter heals visibly into a new id space; the id is read back, not rendered through a double (BUG-2744) (#1906)
* fix(events): a corrupted event sequence counter heals visibly into a new id space, and the id is read back rather than rendered through a double (BUG-2744) * test(events): two fixtures that forced a script failure through the sequence key use routes the guard does not heal (BUG-2744 codex r1) |
||
|
xarmian
|
372f91f03b |
fix(events): a resume from exactly the discarded peak is refused after a counter-backwards reset, instead of being handed the new sequence as though it followed (BUG-3206) (#1530)
* fix(events): a resume from exactly the discarded peak is refused after a counter-backwards reset, instead of being handed the new sequence as though it followed (BUG-3206) Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m * docs(BUG-3206): the floor does not refuse discarded+1, which is narrower than serving it (codex round 1) Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m |
||
|
xarmian
|
0b3ca0d598 |
fix(events): a closed MemoryBus refuses a late subscribe instead of handing out a channel nothing will close (BUG-2737) (#1483)
cmd_server closes the event buses before srv.Shutdown drains handlers, so a request already inside the SSE handler could subscribe after Close. MemoryBus admitted it on all three doors and returned a channel nothing would ever close, so the handler ranged over it until Shutdown's 30s deadline. RedisBus already refused the same call with SubscribeFailed, as measured on main. MemoryBus now answers SubscribeFailed, with a nil channel and no registration, on Subscribe, SubscribeIfAllowed and SubscribeAndReplaySince once closed. The resume door clears its resume-gap report first, as RedisBus does. The SSE handler already maps SubscribeFailed to a 503 with Retry-After, which is the right shutdown answer, and the two buses now answer a late subscriber alike. Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N |
||
|
xarmian
|
7574c4b831 |
fix(events): Publish reports its outcome, and a failed activity publish is counted and logged (BUG-2732) (#1457)
* fix(events): Publish reports its outcome, and a failed activity publish is counted and logged (BUG-2732)
events.EventBus.Publish now returns an error under watchevents' three-way
contract: nil accepted, ErrBusClosed provably not published, anything else
unconfirmed. MemoryBus refuses after Close; RedisBus refuses once its root
context is cancelled and returns every Redis or marshal failure.
pad_eventbus_publish_total keeps its meaning (attempts). A sibling,
pad_eventbus_publish_failures_total{outcome="closed|unconfirmed"}, counts
failures; it is additive and pre-initialised at zero.
All 9 server producers publish through publishActivityEvent, which discards
the error (the write underneath committed) and logs outcome, event type and
workspace, never the payload, rate-bounded (burst 10, then 1/s, with a
suppressed count).
Pinned: an assigned-but-unpublished ID is NOT surfaced as a gap to a
subscriber or a resume, so the counter and the log are the only trace of a
lost event.
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N
* docs(metrics): the failures counter moves during a Redis outage, not only at shutdown (BUG-2732, codex round 1)
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N
* fix(events): the bus no longer logs each failed publish itself; the caller logs it, bounded (BUG-2732, lead review)
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N
|
||
|
xarmian
|
d416a0bc1a |
docs(events, watchevents): a rejected automatic re-SUBSCRIBE is a documented limit, with the operator action (BUG-3155) (#1453)
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N |
||
|
xarmian
|
bd44f5deaa |
fix(events, watchevents): a SUBSCRIBE that Redis rejects with an error reply refuses its callers instead of admitting them (BUG-2799) (#1449)
* test(events): pin that a ReceiveTimeout expiry keeps the subscription and a ctx-deadline expiry re-subscribes (BUG-2799) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N * fix(events): a SUBSCRIBE that Redis rejects with an error reply refuses its callers instead of admitting them (BUG-2799) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N * test(events): fail the rejection test on the outcome, not on the retry count (BUG-2799) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N * fix(watchevents): a SUBSCRIBE rejected at construction leaves the slot empty so the retry gate opens (BUG-2799) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N * test(events): pin that a timer losing to a rejection admits nobody (BUG-2799) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N |
||
|
xarmian
|
9424365ac0 |
test(events): wait on the reset report, not on a state that precedes it (BUG-3145) (#1433)
* test(events): wait on the reset report, not on a state that precedes it (BUG-3145)
Main went red at
|
||
|
xarmian
|
1933041027 |
fix(events): surface the SUBSCRIBE error and refuse callers instead of admitting a dead stream (BUG-2764) (#1215)
* fix(events): surface the SUBSCRIBE error and refuse callers instead of admitting a dead stream (BUG-2764) * fix(watchevents): surface the SUBSCRIBE error at construction and on resubscribe (BUG-2764) * docs(events): the idle-cycle prose and metric Help now name the third install-nothing reason (BUG-2764) * fix(events): retry uncovered workspaces, wait on in-flight records, refuse after Close (BUG-2764 codex round 1) * fix(events): post-loop check waits on an in-flight record before trusting a live entry (BUG-2764 codex round 2) * test(watchevents): assert what the failed subscribe leaves, not how long it takes (BUG-2764 codex round 3) * docs(events): prose says what the code guarantees — delivery failure or shutdown, replacement refusals do not count as cycles (BUG-2764 codex round 4) * test(server): the 503 mapping is asserted on both subscribe branches; docs scope the refusal to the activity stream (BUG-2764 codex round 5, BUG-2800) * fix(events): uncovered-workspace retry logs are quiet once the bus is closing (BUG-2764 codex round 6) * fix(events): the uncovered-retry log promises a retry only while subscribers remain (BUG-2764 codex round 7) |
||
|
xarmian
|
2c74ac59be | test(events): hold the SUBSCRIBE ack at the wire until the timer arm is taken (BUG-2786) (#1214) | ||
|
xarmian
|
cc3cfeef2b |
fix(redis): honour the caller's context on TLS dials (BUG-2754) (#1198)
* fix(redis): honour the caller's context on TLS dials (BUG-2754) go-redis's default dialer (v9.22.0, options.go NewDialer) honours the caller's context on plaintext and NOT on TLS: the TLS branch returns tls.DialWithDialer, which takes no context at all, so a cancelled caller could not shorten the dial and it was bounded only by DialTimeout. BUG-2749 put SSE subscription establishment on the request's context so a client that disconnects stops holding its admission slots. On plaintext that covered the dial. On TLS the dial was the one segment cancellation could not reach, so the guarantee shrank from "released at once" to "released after up to DialTimeout" — and a managed Redis is a rediss:// URL, which is the ordinary production shape rather than an exotic one. Fixed at CLIENT CONSTRUCTION rather than in any consumer, because the same dial serves Publish, the Lua scripts, the presence registry and the watch bus's reads. internal/redisdial is a small package so the thing can be tested directly; cmd/pad/cmd_server.go installs it on the one client Pad builds. THREE THINGS THAT FAIL QUIETLY IF THE REPLACEMENT GETS THEM WRONG, each with a test that fails against getting it wrong: ServerName. tls.DialWithDialer infers it from the dialled address when the config leaves it empty; a hand-rolled tls.Client does not, and an empty ServerName leaves certificate verification with no name to check. That would turn a latency fix into a silent authentication regression. Replicated, on a CLONE — mutating the caller's config would leak one host's name into every later dial that shares it. Tested by dialling a certificate issued for another name and requiring an x509.HostnameError, per the lead's correction: asserting the field is set proves the code sets a field, not that the name is checked. Verified in the pinned source rather than assumed — redis.ParseURL DOES set ServerName for rediss:// (options.go:708), so Pad's path does not depend on the fallback today; it is there because it is what the replaced code did. The timeout must bound the HANDSHAKE, not just the connect. Otherwise a server that accepts and then stalls hangs for as long as the context lives — trading a bounded failure for an unbounded one, worse than the bug being fixed. It must not EXTEND an earlier deadline. context.WithTimeout takes the sooner of the two, matching go-redis's own promise about DialTimeout. PROSE SWEPT, and the sweep found two sites my first pass missed because it only grepped non-test files: five comments across internal/events said the TLS dial could not be cancelled, including one carrying an explicit "See BUG-2754 for the TLS half" forward reference. All five now say what is true, and the confirmTimeout budget comment records that it has been amended twice. Two instrument corrections: the certificate fixture put IP literals in DNSNames where x509 will never match them, and two tests detected their mutations BY HANGING — which is not a result anyone can act on, and which stranded the mutation harness with its edit still applied. Both bound the dial in a goroutine now, so a hang is a named failure. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(redis): resolve DialTimeout, keep the keep-alive, share one budget (r1) Codex round 1 found three, and the P1 was introduced BY the first draft of this fix rather than inherited — the worst kind, since the diff was sold as closing a hang. DIALTIMEOUT READ AS ZERO. go-redis's Options.init() defaults it to 5s, but NewClient CLONES the options first (redis.go:1924), so a caller reading opt.DialTimeout in order to install a Dialer — the only time it can — reads zero for the ordinary URL that sets none. And PubSubPool.NewConn calls the dialer DIRECTLY with no timeout of its own (internal/pool/pubsub.go:45), so nothing downstream supplies one either. Resolved in the package, with the coupling named. The mutation matrix then refused to confirm the failure mode the finding described, which changed the test rather than the fix. An unresolved zero does not hang HERE: this dialer wraps the dial in context.WithTimeout, and a zero duration is an already-expired deadline, so every dial would fail INSTANTLY — nothing connects at all. The original draft would have hung; this one refuses. The assertion that separates them is a healthy server being reached, not a stalled one giving up, and the comment says which draft did which. KEEPALIVECONFIG DROPPED. go-redis's default dialer sets it (options.go:608) and it governs how quickly a dead peer is noticed on every Redis connection this process holds. Reverting to OS defaults would change that across the whole client as an invisible side effect of a cancellation fix — invisible because nothing fails. My first test for it compared our copy against go-redis's published numbers, which says nothing about whether the dialer USES it: deleting the field from the dialer left that test green. Replaced with a Linux-tagged test that reads SO_KEEPALIVE and TCP_KEEPIDLE off the accepted socket. Honest partial, stated at the test: the property is platform-independent, the observation is not, and the Smoke jobs on macOS and Windows skip the file. The value-comparison test is kept as well — it catches the copy drifting from what it mirrors, which the socket test cannot. TWO SEPARATE BUDGETS. DialTimeout was applied to the TCP connect and then a fresh one started for the handshake, allowing up to 2x on the pub/sub path, which has no outer deadline to mask it. tls.DialWithDialer bounds both as one interval; this must not be laxer than the code it replaces. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(redis): honour an explicitly disabled dial timeout; finish the sweep (r2) Codex round 2, four findings and one correction to a claim I had already made. EXPLICIT dial_timeout=0 WAS BEING OVERRULED. ParseURL encodes an explicit zero or negative as -1, which go-redis preserves as "no timeout at all". Treating every non-positive value as unset collapsed that into the 5s default and silently overruled an operator who had deliberately disabled the bound — the same defect as the one round 1 found, in the opposite direction. `== 0` for the unresolved case now, with a negative carried through as no bound, and a test that goes through ParseURL rather than passing -1 by hand so it pins the real path. A DRIFT GUARD for the two copied constants, compared against go-redis's RESOLVED options (NewClient runs init() on its clone and Options() returns the result) rather than against a literal. A copy that silently diverges from what it mirrors is what would make this package worse than none. TWO STALE COMMENTS I HAD CLAIMED WERE FIXED. My sweep commit said "all five now say what is true"; it was three. The first patch batch aborted on a failed anchor and, because that helper writes only after every pair matches, none of its edits landed — I re-applied some by hand and did not re-verify the rest. The grep I ran afterwards searched for phrasings the surviving comments did not use. Both now corrected: establishSubscription's two-bullet plaintext/TLS split and the mutex comment that named TLS as the case cancellation could not reach. TWO TESTS RELABELLED RATHER THAN LEFT LOOKING LIKE COVERAGE. The single-budget test does not discriminate — the server accepts immediately, so the connect consumes none of the budget and the two-budget implementation finishes in the same time. Staging a slow connect against a local listener is not deterministic, so what holds that property is structural (one context, created before the connect, passed through the handshake) and the test says so. And the Linux-only keepalive test now states what its build tag does and does not cost: the behaviour is platform-independent and the full suite runs on Linux CI, so a removal is caught; the macOS and Windows Smoke jobs are build-and-start checks and were never the guard. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
effd0199cd |
fix(events): detect a half-open Redis connection with a bus heartbeat (BUG-2738) (#1195)
* fix(events): detect a half-open Redis connection with a bus heartbeat (BUG-2738)
A Redis connection can stop carrying traffic without closing -- no FIN, no
RST, just a route that stopped working. The instance blocks on a read that
never returns, receives nothing, and its replay buffer goes on looking
complete, so every resume is answered "caught up" from a coverage window that
ended when the route did.
go-redis cannot see it: PubSub.Ping writes the command and never reads a
reply (v9.22.0), so its health check reports healthy for as long as the socket
accepts writes. Measured on day-52 against a proxy that silently stopped
forwarding: no reconnect in 24 seconds.
Each subscription now records when it last received ANYTHING -- event,
heartbeat, or subscription acknowledgement -- and a background pass ends the
coverage of any workspace whose stamp goes stale past 3T, then REPLACES the
connection. Drop alone would not recover: the resync it demands is served from
the same dead socket, so the detector fires again on the next pass.
Dave's day-49 ruling dissolves the threshold rather than tuning it. The bus
publishes its own frame every T=30s and fires at 3T=90s, which turns "is this
workspace quiet or is the route dead?" -- unanswerable, deployment-dependent --
into "did our heartbeat arrive?".
TWO PHASES, ORDER NOT OPTIONAL. The frame must travel on the workspace's event
channel, because that connection is what needs proving. A pre-phase-1 binary
cannot classify it: the frame reaches the event decoder, fails, and since
BUG-2739 that is a hole in coverage -- so an early flip makes every un-upgraded
instance drop its buffer and resync all its clients, every 30s, per workspace,
for the length of a mixed deployment. Phase 1 recognises and ignores;
PAD_EVENTS_HEARTBEAT is phase 2, a constructor parameter with no default so
every call site states its phase.
The idle detector is a THIRD actor in a region whose invariants were designed
around request goroutines plus Close. Four rules, each commented at
cycleIdleSubscriptions and each with a test:
1. It refuses to cycle while pendingSubs holds a record, and MINTS the
record itself before tearing anything down -- subscribeAndReplay checks
pendingSubs before wsSubs, so a subscriber arriving mid-cycle joins the
replacement instead of being admitted into the doomed subscription.
2. lastSeen is stamped at INSTALL, not left at the zero value, which reads
as 1970 and would cycle hardest on an unconfirmed admission -- the
workspaces already having a bad time.
3. wsCounts is re-read under the lock that performs the teardown.
4. Re-establishment runs on b.ctx with a nil establisher; the bus has no
subscriber registration of its own to unwind.
Two decisions beyond the plan:
A NEW COUNTER, not just the reset reason. dropWorkspaceCoverage reports a
reset only when a buffer existed to drop, and the incidents this detector
exists for skew hard toward having none -- a route that wedged early on a
quiet workspace. Reading cycles off the reset label alone would under-report
exactly the case it was built to find, so pad_event_subscription_cycled_total
is the dependable count and idle_timeout is corroboration. Both comments say
which is which.
THE CADENCE IS A LIVE TUNABLE -- a timer re-read under b.mu each pass plus a
buffered kick, not a ticker constructed once. A ticker captures the interval
at goroutine start, which makes the field write-once while its comment calls
it a tunable and makes any later write a data race; it also leaves no
deterministic way to test the WIRING other than a test-only constructor.
decodePayload's signature grew a payloadKind. The classification belongs to
the decoder, not the call site, so no future caller can reintroduce the
coverage drop; and the prefix (rather than an exact payload) means a later
frame version needs no third roll.
Also swept, per the team's prose convention: receiveMessages' doc comment and
deployment.md both said this gap was open and needed a decision. Both now say
what closes it -- and deployment.md says the watch stream still has the same
defect by the same mechanism, which is its own unit.
Trio kept together: ResetReasonIdleTimeout, the metric Help strings, and
docs/deployment.md's rollout order with the mixed-fleet failure named.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): rebuild the instruments the BUG-2738 matrix showed were blind
The mutation matrix found a defect in the fix itself and three tests that
could not have caught what they were named for.
THE DEFECT: the idle scan skipped a subscription whose lastSeen was the zero
value. That reads as belt-and-braces beside the install-time stamp and is the
opposite -- it makes a subscription that has NEVER received anything
permanently uncyclable, which is the BUG-2747 unconfirmed admission: the one
population the plan singles out as mattering most, and the one where a wedged
route would then be undetectable forever. It was also masking rule 2: with the
skip present, removing the install stamp survived every test. Skip removed;
that mutation is now caught. Re-adding it is undetectable by construction and
the comment says so, because a guard that only acts once a real one has broken
converts a caught defect into a silent one.
THREE INSTRUMENTS THAT WERE NOT MEASURING:
- "Drop only, never cycle" passed because establishSubscription overwrites
wsSubs, so a generation check cannot see a replacement installed WITHOUT
tearing the old connection down -- a leaked PubSub, connection and receive
goroutine per cycle, forever, on exactly the wedged route where they never
die on their own. Now asserted on the receive loop exiting.
- The Close test was vacuous. Close drains wsSubs, so a loop that ignored
b.ctx entirely would find no workspaces and publish nothing: silence after
Close was evidence of nothing. maintenanceStopped makes the goroutine's exit
observable, which is the same reason Observer.ReceiveLoopExited exists.
- The joint test HUNG rather than failing under the drop-only mutation: the
seam never fires, so the joiner goroutine was never spawned and an unbounded
receive waited forever. The harness then aborted mid-run and LEFT THE
MUTATION APPLIED to the working tree, which a grep caught and a green test
run would not have. The wait is bounded and names the failure; the harness
bounds each run, reports a hang as its own status, and restores in a finally.
Added: a direct test that a straggler frame from a replaced generation cannot
refresh its successor's liveness -- on a wedged route, the dead connection's
buffered tail would otherwise suppress the detector for the replacement.
RULE 3 IS AN OPTIMISATION, NOT A CORRECTNESS GUARD, and the matrix says so
rather than an argument: removing the whole second read -- liveness, generation
and count terms together -- survives every test, because
establishSubscription's abandon path already refuses to install for an emptied
workspace and retires the record in the same critical section (BUG-2749). The
first read is redundant more sharply still: reaching zero takes the
subscription down with it, so this loop never sees such a workspace. Both are
kept, because neither DEPENDS on that coupling, and both comments now carry the
per-term reading instead of describing tested defence in depth. The generation
term is unreachable while the establishment record is held, by rule 1's own
mechanism.
Matrix: 16/22 detected, plus 4 follow-ups. Every survivor is documented at its
line with why it survives.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): gate idle detection on heartbeat phase 2 (BUG-2738, codex r1)
Codex round 1 found a defect the first draft had shipped WITH A COMMENT
JUSTIFYING IT, plus two coupling hazards.
P2-as-filed, P1 in effect: idle detection ran on every instance from phase 1,
on the reasoning that it could "detect off whatever traffic the deployment
already carries". That holds only for a BUSY workspace. A QUIET one on phase 1
has no events and no heartbeat, so a perfectly healthy subscription crossed
the 90s threshold on every pass and was cycled: replay coverage dropped, every
live subscriber told to resync, indefinitely -- on the DEFAULT configuration
every deployment lands in before it flips anything. A resync storm shipped as
the default, by the feature whose stated purpose is to avoid exactly that load
inversion.
Publishing and detecting are now one switch, which is what they always were:
an instance detects off its OWN frames -- it publishes to the channels it
subscribes to and receives them back -- so it never depended on peers having
flipped, and there was never a reason for the two to be separable. Phase 1 is
"recognise the frame so a phase-2 peer costs you nothing", and nothing else.
Regression test plus its counterfactual, so "no cycles" cannot be satisfied by
a detector that has simply stopped working.
P1: the maintenance loop published heartbeats and scanned for idleness on one
goroutine. publishHeartbeats makes N synchronous Redis publishes, and against
the failure this feature exists to detect those are precisely the calls that
block -- bounded by go-redis's own Dial/Read/WriteTimeout, not by any context
we can pass. A stalled publisher could therefore delay detection for as long
as those timeouts take, on the very instance whose connections had wedged, and
for longer the more workspaces it carried. Two goroutines with their own kick
channels; a stalled publisher now just produces silence, which is what the
detector reads.
P3: the cycle held the workspace's establishment record across a synchronous
observer report, so an Observer callback that subscribed to that workspace
would wait on a record only the reporting goroutine could retire. Moved the
SubscriptionCycled report past establishment. The narrower half is older than
this code -- confirmSubscription's late-acknowledgement path already reported
from inside that window -- so it is documented on the Observer interface as a
contract rather than silently worked around: a callback may publish, read and
unsubscribe; it may not subscribe.
Prose swept for what the gate falsified, per the team convention: the
constructor comment that argued for the defect, config.EventsHeartbeat's
rollback paragraph, the config test's inverted-rationale comment,
ResetReasonIdleTimeout, both metric Help strings, and deployment.md's phase
table and rollback section. All of them now say that phase 1 detects nothing
and that the cycled counter is STRUCTURALLY zero there -- a zero on phase 1
says nothing about whether a route has wedged, which is the reading an
operator would otherwise get wrong.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): prove a resuming joiner is told sync_required across a cycle
Codex round 2 raised that a subscriber arriving DURING an idle cycle gets no
gap signal, because dropWorkspaceCoverage only signals subscribers present
when it runs. True, and for a RESUMING caller the gap signal is not what
protects it: the registration mark is. It registers while the workspace has no
buffer, so its mark cannot match whatever buffer exists by the time it reads,
and eventsSinceMarkLocked answers nil -- sync_required rather than a false
"caught up".
A FRESH caller is deliberately not signalled and the finding is DECLINED for
that case, with reasons recorded at the test: it holds no prior position, so
there is no span it could be missing; it is admitted only after the
replacement subscription is acknowledged, because it waits on the cycle's
establishment record which finishPending closes after the confirmation; and on
the unconfirmed-admission path it IS told to reconcile when the acknowledgement
lands. Signalling it anyway would demand a resync of a client with nothing to
reconcile -- the load inversion this unit already had to fix once.
THE FIRST TWO VERSIONS OF THIS TEST DID NOT DISCRIMINATE, which is the part
worth keeping. Version one asserted the empty case: the cycle leaves no buffer,
so eventsSinceMarkLocked returned nil from its `!ok` term and removing the mark
check entirely still passed. Version two published inside
afterSubscriptionConfirmed so a FRESH buffer exists before the joiner reads --
and deleting the `mark.buffer == nil` term still survived, because the keep
arithmetic in that function already reduces to zero for a nil mark. Only
replacing eventsSinceMarkLocked with the unmarked eventsSinceLocked fails the
test, handing the joiner the post-cycle event as though it followed its cursor.
That is the mutation the test is built against, and the redundancy inside
eventsSinceMarkLocked is recorded rather than mistaken for coverage.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): only count a cycle that actually replaced the connection (codex r3)
Three findings from a fresh-angle round on shutdown, wire format and doc
accuracy. The wire-format angle came back clean -- events:<workspace> cannot
collide with watchevents under validated namespaces, and no valid activity
payload can be mistaken for an hb| frame.
P3, and the one that stings: config.EventsHeartbeat still said phase 1
"already runs idle detection off whatever traffic exists". That is the exact
sentence the previous commit's sweep existed to remove, in a file that sweep
edited. A grep for the phrasing I remembered writing missed the paraphrase
sitting four lines above the paragraph I did fix.
P3: SubscriptionCycled was reported unconditionally after establishSubscription
returned, but establishment has two reasons to install nothing -- the bus
closed, or the workspace emptied while we dialled. The counter's documented
meaning is "torn down AND replaced", and counting an aborted establishment is
wrong in the direction that matters: an operator reading a non-zero rate
concludes connections are being blackholed, so a shutdown would manufacture
that signal. Now reported only when a replacement is installed, verified by
generation. Both Help strings and deployment.md say "counts replacements, not
teardowns"; the teardown stays visible through the idle_timeout reset reason.
P2: Close does not join the maintenance goroutines. Kept that way and
documented on Close, because the publish half makes synchronous Redis calls
bounded by go-redis's own timeouts -- the calls that stall on exactly the
wedged route this feature detects -- so joining would let a dead network hold
shutdown open. What has to hold instead is that a cycle already past its ctx
check leaves nothing behind, which is now pinned by a test that closes the bus
from inside the cycle's establishment: no subscription installed, no
establishment record stranded, no counter moved.
liveGen moved from the test file into the package -- production needs it now.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): restore the coverage the phase gate silently removed
The mutation matrix, re-run against the post-codex code, showed M3 -- removing
the install-time lastSeen stamp -- going from DETECTED back to SURVIVED. The
cause was my own round-1 fix: gating idle detection on heartbeat phase 2 means
a phase-1 bus never scans, and TestAnUnconfirmedAdmissionIsNotCycledAsIdle
built its own phase-1 bus. It was the only test that could observe a zero
lastSeen, because the plain fresh-subscription case is stamped twice over --
at install, and again by the acknowledgement. Flipped to phase 2 and
re-verified: removing the stamp fails it again.
Worth naming the shape rather than just the fix. A behaviour change that
narrows when code runs silently narrows what the tests reach, and nothing in a
green suite says so -- the tests still pass, they just stopped asking. Only
re-running the matrix after the change surfaced it.
Two harness bugs fixed alongside, both of which had been reporting
non-results as if they were readings:
- A mutation that INSERTS keeps its own anchor, so the "did the edit land?"
check read every insertion as ANCHOR-ERROR. It compares the file now.
- The two rule-3 mutations left `sub`/`live` unused and came back BUILD-BREAK
rather than answering the question; they carry the same discard the
follow-up harness already used.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): close the wiring and barrier gaps codex round 4 found
Concurrency and lock discipline came back CLEAN -- the establishment record
and the generation checks cover two racing cycles, Unsubscribe, Publish and a
stale resubscription frame, with no lock-order deadlock. The four findings
were all about whether the tests measure what they claim.
P2, and it is the convention I had cited three commits earlier: the heartbeat
flip had no wiring test. internal/events proves a bus built with
publishHeartbeat=true emits frames and detects idleness, and every one of
those tests passes if newObservedEventBus hardcodes false -- the deployment
would simply never detect a wedged connection, which is indistinguishable from
a deployment that has none. Both directions asserted, because a helper that
ignored its config and hardcoded EITHER value passes a one-directional test.
Mutation-checked against exactly that edit.
P2: the metrics adapter test never touched SubscriptionCycled or the
idle_timeout reason, so an adapter that folded the counter into the reset
series -- destroying the very distinction those two are built to keep apart --
would have passed. Both added with counts that differ from their neighbours',
the pattern that file already uses so a label-dropping adapter cannot satisfy
the totals by coincidence.
P3: TestAHeartbeatConsumesNoEventID "waited" on a predicate that returned true
unconditionally. Not a slow wait -- no wait at all: the counter was read with
the publishes still in flight, so a heartbeat that DID consume an id could
land afterwards and the test would still pass. It now waits on the frames
arriving, and fails against a mutation that publishes an event alongside each
heartbeat.
P3: the maintenance goroutines started on phase 1, where both halves are
guaranteed no-ops -- two goroutines and two timers per process waking every
30s for the life of a deployment that asked for none of it, and phase 1 is the
DEFAULT. The flag is constructor-only so the decision is taken once. The
in-function gates stay: those are the correctness ones, and the tests reach
them directly without a loop.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): validate the heartbeat frame and stop serialising recovery (r5)
Client-facing behaviour came back CLEAN: an idle cycle signals each local
subscriber, the SSE handler emits an in-band sync_required with an empty id
while holding the connection open, EventSource retires its cursor and the web
client runs the documented reconciliation. Two P2s on the other angles.
FRAME VALIDATION. Accepting any "hb|..." created a silently-ignored class on
the workspace event channel, where before this feature EVERY unreadable
payload ended coverage loudly and moved undecodable_message -- the counter
whose documented job is "suspect a namespace collision". A foreign or buggy
publisher whose bytes happened to start with the prefix slipped through that
signal without a trace. A frame is now hb|<version> plus optional short tokens
under a length cap; anything else wearing the prefix goes back to being a
coverage-ending decode failure, and the forward compatibility the prefix was
chosen for survives for a disciplined future frame.
What this deliberately does NOT try to fix, because it is not a hole: a forged
frame cannot fake liveness. Liveness means "this socket carried traffic", and a
frame that ARRIVES demonstrates exactly that whoever sent it -- which is why
stampLastSeen already fires for undecodable frames. There is no coverage claim
inside a heartbeat to forge.
CADENCE DRIFT, which was self-defeating rather than merely untidy. The timer
restarted after each pass, so the real period was T plus however long the pass
took. For the publisher that means an instance whose publishes are slow emits
heartbeats further apart, its own subscription sees them further apart, and it
can cross its own 3T threshold and cycle connections that were never wedged --
the slowness manufacturing the incident. Scheduling is deadline-based now, and
resets rather than bursting when a pass overruns badly.
SERIAL RECOVERY. One idle pass re-established every due workspace in sequence,
each re-dial bounded by go-redis's own timeouts, so recovery took N x that
timeout with the last workspaces reporting themselves uncovered throughout.
The failure that puts many workspaces on the due list at once is a Redis
failover, so the serial case was the common one. Bounded-parallel at 8 -- each
entry already owns its establishment record so they are independent by
construction, and an unbounded fan-out would answer a struggling Redis with one
dial per workspace at once. Test covers more workspaces than the cap, and
fails against a version that drops the overflow.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* docs(events): idle_timeout means coverage ended, not connection replaced (r6)
Codex round 6 came back clean on the non-Redis path (MemoryBus ignores the
Redis-only flag; EventBus and Close have not drifted), on the rollback
rehearsal (phase-2 to phase-1 and a mixed fleet are safe as documented,
including a bus mid-cycle -- Close cancels it, prevents installation and
retires its pending record), and on the operator surface
(PAD_EVENTS_HEARTBEAT is a server env/TOML setting; `pad configure` is client
connection config and needs no new surface).
The one finding is a contract drift I introduced two commits ago and then
wrote prose for in the same commit. Making SubscriptionCycled mean "replaced"
was right; what I missed is that the idle_timeout RESET REASON is emitted
earlier -- dropWorkspaceCoverage runs before the re-establishment -- so it can
fire when nothing is replaced, which is exactly the shutdown case the counter
was changed to exclude. Three doc sites and one log line said "replaced the
connection" anyway.
They now say what is true at the moment each fires: idle_timeout means
COVERAGE ENDED, only pad_event_subscription_cycled_total proves a replacement,
and the log says "attempting to replace" rather than "replacing". The log
wording matters on its own -- an operator correlating it with the counter
would otherwise find the log without the counter and go hunting a bug that
isn't there.
Third time this unit has produced prose the next change falsified, and each
time a different reviewer angle caught it rather than the sweep I ran at the
time. The pattern is that a behaviour change and the prose describing it land
in one commit, so there is no diff between them to notice.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(cmd): drive the heartbeat wiring test instead of sleeping at it (r7)
Codex round 7 found no leftovers across seven rounds of edits, and confirmed
the mass-cycle case does NOT produce a reconnect storm -- the SSE connections
stay open across a sync_required, so the admission limits are never consulted.
P3, and it is the failure I have been criticising in other people's tests: the
wiring test used a 300ms sleep as its ordering barrier. Under -race or on a
loaded CI box, a phase-1 bus that is correctly silent and a phase-2 goroutine
that merely has not been scheduled yet are indistinguishable, so the test could
pass or fail for reasons unrelated to the flip it exists to check. It now
drives one publish pass synchronously through a named test hook and uses an
ordinary event on the same channel as the barrier, which Redis delivers in
publish order. No timing left. Verified: still fails against the flag being
hardcoded false, and ten consecutive -race runs are green.
That replaces SetMaintenanceCadenceForTest with PublishHeartbeatsForTest rather
than adding to the exported test surface -- the loop's own wiring is covered
inside internal/events, where the unexported setter is available.
P2 is FILED, NOT FIXED, as BUG-2761: a mass coverage drop tells every connected
subscriber of every affected workspace to resync at once, and each browser tab
independently calls /changes with per-tab coalescing but no jitter and no
global budget. The fix is a web-client change plus possibly a wire-format hint,
which is independent of half-open detection and would materially expand this
diff. Worth filing rather than shrugging at because this unit makes the
simultaneous case MORE likely: it adds a third trigger of a class that already
existed (Redis failover, epoch change), and its natural cause is exactly a
network event that wedges many routes at once. deployment.md carries the
residual with the bug ref so an operator meets it before the incident does.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): make the tests prove what their comments claim (codex r8)
Round 8 was claim verification rather than bug hunting -- check the diff's
load-bearing assertions against the actual code -- and it was the highest-yield
round of the eight. The go-redis assertions (Ping writes without reading, the
channel path sets no read deadline, TLS dials ignore cancellation) and the four
claims about neighbouring functions all held. Seven other assertions did not.
TESTS THAT DID NOT PROVE THEIR OWN HEADLINE. This is the substance of the
round, and every one of these passed before and after:
- The JOINT TEST -- this unit's flagship -- claimed to discriminate the
two-subscriptions failure and did not. Fan-out is per subscriber, so a joiner
that opened its OWN second subscription still delivers the event to everyone
exactly as the test expected. Nothing separates one subscription from two
except counting them, which it now does at Redis, plus a duplicate-delivery
check for the second receive loop. Fails against the pending record not being
minted in the scan.
- The remedy test said "the old connection must also be gone" and waited for a
receive-loop exit. stopRedisSubscription does two things and the loop exits on
the first alone, so it passed against a version that cancelled the loop and
left the PubSub and its health check open. Counted at Redis now; fails against
exactly that mutation.
- The parallel-recovery test could not tell serial from parallel -- a serial
pass cycles all thirteen workspaces too. It now uses a rendezvous, asserts the
peak concurrency is above one AND within the cap, and fails against a serial
implementation.
- The prefixed-garbage test only exercised the classifier. Whether
receiveMessages ACTS on the error is a different claim, now driven through
the real Redis path.
- The metrics adapter test's comment said "every reason this bus can emit"
while subscription_unconfirmed was missing; its zero-assertion proved
non-leakage, not mapping. Emitted now with a count distinct from its
neighbour's, so a merging adapter cannot satisfy both.
PROSE THAT OUTLIVED THE CODE, again. The latency arithmetic still described the
single shared ticker that round 5 replaced with two independent loops; from
lastSeen [3T,4T) still holds, but from FAULT ONSET it is roughly [2T,4T)
because the publisher has its own phase. And a second "and replaces the
connection" in deployment.md that round 6's sweep missed.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* docs(events): correct three contract statements (codex r9)
Round 9 was cross-artifact conformance: every commitment the plan made was
checked against the code. All met -- wire classifier, lastSeen placement and
locking and install stamp and every-frame stamping, heartbeats bypassing
Publish and the shared counter, the drop-and-cycle remedy under the
single-establisher invariant, all four joint rules, the two-phase rollout with
its inverted-rationale test, and the reason/Help/deployment.md trio with the
rollout order. It also confirmed the three documented mutation survivors are
correctly dispositioned: both wsCounts checks are redundant-but-cheap under the
current invariant, and omitting the lastSeen.IsZero() skip is right because
adding it would mask a regression in the install stamp.
Three statements were wrong.
The env-var contract. My test comment said an unparseable PAD_EVENTS_HEARTBEAT
"must leave the flip off", which is true from a default config and false from a
config file that set it true -- there the value is left alone, as the
precedence test already asserts. The BEHAVIOUR is right and matches the epoch
flag: a typo must not move a migration in either direction, and silently
rolling an operator back to phase 1 would disable detection on a fleet that had
opted in with nothing saying so. Only the prose overclaimed, and it overclaimed
in the direction that invites someone to "fix" the ignore into a fail-closed
reset.
The constructor. NewRedisBusWithKeys documented publishEpoch and said nothing
about publishHeartbeat sitting next to it -- two adjacent booleans of the same
type belonging to two independent migrations, which is a shape that gets
swapped or dropped in a maintenance edit. Both now documented in order, with a
note that any combination is valid.
A stale count. EventSequenceResetsTotal's comment said "Five reasons" and there
are seven; it was already wrong by one before this unit added another. Replaced
with the count plus a pointer to the three artifacts that are authoritative and
move together, since the count itself is the part that goes stale first and is
read last.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): make the cadence arithmetic testable, and justify a guard pair
Matrix 5 (29 mutations, 21 detected) surfaced two things the previous run
could not, because both concern code the codex rounds added.
THE DRIFT FIX HAD NO TEST. Restoring the sleep-after-work form survived every
test in the package, and would have kept surviving: the only way to observe
drift through the loop is to time it, and a timing assertion is a flaky
assertion. Extracting nextTick makes the arithmetic checkable without a clock,
and the four cases now pin what the schedule is for -- a slow pass does not
push the next tick out, ten slow passes accumulate no drift, an overrun beyond
one interval resets instead of replaying the missed ticks, and an overrun
WITHIN one interval still catches up rather than re-phasing the schedule
permanently. Both directions mutation-checked.
The property is worth this much because breaking it is self-defeating rather
than merely untidy: an instance whose passes are slow emits heartbeats further
apart, its own subscription sees them further apart, and it crosses its own 3T
threshold and cycles connections that were never wedged.
A GUARD PAIR THAT ONLY DIES TOGETHER, which the team lesson says to treat as a
question rather than a clearance. The loop's ctx.Done select arm and its
post-wait ctx check each survive removal alone. Checked rather than assumed:
they cover disjoint moments and each is independently right -- the select arm
is the exit while WAITING, which is where the goroutine spends its life, and
the post-wait check stops a bus that closed DURING a pass from starting
another one against a cancelled context and a drained wsSubs. Removing BOTH is
detected. Reasoning recorded at the code, and the combined mutation added to
the matrix so the pair cannot quietly become a single point of failure.
Also fixed an ineffassign the lint gate caught in the new test.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* docs(events): state what the detector does not cover (codex r10)
Round 10 was adversarial: refute the unit's central claim rather than look for
defects in it. It partly succeeded, and the corrections are worth more than
most of the bug findings.
The claim was "a wedged connection is detected, coverage is ended, and the
connection is replaced so delivery resumes". Three parts of that were too
strong, and all three limits were checked against go-redis v9.22.0 rather than
argued:
IT IS A RECEIVE-SIDE DETECTOR, not a round-trip health check. It measures
whether frames ARRIVE. A subscription whose outbound direction is broken but
which still receives reads as healthy -- correctly, since nothing is lost, but
that is a narrower claim than "the connection is healthy".
IT CANNOT COVER THE PUBLISH PATH. PUBLISH travels on the client's connPool
while a subscription holds a connection from the separate pubSubPool
(redis.go:363, :1956) -- different sockets, different fates, and a reconnect of
one repairs nothing about the other. An instance whose publish path is wedged
loses its own events for every other instance and this feature will not say so.
That is a real gap in the family's coverage, now written down rather than
implied away.
REPLACEMENT IS ATTEMPTED, NOT GUARANTEED. If the path is still blackholed when
the cycle re-dials, the replacement cannot receive either. Coverage stays ended
so nothing is falsely claimed, but delivery resuming is a statement about the
network rather than about this code.
Filed BUG-2764 rather than folded in: establishSubscription's
`b.client.Subscribe(dialCtx, channel)` silently discards the SUBSCRIBE error,
because go-redis's own Client.Subscribe drops it (`_ = pubsub.Subscribe(...)`,
redis.go). A failed subscribe therefore installs a connection that looks live
and is subscribed to nothing. It is pre-existing, it lives in the establishment
path three bugs have already converged on, and changing how that function
issues its SUBSCRIBE does not belong in a diff about idle detection. Worth
knowing here because it is the one way the replacement can fail on a HEALTHY
network -- and because the detector now cycles it on the next pass, which is
why it self-heals on phase 2 and stays dead forever on phase 1.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): do not cycle a workspace that recovered before its turn (r11 P1)
Codex round 11 attacked three claims. Phase-1 safety and rollback safety both
came back clean -- a phase-1 receiver stamps lastSeen and nothing else, touches
no buffer, metric, client, ID or epoch, and its maintenance loop is not started
at all, so that timestamp is inert; heartbeats leave no state in Redis or
across a process replacement, and a mid-cycle shutdown rechecks b.ctx before
installing. The third claim did not survive.
FALSE POSITIVES ON A HEALTHY SYSTEM, which is the property this design cares
about most: cycling a working subscription drops its coverage and resyncs every
one of its subscribers for nothing.
cycleIdleSubscriptions selects its victims under the lock and releases it; the
cycles run afterwards. Its re-checks asked about generation, subscriber count
and bus liveness -- and never re-asked the question the scan had asked. A
subscription that started receiving again in that window was cycled anyway.
The window is not theoretical, and this unit widened it itself: the 8-way
concurrency cap added in round 5 makes a workspace wait behind earlier batches
of slow replacement dials, and a GC or CPU pause leaves a backlog of heartbeats
undrained in the receive loop. Both are ordinary conditions on a loaded box.
cycleOne now validates, ends coverage and tears down WITHOUT RELEASING THE LOCK
in between, which needed dropWorkspaceCoverage split into a locked variant that
returns its reason for the caller to report after unlocking. That also removes
the ordering fragility the previous version documented rather than fixed: there
is no longer any window in which coverage is ended for a workspace this
function then decides to leave alone. The log moved after the decision for the
same reason -- it could previously describe a cycle that then abandoned.
The freshness term is load-bearing and says so, next to the three neighbouring
terms whose mutation survivals are recorded as redundant-but-cheap. Removing it
is detected, by a test that lands the recovery in the exact gap through a new
positional seam.
NTP steps were checked and are not a hazard: time.Time carries a monotonic
reading, so a wall-clock step cannot make a subscription look idle.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* perf(events): take logging and PubSub.Close off the global lock (codex r12)
Round 12 verified round 11's freshness fix: validation, coverage invalidation
and teardown are atomic under b.mu with no lock cycle,
dropWorkspaceCoverageLocked preserved the original semantics exactly including
the no-buffer branch that still signals subscribers, reset reporting happens
after unlocking, and the replacement metric still lands only when a new
generation does. Slow establishment stays outside b.mu, wg.Wait only delays the
next pass, and Close cancellation retires pending records.
Two P2s, both about what round 11 put UNDER that lock:
slog.Warn ran while b.mu was held. slog invokes the installed handler
synchronously, and b.mu is the lock every fan-out and every Subscribe on the
instance contends for -- a slow or custom handler stalls all of them, and one
that calls back into the bus deadlocks. Moved after the unlock; it still has to
come after the DECISION, for round 6's reason, so both constraints are now
stated together at the call.
PubSub.Close ran under b.mu too. It takes go-redis's own mutex, which the
health check can hold across reconnect work, so a network-bound wait sat inside
the instance's hottest lock. That was survivable when teardown only happened as
a workspace lost its last subscriber; the idle detector makes it happen on
every cycle, which is what turned a latent cost into a real one. Handed off to
a goroutine: nothing references the PubSub once the map entry is gone, and
cancel() -- which is what actually stops delivery -- still happens under the
lock.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): do not read our own failed probe as a dead peer (codex r13)
Round 13 asked for a production-approval review. Four findings; the second is
the sharpest of the whole run because it is the mirror image of the failure
this feature exists to find.
A FAILED HEARTBEAT PUBLISH WAS READ AS A DEAD SUBSCRIPTION. The detector's
inference is "we published a frame and nothing came back, so the receive path
is dead" -- valid only if the publish actually happened. PUBLISH travels on the
client's connPool while the subscription holds a connection from the separate
pubSubPool, so a publish-side failure (pool exhaustion, a wedged outbound
route, Redis refusing writes) says nothing about whether that subscription can
receive. The detector was reading its own inability to probe as evidence about
the peer, and tearing down healthy connections on a schedule: a resync for
every subscriber of every workspace, every 90s, for as long as the outbound
path stayed broken. The third load inversion this unit has had to fix.
redisSub.lastProbeOK now records the last SUCCESSFUL publish, and detection is
suspended while it is stale -- checked in the scan and again in cycleOne, which
is a pair that only dies together and is therefore justified at the code:
the scan's keeps a workspace off the due list so no record is minted and no
joiner waits, cycleOne's covers the probe failing AFTER selection, a window the
concurrency cap makes real. Neither subsumes the other; removing both is
detected. New counter pad_event_heartbeat_publish_failures_total, documented as
DETECTION DEGRADED rather than as a peer being broken.
THE END-TO-END TEST THAT DID NOT EXIST. Every other test drives this through a
fake clock -- necessary, since the threshold is 90s by construction and
miniredis always answers, but it means they all ASSUME the wedge rather than
produce it. A TCP proxy that stops delivering server->client on the connections
already open, while writes keep succeeding and new connections stay healthy,
produces the real thing. The test asserts both halves of the claim: the wedge
is detected, and the replacement delivers. Both halves mutation-checked
(detector disabled; drop-only with no replacement).
The proxy's first version was vacuous -- a global flag consulted at read time
meant re-enabling delivery for future connections also revived the ones meant
to be dark. Per-connection now, and the comment says why.
Also: PubSub.Close taken off b.mu in Close() too (round 12 fixed only the cycle
path), and the replacement counter now takes an explicit installed result from
establishSubscription rather than inferring one from the live generation --
inference misattributed an unrelated caller's fresh subscription as this
cycle's replacement, and missed a real replacement that had lost its last
subscriber. Both mutation-checked.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): bind the probe stamp to a generation; make the proxy test honest
Round 14 returned a BLOCK verdict on two P2s, both mine, both in the fix that
round 13 had just added.
lastProbeOK WAS NOT GENERATION-BOUND. publishHeartbeats snapshots the workspace
list, publishes off the lock -- for as long as go-redis's timeouts allow -- and
then stamped whatever subscription occupied that workspace by the time it
returned. A probe sent for generation A could credit generation B, which never
received one; if later probes then failed, B could be cycled while looking
recently probed. Exactly the hazard stampLastSeen already guards on the same
map, and I did not carry it across. The generation now travels with the
snapshot and is validated before stamping.
THE END-TO-END TEST COULD PASS WITHOUT EXERCISING WHAT IT CLAIMED. It darkened
the receive direction of every open connection, including the ordinary pooled
connection PUBLISH uses -- so the probe may have been failing too, and the run
would then have been exercising the cannot-probe path rather than a half-open
route, which is the very distinction round 13 added the premise check for. The
proxy now classifies connections as it forwards and darkens only one that has
carried a SUBSCRIBE, leaving the publish path healthy, and the test asserts
zero probe failures so a run that drifts back into the other case fails loudly
instead of passing quietly. Still fails against a disabled detector and against
drop-only.
Also covered the new counter's mapping in the metrics adapter test, with a
count distinct from both neighbours -- cycled, idle_timeout and
heartbeat-publish-failure say three different things and an operator acts on
the difference.
Verified by the same round: install-time stamping does not permanently suppress
detection, establishSubscription returns false only on abandon and true on all
three installed paths including the cancelled-establisher goroutine, and
Close's deferred PubSub.Close runs after the unlock.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): pin the probe-across-replacement race (closes r15's residual)
Round 15 returned CLEAN and approve-with-comments, naming one residual: the
generation binding on lastProbeOK had no deterministic test, only the argument
that it mirrors stampLastSeen. This closes it with a positional seam between
the publish and the stamp, which is the only place that interleave can be
forced.
TWO INSTRUMENT DEFECTS ON THE WAY, both caught by mutation rather than by
reading:
The first version compared the credited stamp against the PROBE's timestamp.
On a frozen clock the replacement's install stamp and a wrongly-credited probe
are the same value, so it could not tell them apart -- it failed on the install
stamp while claiming a credit had happened, and removing the generation binding
still passed. It now compares against what the replacement was INSTALLED with,
and the clock advances inside the seam so a buggy write lands strictly later.
The second version was FLAKY: 2 failures in 3 runs. The heartbeat that was just
published comes back through miniredis on another goroutine, and if it lands
between the forced-stale write and the scan it refreshes lastSeen, the
workspace is not due, and no replacement happens. Retried until the generation
actually moves. Now 5 of 5 green unmutated and 5 of 5 detected mutated -- which
is the bar, because a 2-in-3 detector reads as coverage while being noise.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): on-call signals — log the cycle outcome, correct two claims (r16)
Round 16 read the diff as the person paged at 3am. Four findings.
THE CYCLE LOGGED ITS ATTEMPT AND NEVER ITS OUTCOME. The line says "attempting
to replace", which is correct and, on the one path where the replacement does
not happen, left an on-call with a warning, no counter movement, and no
explanation. Now there is a second line naming the reason.
pad_event_receive_loop_exits_total's documentation was falsified by this unit
and neither doc site said so: every idle cycle stops a receive loop while its
subscribers are still connected, and the comment still claimed exits happen
only at shutdown or when the last subscriber leaves. Both sites corrected, with
the expectation that it tracks the cycle counter during an incident.
A CLAIM I MADE AND THEN COULD NOT SUPPORT, recorded rather than quietly kept.
Round 16 argued the age-based premise check ("has a probe succeeded within the
threshold") failed to suspend detection where an ordering rule ("has a probe
succeeded since anything last arrived") would, and I rewrote the rule on that
argument and wrote a test named for the defect. The mutation matrix then
refused to confirm it: reverting to the age form leaves the test green, and so
does removing both copies of the check, and no case separates the two — on any
healthy path the two stamps advance together, because a probe whose frame
arrives sets both, and they diverge only on the wedge where both forms cycle.
The ordering rule is kept, because it states the intent exactly and is never
weaker. But the test and the comment now say what they actually establish —
that a probe which has started failing stops the detector concluding from
silence, which is the property both forms share and neither had before — rather
than claiming a fixed defect I cannot demonstrate.
The two remaining P2s are already-filed residuals: the cycled counter proves an
install rather than a working replacement (BUG-2764), and repeated cycling
amplifies /changes load with no jitter or global budget (BUG-2761). Both are
documented in deployment.md with their refs.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* docs(events): record what the final matrix actually says about four guards
Final matrix: 34 mutations, 22 detected, baseline restored green. Every
survivor is now documented at its line with why it survives, and two of them
turned out to be instrument defects rather than coverage gaps.
lastProbeOK's INSTALL STAMP IS REDUNDANT and the comment claimed otherwise. It
said a zero value "would permanently disqualify a subscription from ever being
cycled" -- true of the age-based premise it was written for, false under the
ordering rule that replaced it, because a zero value fails
`lastProbeOK.After(lastSeen)` exactly as an install stamp equal to lastSeen
does. Kept, for a reason it earns: it makes the field's invariant true by
construction, so a future rule reasoning about this value's AGE gets a real
timestamp rather than 1970 -- which is the trap the age-based rule fell into
one field over.
THE TWO cycleOne ABANDON GUARDS DIE ONLY TOGETHER AND ARE NOT REDUNDANT, which
took checking rather than assuming. They catch different shapes of the same
recovery: an arrival that has not been re-probed pushes lastSeen past
lastProbeOK so the premise case fires and the freshness case is unreachable --
that is the shape the test produces, and it is why removing either alone stays
green. But the publisher runs on its own goroutine at its own cadence and can
land a successful probe between the arrival and the decision, putting
lastProbeOK ahead again; there only the freshness case stops a healthy
subscription being torn down. Deleting it on the strength of the matrix would
remove the second shape's only guard.
Close's off-the-lock PubSub.Close is UNTESTED BY DESIGN, recorded rather than
papered over. It is a contention property, and the only assertion that
separates it is a timing one, which in this suite is a flaky one.
TWO HARNESS DEFECTS, both of which produced false survivors that would have
gone into the evidence package as findings. M11a inserted its mutation AFTER
the gate it was meant to disable -- unique anchor, wrong placement, so the
early return still fired and nothing changed; with a correct anchor it is
detected. M20 left variables unused and came back BUILD-BREAK rather than
answering; in compiling form it genuinely survives, consistent with
establishSubscription's abandon path already covering it.
The lesson worth keeping: when I rewrote all 34 anchors against current source
I verified each matched exactly ONCE, and uniqueness is not placement. An
anchor can be unique and still land somewhere that changes no behaviour.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): barrier the probe test on delivery — it was flaky, CI caught it
Go (PostgreSQL) failed on
|
||
|
xarmian
|
72336aacb5 |
fix(events): release SSE admission slots when a client leaves mid-establishment (BUG-2749) (#1186)
`GET /api/v1/events` reserved its admission slot, then blocked in `SubscribeAndReplaySince` while the workspace's Redis subscription was dialled and — since BUG-2747 — acknowledged. Nothing propagated the request's cancellation into that wait, so a client that disconnected during establishment left a process-wide slot, a per-principal slot and a per-workspace slot held for the whole of it. The connection was gone; the capacity was not. Cancellation is now DEREGISTRATION, and `wsCounts` — which already answers "is anyone still here" — decides everything downstream. No ownership hand-off and no reaper: the arbiter already existed. (One thing IS handed off, and only one — the remainder of the confirmation wait; see below.) The two cancellation positions take different paths, and only one of them owes the joiners anything: - Before the install: the existing post-dial critical section already abandons and retires correctly when nobody is left. It needed one ordering rule — the departed establisher stops being counted IN THAT SAME SECTION, before the count is read. If joiners registered while we dialled, the count is still non-zero and they get the subscription; that is the hand-off the filing asked about, expressed as a count rather than a transfer of ownership. - During the confirmation wait: the subscription is already installed with its receive loop running, so the connection is not at risk — but the WAIT is what releases the joiners, and dropping it would admit them into a subscription Redis has not acknowledged while telling them nothing. That is BUG-2747's defect re-created at the seam between the two designs. So the remainder of the wait moves to a goroutine that finishes exactly as the caller would have: same arms, same `markUnconfirmedAdmission` on the bound, same `finishPending`. Bounded by `confirmTimeout`; no reaper needed, because teardown stays count-driven. A departure is not a refusal. `ok bool` is replaced by a `SubscribeOutcome` enum across the three `EventBus` Subscribe methods, so `SubscribeWorkspaceLimit` and `SubscribeCancelled` cannot be collapsed: answering a departed client with 429 would have written a limit refusal into the logs and counters that anyone would use to tune that limit. An enum rather than a second bool or an error because the switch has to name the case — by construction rather than by argument. Caller population, with its search boundary: 3 production implementations (events.MemoryBus, events.RedisBus, metrics.InstrumentedBus), 1 test double (server.gapEventBus, which embeds the interface), 2 production call sites (both in handlers_events.go). Searched this repo four ways — the three method names, `.Subscribe(`, method declarations, and interface embedding. collab.OpBus and watchevents.Bus are different interfaces and are out of scope; no other repo links this package. WHAT THIS DOES NOT FIX, verified in go-redis v9.22.0 rather than inferred from its doc comment (which says Subscribe "does not wait on a response from Redis" and so reads as though no dial happens on the request path — it does; only the reply is unawaited). On plaintext, dialConn derives its per-attempt deadline from the caller's context and the default dialer is net.Dialer.DialContext, so cancellation aborts the dial. Under TLS the same dialer calls tls.DialWithDialer, which takes no context, so the dial stays bounded by DialTimeout alone. On a TLS deployment this shrinks the held slot from (dial + confirm bound) to (dial), not to zero. Review round 2 (codex) found a P1 in this unit's own first draft, of exactly the shape the filing warned about. A cancellation check at the top of the establish loop could return while the caller still OWNED an unretired establishment record: section 1 had already named it the establisher, so the record stayed in pendingSubs with nobody behind it, its done channel never closed. The next subscriber for that workspace would join it and wait forever — and its own registration keeps wsCounts non-zero, so no later caller would establish either. A permanently dead stream that looks alive, produced by a guard whose only purpose was to save a dial. The guard is gone: a cancelled caller now goes THROUGH establishSubscription, which is the only code that knows how to put the record down. Regression test included, and reinstating the guard turns it red. Round 2 also found a P2 shutdown regression: routing the dial to the caller's context alone took away Close()'s ability to interrupt a stalled dial, which it had before. The dial now runs on a context ended by EITHER the caller or the bus, and each half is pinned by its own test — dropping either one is detected. Review round 1 (codex): no P1. One nit fixed as a class — three comments elsewhere in the file asserted the dial was "NOT bounded by the context we pass", which this change falsified; the sweep found and corrected all three (establishSubscription, defaultSubscribeConfirmTimeout, Subscribe). The TLS half of its P2 is filed as BUG-2754: the fix belongs at client construction, where it covers every Redis call rather than this one. Class sweep filed separately as BUG-2751 (lead-ruled: one region, one design per diff): internal/watchevents has no per-request establishment, but its resume path blocks on a 250ms settle window bound to the bus's context rather than the request's, while /api/v1/events/stream holds the same admission slots across it. Tests: five cancellation cases in internal/events (before install, during the wait alone, during the wait with a joiner, a cancelled joiner, an already-dead caller), a dial-binding assertion, and the handler-level binding in internal/server asserting the admission slot itself is released — the half of the bug that does not live in the bus. Mutation matrix, 8 mutations: 7 detected, each by the test named for it. The one survivor is the ctx term in the retry re-decide, and it survives because it is an OPTIMISATION rather than a correctness guard — a departed caller that mints a second record still establishes, deregisters and retires correctly; the term only saves a pointless dial. The code says so rather than implying the guard is load-bearing. The earlier draft's entry guard and loop-top break formed a redundant pair the matrix could only detect when both were removed. That redundancy was the smell, and round 2 found the substance under it: one of the two was not redundant, it was wrong. With it gone the entry guard is detected on its own. |
||
|
xarmian
|
cbf2dd29c8 |
test(events,metrics): assert exactly-once, cover the new counter, drop an overclaim
Codex round 7. The confirmation bound's comment said it bounds establishment. It does not: the dial and go-redis's HELLO/AUTH handshake run inside client.Subscribe before the timer starts and are bounded by the CLIENT's DialTimeout instead, so the worst case composes to roughly DialTimeout plus this. Anyone reasoning about connect latency needs both numbers. The concurrency test asserted topology — subscriber count and pendingSubs — while reading one event per channel, so a duplicate from a second establishment could sit in the channel undetected. It now asserts exactly-once, which is the behaviour the topology was standing in for. The new Observer method, counter and deployment contract had no adapter test. Added, including the half that matters: the count must NOT also land on the reset series, since an adapter that merged them would pass a total-only assertion while destroying the distinction an operator acts on. And the abandonment test now says out loud that it fabricates its state, because the establishing caller is blocked inside Subscribe and nobody can unsubscribe it — the same reason the retry it exercises is defence in depth rather than a reachable path. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
4d84453f41 |
fix(events): apply the registration bound to the buffer, before the cursor filter
Codex round 5. The P1 is the same two-spaces mistake the ID-valued ceiling made, committed again inside the fix for it: asking since() for the events above the cursor and then dropping the last (appends - mark) of them mixes a filtered list with an unfiltered count. A post-registration straggler whose id falls at or below the cursor is absent from the slice but still counted in the drop, so the count eats a legitimate pre-registration event instead. Pre-mark [5 30 20], post-mark [6 40], cursor 10 handed the caller [30] and lost 20. replayBuffer.sinceBounded applies the window to the BUFFER and shares every coverage rule with since(), which now delegates to it, so the two cannot disagree about what cannot-vouch means. This also closes the residual round 3 left accepted: if the wait's appends evict everything the buffer held at registration, keep goes to zero and the span is refused rather than partially served. Three overclaiming comments corrected — the wait is bounded, and saying Subscribe returns only once Redis has acknowledged is false on the timeout path. And the ceiling test's own claim: it appends straight to the buffer, so it pins the boundary arithmetic and says nothing about replay-XOR-channel, which is a different test. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
f5ca67cbca |
perf(events): shorten the subscribe-confirmation bound to a measured 1s
Lead ruling on codex round 3's second finding: file the ctx-plumbing as its own unit (BUG-2749), bound the exposure here. The bound is not a guess at how fast Redis is. Establishment either completes in single-digit milliseconds or does not complete at all, so past the top of the fast mode waiting longer buys nothing and only holds an SSE admission slot, global and per-workspace, for a client that may already be gone. Measured on a containerised Redis over loopback, 300 establishments, timing the whole of Subscribe: p50 388us / p99 679us / max 1.73ms idle; p50 693us / p90 5.1ms / p99 12.1ms / max 18.5ms under 24 busy loops on 8 cores. One second is ~54x the loaded maximum. Being too short costs an admission whose coverage this instance cannot describe — counted, logged, and reconciled to the client when the acknowledgement lands. Waiting is the silent direction. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
650d9df270 |
fix(events): bound the replay by append POSITION and buffer identity
Codex round 3. Both P1s real. The ceiling used lastAppendedID as if it were a time boundary. This bus's ids come from a counter shared across workspaces and a phase-1 publish assigns and publishes in two calls, so arrival order and numeric order genuinely disagree. Against an id-valued bound both directions break at once: a straggler arriving after registration is replayed although it also went to the caller's channel, and a pre-registration event carrying a higher id is filtered out and never replayed at all. replayBuffer now counts its appends, and the bound is a position — the entries to withhold are simply the final (appends - mark) of whatever since() returned, which may trim from the front but never the back. The mark also carries the BUFFER, not just a position in it. An ID-space reset during the wait replaces the buffer wholesale; a position in the old one describes nothing in the new one, and knownFrom may still accept an adjacent cursor, so the mismatch does not announce itself. Also corrected, all found by the same round and all mine: the Observer comment claimed this counter never reaches SequenceReset, which the late-confirmation path contradicts; the reason enumeration in metrics.go, its Help string and docs/deployment.md were never updated for the sixth reason; and both the metric and its comment said every increment is a client when it is one establishment however many subscribers were waiting. Accepted, not fixed: since() evaluates eviction over the whole buffer including post-registration appends, so a flood inside the wait can evict a cursor that missed nothing and force a sync_required. It costs a spurious resync, never silent loss, which is the direction this family chooses every time. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
cadf0fab33 |
docs(events): name the joiner retry as defence in depth, and make it loud
The mutation matrix could not reach it: no mutation of the surrounding code makes a test take that path, because the same-lock retire in the abandon path makes the strand unreachable rather than recoverable. A joiner increments wsCounts under b.mu before the establisher's count check reads it, so a registered joiner prevents the abandon; a joiner arriving after the check cannot find the record, because it is gone in that same section. That is an argument, not a measurement, so the retry stays — a permanently dead stream that looks alive is worth one wasted pass in a case that should never happen — but it now logs when it fires, so a wrong argument surfaces in production instead of limping silently. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
1f4ab9b549 |
fix(events): abandon and teardown must not strand a joiner or leak a PubSub
Codex round 2, lifecycle angle. Both findings real, plus one my own fix introduced. P1 — Close cancels the context before it takes the lock and drains wsSubs, so an establishment that locked afterwards installed into a map Close had already emptied. Its receive loop exits on the cancelled context and neither subCancel nor pubsub.Close ever runs: the PubSub and its health-check goroutine outlive the bus. establishSubscription now refuses to install into a closing bus, and Close clears wsCounts alongside the subscribers it counts, so the two structures cannot disagree. P1 — abandoning because the workspace emptied retired the establishment record in a separate critical section from the decision. A subscriber arriving in between registered, waited on a promise nobody would keep, and returned with a channel wired to nothing — permanently, since its own registration keeps wsCounts non-zero so no later caller establishes either. The record is now retired under the same lock as the decision, and a joiner verifies a live subscription afterwards rather than assuming one, taking the establishment over once if there is none. And one I introduced writing that: the re-check created a pending record on the loop's final pass with nobody left to establish behind it, which is a worse version of the same defect. Records are now created only at the top of an iteration that will use them. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
c719bf44b1 |
test(events): make the confirm-versus-timer race deterministic
The repetition version caught the mutation that removes the confirmClosed re-check in 0 of 10 runs at 500 establishments each: a near-zero bound makes the timer win outright far more often than it ties, and winning outright is the ordinary timeout path, not the race. A one-in-ten detector reads as coverage and is not. beforeUnconfirmedMark holds the mark until the acknowledgement has landed, reproducing the interleave every time. 10 of 10 against the same mutation. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
703e746922 |
fix(events): bound the replay at registration instead of withholding fan-out
Codex round 1, three findings, all real. P1 — a subscriber arriving mid-establishment was admitted immediately. establishSubscription installs wsSubs and only THEN waits for the acknowledgement, so for that interval the workspace looks live and is not. Reading wsSubs first let a second subscriber straight into the unconfirmed window this change exists to close. pendingSubs is now checked first. P1 — withholding fan-out from a not-yet-admitted subscriber dropped events for the very population BUG-2747 is about. A fresh subscriber (sinceID == 0) reads no replay at all, so an event skipped on the theory that the replay would carry it was skipped and then never replayed. Replaced with a replay CEILING captured at registration: the subscriber is live in fan-out from the moment it registers and receives everything after that on its channel, while its replay is bounded above by what the buffer held then. That is the same division the single critical section gave for free, generalised to the case where a wait separates the two halves. P2 — the confirm timer could set unconfirmedAdmitted after the acknowledgement had already cleared it, leaving a subscriber counted as unconfirmed and never told to reconcile. markUnconfirmedAdmission now checks confirmClosed under the lock that closes it. Two tests added for the two P1s. The double-delivery test moved from the establishing caller to a JOINER, because the establisher can never exercise it: establishing implies no live subscription, losing the last subscriber deletes the buffer, so its replay is always nil and there is nothing to duplicate. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
739f573b3b |
docs(events): the namespace helper's comment described a window this fix closed
It said the wait was covering up a production window with no remedy, pointing at BUG-2747 as where it was tracked. That is now the fixed thing, so the comment was teaching the next reader something false about the bus they were looking at. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
4148278ef6 |
test(events): drive the double-delivery window through the gap it actually opens
The first version hung the publish off afterSubscribeRegister, which now runs under b.mu in the second critical section — so the fan-out it was meant to race could not happen until the lock was released and the subscriber was already admitted. It passed against a mutation that delivers to unadmitted subscribers, which means it was not testing the flag at all. afterSubscriptionConfirmed opens at the real gap: acknowledged, so events arrive, and before the establishing caller re-acquires b.mu to read its replay. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
ad8853f0bc |
test(events): assert the widened-window premise after the defect assertion
Checked before the publish, it races the unfixed code — which returns from Subscribe while the SUBSCRIBE is still on its way into the proxy — so the test reported a broken instrument instead of the defect. Last, it can only fire on a vacuous pass, which is the thing it is for. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
548b31588b |
fix(events): await Redis registration before admitting an SSE subscriber
RedisBus.startRedisSubscription wrote the SUBSCRIBE and returned. Events Redis processed before it registered the subscription reached nobody — and since this bus has no local fan-out, that includes events published by the very instance serving the client's stream. A resuming client was already protected: a first subscription has no replay buffer, so eventsSinceLocked returns nil for sinceID > 0 and the handler emits sync_required. The silent loss reached only sinceID == 0 — the SubscribeIfAllowed path — whose buffer coverage begins at the first event that DOES arrive, leaving the hole below anything the buffer ever claimed. Subscription establishment now runs OUTSIDE b.mu (BUG-2748) and Subscribe does not return until Redis has acknowledged it. All three entry points share one body, subscribeAndReplay, so none can drift. The confirmation is signalled from inside receiveMessages rather than by a Receive placed ahead of it. That loop treats its first *redis.Subscription as the initial acknowledgement and every later one as a RESUBSCRIPTION that ends coverage (BUG-2739); consuming the first with an earlier Receive would leave it swallowing the first genuine resubscription instead. Splitting the register and the replay read into two critical sections put SubscribeAndReplaySince's replay-XOR-channel guarantee at risk, so a subscriber is now REGISTERED but not ADMITTED across the wait: fan-out appends to the buffer and skips its channel, and the replay read that follows delivers it. The flag's zero value is admitted, so MemoryBus and the already-live fast path are unaffected by construction. Failure path admits rather than refuses — every subscriber was admitted into an unconfirmed subscription before this, so refusing would be new strictness that turns a Redis blip into failed connects. Instead the span is reconciled to the client through BUG-2730's mid-stream signal when the acknowledgement lands, and counted for the operator via a new pad_event_subscription_unconfirmed_total. BUG-2747, BUG-2748 Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
86277915e1 | Merge branch 'main' into fix/generation-key-corruption-guard | ||
|
xarmian
|
9a8e41f8ab |
test(events): keep invariants in the comments and process history in the trail (BUG-2742, codex round 12)
Asked to critique the change for scope and restraint. Accepted: several comments carried trial-specific numbers and round labels — catch rates, CI counts, which review round corrected whom. Those belong in commit messages and on the item, which is where someone looking for the history will go; a code comment that quotes a measurement acquires a maintenance obligation nobody will honour. Trimmed to the invariant and the reason, with a pointer to BUG-2742 for the figures and, where it matters, the instruction not to trade rounds back for a bigger burst. Accepted: subscriberChanDepth's comment described a test premise before it described what the number means in production. Reversed. Declined, three, with reasons since two of them argue against earlier rounds of this same review: - That the waits on the two key-only namespace tests are unnecessary and add a failure mode. They are unnecessary for those tests' own assertions, which is why round 10 asked for them: those are the shortest Redis-bus tests in the package and therefore the ones a newcomer copies. The failure mode added is "registration never happens", which is a defect rather than a flake. - That this should be several units. It is one bug covering one family in one package, and the item asks for the family swept together rather than test-by-test. The steps are separate COMMITS, which is the separation that helps a reader. - That the compile-time premise check is over-engineered next to a runtime one. Its whole value is failing EARLIER than a run: the failure it guards is the test going silently vacuous, which a runtime check inside the vacuous test cannot reliably announce. |
||
|
xarmian
|
bfaa1d88a9 |
test(events): make the nearest example safe to copy (BUG-2742, codex round 10)
Asked whether this package now makes the correct thing easy and the flaky thing hard for someone adding a Redis-backed delivery test. It did not. The two shortest Redis-bus tests — the ones a newcomer copies — still read Subscribe, then Publish, with the registration wait afterwards or absent. Neither NEEDS the wait, because both assert on Redis keys, which a publish writes whether or not anyone is listening. But being correct for a reason particular to themselves does not stop them teaching the race to whoever copies their shape, so both now wait immediately after subscribing and say why in one line. The usage rule was also in the wrong place: waitForSubscribers' own comment described what it polls, while the warning about WHEN to call it sat on pollSubscriberCount, an unexported helper no caller has a reason to open. The rule now leads the comment on the function people actually call, along with the case that does not need it, so the exception is visible rather than inferred from a test that omits it. |
||
|
xarmian
|
7378707350 |
test(events): prove the replacement subscription is live by getting an event back (BUG-2742, codex round 9)
Round 8 left the post-reconnect publish protected only by the 2s poll above it, documented as margin by accident. Asked whether anything in the package could still redden CI, Codex went straight there: on a slow, -race or single-core runner the reconnect may still be pending after 2s, the publish is dropped, and drain times out. A real flake, and documenting it is not fixing it. The construction that works is the one the failure itself suggests. Counting subscribers cannot distinguish the stale registration from the replacement, and waiting for the count to fall can hang — both measured in round 8. But an event ARRIVING is proof that cannot be satisfied by the dead connection. So the test publishes until one comes back: instant when the subscription is already live, and incapable of passing early when it is not. The extra events some rounds publish are harmless. What the rest of the test needs is a buffer with something in it, not a buffer with exactly one thing in it, and the assertion that follows — that a reconnect WITH a buffer does report a coverage break — is unchanged. This also removes a vacuity I flagged in round 8 and had put out of scope: the 2s poll was the only thing establishing that a reconnect had happened at all, so a slower reconnect would have made the first assertion pass without ever reconnecting. The loop makes that premise explicit and fails loudly when it does not hold. |
||
|
xarmian
|
6ba8e8dd79 |
test(events): retract the reconnect wait, which guaranteed nothing (BUG-2742, codex round 8)
Two findings from reviewing the new test code as production code. The phase-two collector read `msg := <-incoming` without checking whether the channel was still open. A closed pubsub channel hands back a nil *Message, so the next line panicked the test binary with a stack instead of naming the failure — in the one goroutine whose entire job is to report what went wrong. It reports the close now. The second retracts my own round-4 change. I added waitForSubscribers before the post-reconnect publish and claimed it turned safety by accident into safety by construction. Measured, that claim is false: across the cut the subscriber count goes 1 -> 0 -> 1 over a couple of milliseconds, because the STALE registration still reads 1 until miniredis notices the closed socket. A count-based wait placed after the cut returns immediately on the dead subscription and guarantees nothing. Waiting for the 1 -> 0 transition instead is not available either: in one trial of five the count never dropped inside 3s, so that wait can hang. What actually protects that publish is the 2s poll above it, which runs to completion whenever no reset is reported — the case this test asserts. That is margin by accident, and the comment now says exactly that, with the measurement, rather than carrying a wait that reads as a guarantee. Worth its own note: that same 2s poll is how the test establishes a reconnect happened at all, so if a reconnect ever took longer the test would pass vacuously. Real, out of scope here, recorded on BUG-2742. |
||
|
xarmian
|
f6a58b6b80 |
test(events): finish the depth migration, close the go-redis bound, correct a rationale (BUG-2742, codex round 7)
Asked to judge the accumulated result rather than the sequence. The depth-constant migration was HALF DONE. I converted the two tests the previous round named and never enumerated the rest, leaving four literals in two more tests — including the fanOutLocally boundary cases, whose whole point is the event at depth+1. A partial migration is worse than none: it leaves two groups of tests disagreeing about the same number. All six sites derive from subscriberChanDepth now, and a grep for the literal comes back with only the prose that quotes the CI failures. The phase-two ordering test kept a bound I had described as unavoidable. It is not: ps.Channel takes redis.WithChannelSize, so sizing it above the message count makes truncation structurally impossible instead of merely unlikely, which is the same standard applied to the sibling test rather than a weaker one. The early reader stays, because the two defend different things — the size stops the buffer truncating the sample, starting early stops the test pretending a burst is consumed instantly. Re-measured against the two-call INCR-then-PUBLISH mutation after the change: 8 of 8, unchanged. And a rationale of mine was simply false. I wrote that the old probe-publishing helper was harmless because every call site ran before the first publish. Four of them run after one, so it really was ending coverage on live buffers. It was harmless for a thinner reason — those four go on to assert about Redis KEYS rather than about coverage — which is a property of what each caller happens to assert next, and stops holding the moment someone adds a coverage assertion after a wait. The comment says the true thing, which makes the switch to PubSubNumSub better motivated than my own argument for it was. |
||
|
xarmian
|
0d501761b9 |
test(events): derive the drop-boundary tests from the depth, and name what the waits stop testing (BUG-2742, codex round 6)
Asked what a rewrite might have silently REMOVED. Declined, on checked grounds: that the new ordering test no longer overflows the subscriber channel. That path is covered better elsewhere than the old test covered it — TestSlowConsumerDropsEvents asserts exactly a buffer's worth survives an overfill, and TestDropSignalsOnlyTheSubscriberItHappenedTo asserts the boundary itself, that filling the channel raises nothing and one more raises the flag on the slow subscriber and not the fast one. The old ordering test exercised drops incidentally and asserted nothing about them. Both of those, however, restated the depth as a literal 64 — the duplicated premise the new constant exists to remove, in the two tests whose entire claim is about the boundary AT that depth. Derived now. Declined: that the phase-two test's per-message bound went 5s to 30s and so can no longer detect a 5-30s stall. It never asserted latency, and detecting a stall that way has the same load-dependence this whole change removes. The bound exists to fail a hang. Accepted and recorded rather than fixed: the registration waits mean these tests no longer exercise the lost-publish window at all. That window is real in production, where nothing waits, and the honest place for it is a filing rather than tests that fail once in about a hundred loaded runs — a detector nobody can act on. pollSubscriberCount now says so, and points at BUG-2747, which carries the asymmetry with internal/watchevents. |
||
|
xarmian
|
6c81cf923c |
test(events): make three comments true of the code they sit on (BUG-2742, codex round 5)
Read as the next maintainer, three claims did not survive. waitForSubscribers' own doc comment still described the mechanism this branch REMOVED — that a "probe" payload is published and the resulting unmarshal error is the confirmation. The helper reads PubSubNumSub now and publishes nothing. It also said Subscribe returns before the SUBSCRIBE reaches the server; go-redis writes it synchronously and simply does not await the server's confirmation. I corrected that mechanism in my own comments one round earlier and did not carry the correction to the pre-existing comment sitting on the function I had just rewritten. "Draining concurrently keeps the buffer from ever being the constraint" was overstated: a starved reader can still fall 100 behind. Unlike the sibling test, that bound belongs to go-redis rather than to us, so it cannot be made structurally impossible the same way — the comment now says so, and says to raise ChannelSize rather than lower the assertion if it ever does flake. "Load-independent" was too strong for the ordering test. The ASSERTION path is load-independent; detection is not, since whether a broken bus interleaves on a given round is still up to the scheduler. Scoped to the half that is true, which is also the half that fixed the CI failures. Declined: that the two earlier commit messages still contain the claims later commits correct. They do, and each correction names the claim it reverses, so a git log reads as the sequence it was. Rewriting them would erase a review round finding something real, and this branch's history is meant to show that. Then, applying the correction-must-travel rule to the repo rather than to the files a reviewer named: grepping the claim string turned up the same sentence in internal/watchevents/redis_bus.go, where it introduces the FIX — that bus waits for its SUBSCRIBE to be confirmed at construction. Which retires a bug I filed earlier today claiming watchevents shared this exposure (it does not; that is why it survived the load that broke this package) and sharpens the remaining one from an open design question into a named asymmetry. See BUG-2746 (wontfix, retracted) and BUG-2747. |
||
|
xarmian
|
8597d60884 |
test(events): close the two remaining instances of both shapes (BUG-2742, codex round 4)
My own sweep found the second shape exhaustively and anchored the first on the test CI had named. Codex, asked to look for both plus a third, found one more of each. Neither reproduced in 50 invocations under the load that made the already-fixed four fail 5 of 6 rounds, so these are shape fixes rather than observed-failure fixes. TestConcurrentPhaseTwoPublishesArriveInIDOrder published all 300 messages before anything drained ps.Channel(), which go-redis bounds at ~100. That makes the arrival count a function of how much CPU the reader gets — the same defect as the sibling test, with go-redis's buffer as the bound instead of the bus's. The reader now starts before the publishes, so the buffer is never the constraint. Still one reader, so recorded order is still arrival order, and the ordering assertion is unchanged. Re-measured against the two-call INCR-then-PUBLISH mutation this test exists to catch: 8 of 8, so the restructure did not weaken it. TestAReconnectOnAnIdleWorkspaceIsNotAReset publishes after a cut, which means after a re-subscription, with no wait for the replacement registration. It is safe today only because the 2s poll above it leaves room — and that poll exists to prove a reset was NOT reported, so it exits early the moment that changes, taking the margin with it. Safety by accident becomes safety by construction for the cost of a few milliseconds. Third shape: Codex found none, agreeing with my own enumeration. |
||
|
xarmian
|
79b0f23c45 |
test(events): strengthen the ordering test and correct the registration mechanism (BUG-2742, codex rounds 2-3)
Round 2, on the rewritten ordering test: - Deriving `each` from subscriberChanDepth by integer division silently yields 0 the day the depth drops below the publisher count, making total 0 and passing `count == total` as 0 == 0. The test would go permanently vacuous with no signal. The numbers are stated and a constant declaration fails the BUILD when they stop being compatible; verified by setting each = 4 and watching vet refuse it. - The assertion checked increasing ids, which a bus that dropped some of these and delivered something else with a bigger id also satisfies. One bus, one workspace and nothing else publishing means the assigned ids are exactly base+1..base+total, so the delivered sequence must be CONTIGUOUS. Re-measured against both mutations after strengthening: 25/25 each. - Two comments claimed FEWER events than the channel is deep; total equals the depth exactly. Still safe, since the channel starts empty and every publish fits, but the stated premise was wrong. Round 3 corrects a mechanism I asserted without reading go-redis. The SUBSCRIBE is NOT sent by the receive goroutine: Client.Subscribe calls PubSub.subscribe synchronously, which writes the command. What it does not do is wait for the server to confirm it, and the publish that follows travels on a different connection — so Redis can process the publish first. Same window, same fix, but the comment now says the true thing. Declined from round 2: that the test only samples scheduler behaviour rather than forcing the interleave. That is true and is why the comment quotes a measured catch rate rather than claiming a proof. Forcing it would need a pause point in Publish, and the property under test is precisely that no such point exists. |
||
|
xarmian
|
e1f256b48c |
test(events): wait for the subscription to register before publishing (BUG-2742)
RedisBus.Subscribe returns before Redis knows about the subscription: the SUBSCRIBE command goes out from the receive goroutine, which has not necessarily been scheduled yet. RedisBus.Publish has no local fan-out -- local subscribers are served by the receive path like any other instance's -- so a publish that lands inside that window is lost OUTRIGHT. Nothing replays a pub/sub message nobody was listening for. That is why these failures survive every timeout: the observed round trip is bimodal, either tens of milliseconds or never, and no bound distinguishes the second case from a hang. Under a busy-loop load the affected shape lost about 1 arrival in 100 and lost none once the wait was in place. Four tests published immediately after subscribing and then waited for arrival. CI named three of them; the fourth, TestAMixedPhaseDeploymentDelivers- BothWays, is in the same class and had not fired yet. It needs a COUNT rather than a presence check: its claim is that two replicas see each other's events, and waiting for "a subscriber" is satisfied by whichever registers first, so waitForSubscriberCount is added for it. waitForSubscribers stops inferring registration by publishing a probe and reads the server's state instead. The probe was a trap for the next caller: "probe" is not a decodable payload, so on a bus that reads the wire prefix it is an UNDECODABLE MESSAGE -- the exact condition several tests here assert about, injected by the helper they would call to arrange themselves. It was harmless only because an undecodable message ends coverage that EXISTS and every call site happened to run before the first publish. That is a property of the call sites, not of the helper, and adding call sites is what this commit does. |
||
|
xarmian
|
63c45bc651 |
test(events): buy ordering-race detection with repetition, not a load-dependent sample (BUG-2742)
TestConcurrentPublishesDeliverInIDOrder published 320 events while a goroutine drained concurrently, then guarded against vacuity with "at least half must arrive". That guard is a bet on runner speed. The bus DELIBERATELY drops for a subscriber that cannot keep up, so the arrival count is a continuous function of how much CPU the reader gets: across four CI instances on three PRs it reported 64, 65, 144 and 150 of 320, and 64 is exactly the subscriber channel depth -- one buffer-full and nothing after it. Any "require N of 320" line is a bet on the same variable, which is why this is not a threshold change. The sample is made untruncatable instead of large. No reader runs during the publishes and fewer events are published than the channel is deep, so a drop cannot occur; the vacuity guard becomes count == total, an equality rather than a threshold. Nothing in the assertion path waits on a goroutine being scheduled, so the result no longer depends on load at all. Detection power then comes from repeating the round, which is what the bounded channel could not provide. Measured against the mutation the test exists to catch (release replayMu after the append so fan-out runs outside it): the version replaced here caught it 9 times in 30 runs; this one caught it 25 times in 25, at about 10ms against the old fixed 0.5s. subscriberChanDepth is named rather than inlined so the test states its no-drop premise in terms of the real depth. A test that hardcoded 64 next to it would silently stop proving what it claims the day the depth changed. |
||
|
xarmian
|
a04912197a |
test(events): the clamp table needs no Redis (BUG-2740)
Five subtests each stood up a bus and a miniredis to assert arithmetic. The clamp's behaviour is a pure function of an int64; the servers bought nothing and cost five of them in a package whose timing-sensitive tests are already load-fragile (BUG-2742). Split into a pure table plus ONE end-to-end case, which is what stops the table being a unit test for a function nothing calls. Zero is the case driven end to end, because it is the one that reproduced the original failure: '0' fails the epoch guard's ^[1-9] and sent the script back into the repair for the same bad seed. Mutation-verified after the split: removing the clamp and removing its floor are both still caught, the first by the wired case and the second by both. This is a footprint reduction, NOT a fix for #1182's Go leg. See the trail — that failure is BUG-2742's family and I have not established that this branch causes it. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
161e60334e |
docs(events): why per-workspace backward detection is enough for a global counter (BUG-2740, codex round 8)
Two separate review rounds have now raised the same shape: the sequence counter is shared across workspaces, but counter_backward compares only against the affected workspace's high-water mark, so a restarted counter whose reissued ids are consumed by OTHER workspaces appears to bypass detection. Refuted by probe rather than by argument, and the reasoning is now where the next reviewer will ask it. The unit of coherence is the per-workspace buffer and the per-workspace cursor: an id can only corrupt a buffer by colliding with one already in it, and an id at or below that workspace's high-water mark is precisely what this arm catches. A reissued id landing in a different workspace collides with nothing. Both shapes measured. A counter DELETED restarts at 1, which takes publishScript's id == 1 branch and rotates the epoch unconditionally — the probe reported epoch_change and a refused cursor, so this arm is never reached. A counter SET backwards above 1 skips that rotation, and then the workspace whose own ids are reissued sees them as backward and lands here, while a workspace that never held them keeps a strictly increasing stream and correctly needs no reset. No code change; this is BUG-2736's mechanism, not BUG-2740's, and it is unmodified. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
99cb56f50a |
fix(events): a broken host clock must not make the repair fatal (BUG-2740, codex round 7)
The seed is an INPUT, and nothing validated it. An unset or misconfigured host clock reports zero or a negative second; the repair then SET that at the generation key and returned it as the epoch. The epoch guard rejects anything not matching ^[1-9][0-9]*$, so it rotated, called back into the repair, received the SAME bad seed, and assigned it to the epoch WITHOUT revalidating — putting an unparseable epoch on the wire, which every receiver rejects. A total, permanent drop, reached through the mechanism written to prevent one. clampGenerationSeed forces the seed into the shape the guards accept: a positive integer of at most 17 digits. The floor is 1 rather than anything cleverer, and it gives up the ruling's property deliberately — a broken clock cannot deliver 'above any counted history' at all, so the choice is between a value that is merely LOW and one that is FATAL. A low generation is detected as epoch_regressed and costs a round of resyncs; an invalid one drops every event until a human intervenes. The ceiling exists for next_gen's reason: a value over 17 digits is not usable as a generation, so seeding one would repair the key into a state the next rotation rejects. THE FIRST VERSION OF THE TEST WAS VACUOUS IN THREE OF FIVE CASES, and only the other two exposed it. Without clearing the epoch no rotation branch fires, so the repair never ran and the second publish just reused the epoch the fixture minted — which is 1, matching three of the expected clamps exactly. The two cases that expected something else failed and gave it away. Matrix: 3 applied, 3 caught — no clamp, floor removed, ceiling removed, each killed by the cases that specifically exercise it. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
0accca7a40 |
test(events,docs): pin the detection chain a colliding repair actually relies on (BUG-2740, codex round 5)
The repair seeds from wall-clock seconds, which is above any COUNTED history and is not a monotonicity guarantee. Corrupt the key twice inside one second and both repairs seed the same value, so two genuinely different id spaces carry the identical epoch — and an equal epoch means 'same space' by design, so neither epoch_change nor epoch_regressed fires. It is still not silent, and the reason is worth pinning because it is not the one the epoch mechanism suggests: a merge needs ids REUSED at a receiver, reuse needs the sequence to go BACKWARDS, and backwards is detected whatever the epoch says. counter_backward drops the affected buffers and refuses cursors below the discarded high-water mark. That was folklore until it was measured. The test drives the whole sequence — two repairs seeding the same value, a sequence reset between them — and asserts its own premise first (the two spaces really do share an epoch), that no epoch-based reason fires, that counter_backward does, and that the old cursor is refused rather than replayed the new space's events. Mutation-verified: stop reporting counter_backward and it fails. The docs carry the chain as a quoted rule, plus the two cases that look like it and are not — a counter set FORWARD is a jump inside one space with no reuse, and a receiver that never held the colliding range experiences a gap, which is BUG-2735's pre-existing class rather than anything this introduces. Lead re-ruled on the probed fact: residual ACCEPTED because it is detected, attribution corrected from epoch_regressed to counter_backward. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
e00b1d2a1a |
fix(events): the generation ceiling is one digit under the epoch ceiling (BUG-2740, codex round 4)
The two guards interact, and accepting 18 digits here made them fight. The value next_gen accepts is about to be INCREMENTED, and the result becomes the EPOCH — which the guard further down rejects above 18 digits. So a counter at 999999999999999999 was accepted, incremented into a 19-digit epoch, and the epoch guard fired: the script rotated a SECOND time inside one publish, found a 19-digit generation, repaired it to the wall-clock seed, and published a generation far below the one receivers hold. Measured rather than reasoned: seeded at 18 digits the published epoch came back as the seed; at 17 it came back as the ordinary increment. The ceiling for what is USABLE has to sit one digit under the ceiling for what is PUBLISHABLE, and it is now derived from that rather than chosen to match. THE FIRST VERSION OF THE TEST DID NOT CATCH IT. Reverting the ceiling to 18 passed, because 999999999999999999 lands on the seed either way — once by a single repair, once by an increment plus a second rotation. Identical end state, different mechanism, which is CONVE-12 exactly. The discriminating row is an 18-digit value whose increment STAYS 18 digits (100000000000000000): over the line at 17 and repaired, accepted at 18 and merely incremented, with no second rotation to disguise the difference. The precision test moved down with the ceiling, and its magnitude is now chosen rather than incidental: doubles are exact only to 2^53 (9007199254740992, 16 digits), so tostring divergence begins well BELOW the 17-digit ceiling — measured at 99999999999999998, which increments to 99999999999999999 in the key while tostring renders it 100000000000000000. The GET fix is load-bearing across much of the admitted range, not just at its edge. Also corrects the BUG-2744 note, which said the script's returned id is compared numerically on the Go side. It is not: this caller discards the result with .Err(), so a wrong id reaches the wire with nothing Go-side to notice. assignScript's id IS consumed, which is why the remedy spans both paths and belongs to that item. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
729abfd3b2 |
test(events): isolate the id==1 rotation branch, and record the ruling on the seed (BUG-2740, codex round 3)
The 'sequence starting at 1' case cleared the sequence AND the epoch, which is the obvious setup and does not isolate anything: with the epoch gone, the absent-epoch branch fires instead and produces an identical result, so the case passed with the id == 1 body deleted outright. It now leaves a LIVE epoch in place and asserts that as a fixture precondition — which is also the real shape, since event_seq is what gets evicted and the epoch is what survives pointing at the abandoned space. Mutation-verified: removing the branch now fails it with 'got 1'. Same lesson as N6 one round earlier, from the other side: a table that clears state uniformly only ever drives one branch, whether the uniformity hides a missing guard or a missing branch. Also records the lead's day-55 ruling against max(seed, current_epoch + 1), with the deciding reason rather than just the verdict: a repair path that reads a NEIGHBOURING shared key to compute its seed takes a dependency on that neighbour's health in exactly the state where neighbours are suspect. The residual is accepted as inside the ruling's intent — bounded to one resync, detected as epoch_regressed, never merges id spaces — and is reversible in one line. The sequence id's identical stringification hazard is filed as BUG-2744 and noted at the line, with the reachability corrected: it is reached by corruption, not by counting. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
d81f67a7a6 |
fix(events): publish the generation Redis holds, not Lua's rendering of it (BUG-2740, codex round 2)
Redis hands an integer reply to Lua as a NUMBER, and Lua 5.1 numbers are doubles printed with %.14g — so tostring() stops being faithful BELOW the 18 digits this guard admits. Measured at the boundary rather than reasoned about: a counter at 999999999999999998 increments to 999999999999999999, and tostring() renders that 1000000000000000000 while GET returns it exactly. The published epoch and the stored generation would disagree, and the receiver would adopt a generation the publisher does not hold. next_gen now reads the value back with GET. Pinned by a test at the guard's own limit, and mutation-verified: reverting to tostring fails it. 18 digits is not a magnitude a generation reaches by counting — it is the magnitude a hand-edited or collided key arrives at, which is exactly the class of event this guard exists for. The same hazard is pre-existing on the sequence id concatenated two lines later. Not touched here: ids reach that magnitude only after 1e18 events, and it is a different claim. Also documents what the wall-clock seed does NOT guarantee. 'Strictly above any increment-from-1 history' is the ruling's premise and is not the same as monotonic: a backwards clock step, or the key corrupted twice within one second, can seed at or below the generation receivers hold. Bounded two ways — consecutive repairs are already safe without the clock, since the first leaves a valid counter for the next publisher to increment; and a backwards generation is DETECTED as epoch_regressed rather than silently merging two id spaces. The stronger max(seed, epoch+1) repair is noted and deliberately not taken unilaterally, because the seed value is a ruling and that refinement only helps on the branches where the epoch is readable. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
483ac6e097 | test(events): drive all three rotation branches, not one (BUG-2740) | ||
|
xarmian
|
538b0d5ed4 |
fix(events): guard the generation counter the way the epoch key beside it is guarded (BUG-2740)
Every branch of publishScript that rotates the id space INCRs
pad:event_epoch_gen, and none checked the key first. The epoch key next to it
does, and its comment says exactly why — but the generation key never got the
same treatment.
An INCR against a corrupted key ABORTS THE SCRIPT, after the sequence INCR has
already landed. Redis is atomic against interleaving but does not roll back a
script's earlier writes, so the failure burns an id (a hole to every
receiver), repeats on the next publish, and never self-heals: the branch that
would rotate the generation is the branch that cannot run.
FOUR ABORT MODES, not the two the filing named. Measured against the pinned
miniredis rather than assumed:
list -> WRONGTYPE
hash -> WRONGTYPE
string 'abc' -> ERR value is not an integer or out of range
string '9223372036854775807' -> ERR increment or decrement would overflow
A TYPE check alone — the fix the filing describes — would have covered half of
them. The value is validated on the same terms the epoch guard uses: a
positive integer of at most 18 digits, which is what the receiver's
strconv.ParseInt can read back.
RESTART AT WALL-CLOCK SECONDS, per Dave's day-49 ruling, read from the item's
trail rather than from the dispatch message. Generations only have to be
orderable among themselves, and the corrupted key is the only witness to the
previous one, so nothing can be derived from it. Restarting at 1 would put the
new generation BELOW ones receivers have already adopted, which BUG-2736's
design reads as a regression.
The seed is passed as ARGV[3] rather than read with redis.call('TIME'), and
not for a replication reason — TIME does work here, measured. It keeps the
script deterministic and lets a test inject a fixed seed and assert the EXACT
repaired value, which is what tells 'repaired' apart from 'repaired to the
wrong thing'. Clock skew between publishers is harmless: the script is atomic,
so the first to reach a corrupted key repairs it and the rest simply
increment; no two seeds are ever compared.
Repair is SET, not DEL — SET replaces a key of any type (measured), and
BUG-2736's mutation matrix already established here that a DEL is removable
without any test noticing.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
|
||
|
xarmian
|
d075b39a73 |
test(events): the Redis atomicity test now exercises its interleaving (BUG-2730, codex round 20)
The test added one round earlier published only before the call and after it returned, which proves nothing about the guarantee it is named for: a version that released the lock between registering the subscriber and reading the buffer would pass it. events.RedisBus gets the same afterSubscribeRegister seam MemoryBus and watchevents.RedisBus already carry, and the test attempts a fan-out from inside the critical section — the one moment the guarantee is observable. Worth recording how the mutation went, because the first one was not a fair test either. Unlocking and immediately re-locking around the buffer read SURVIVED: Go hands the mutex straight back to the barging goroutine, so the window never opened and the "mutant" was not the wrong implementation, just a slower right one. The mutation that discriminates is the genuine two-critical-section structure — register, release, read under a fresh acquire — which is what the old handler did and what the guarantee forbids. That one fails. |
||
|
xarmian
|
0eb274fed8 |
test: close the mutation gaps a coverage audit named (BUG-2730, codex round 18)
Round 18 walked every behavioural change in the diff, named the smallest edit that would break it, and listed the ones no test caught. Twelve. All but two are now covered, and each new test was verified against the mutation it exists for: - the activity bus's gap channel coalescing (the watch twin had it, this one did not) - Redis-backed atomic subscribe-and-replay, which reaches the guarantee by a different mechanism than MemoryBus and would break alone - a resuming client being held to the per-workspace limit, so the new API is not a second door past a bound the fresh path enforces - the resume-gap report on the new path, with a fresh-subscription control so it cannot fire on every subscribe and still pass - the Redis drop metric, asserted per DROPPED SUBSCRIBER with two slow subscribers, so a report hoisted out of the fan-out loop halves the count and fails - the gap channel surviving Unsubscribe, since closing it would make a consumer's select spin - every subscribe API returning a non-nil signal - the watch handler incrementing the WATCH counter (countMidStreamResync takes a bool to choose, which is the kind of argument that gets passed the wrong way round), with both wrong-counter legs asserted - the production cooldown, which every handler test overrides, so nothing else would notice it set to zero - the wrapper's gauge on the atomic-resume path, which the previous assertion checked only for non-nil-ness Two left uncovered deliberately: an interleaving test at the handler level for the atomic API (the bus-level tests carry that guarantee and the handler cannot arrange the interleaving), and the same for the handler choosing the atomic call over subscribe-plus-EventsSince. Two existing tests were also repaired rather than kept green by luck: the ordering stress test demanded every published event and a slow reader legitimately loses some, and the new Redis atomicity test resumed against a workspace the bus was not covering. |
||
|
xarmian
|
d6832d6604 |
docs(bus): state the gap seam's invariant for the third cause (BUG-2730, codex round 17)
Asked what a maintainer adding a third reason a subscriber can be signalled would get wrong: nothing stopped them routing a cause with DIFFERENT remediation through the same payload-free channel, passing every test, and silently losing the distinction. Stated as a precondition on both signalGap implementations, with the enforcement named — the channel carries no payload, so a distinction it cannot represent cannot be lost downstream, because it cannot be put in. Coalescing makes the same point from the other side: two causes between two reads become one signal, so per-cause handling was always undecidable here. A condition needing different handling belongs on its own seam. A typed reason was the alternative and is not worth it for two causes with identical remediation — it would buy a field nobody reads and reintroduce a which-reason-wins question that coalescing has no good answer to. |
||
|
xarmian
|
1e839331db |
fix(events): concurrent publishes must deliver in ID order (BUG-2730, codex round 9)
The composition angle found the duplicate this unit did not close. SubscribeAndReplaySince shuts the window between subscribing and reading the replay, but MemoryBus.Publish still assigned the ID under replayMu and fanned out after releasing it — so two concurrent publishes could take N and N+1 and deliver in the other order. A subscriber that sees N+1 then N has a cursor that REGRESSED, and its next reconnect replays N+1 a second time. Same symptom as the window this unit is about, reached by a different route, and closing one while leaving the other open would have been a half-answer. Pre-existing rather than introduced here, and MemoryBus was the outlier: events.RedisBus holds its mutex across append and fan-out, and watchevents.MemoryBus holds its single mutex across both. This one now holds replayMu through the fan-out, which costs an O(subscribers) walk inside a lock a resume read also wants — the trade RedisBus has always made. Pinned by a stress test rather than a seam, because the fix is precisely that there is no longer a point between the two halves to pause at. It fails against the unserialized version on the first of five runs. |